The scientific A-Team saving the world from killer viruses, rogue AI and the paperclip apo

[geomancer]

https://www.theguardian.com/technolo...men-save-world

The scientific A-Team saving the world from killer viruses, rogue AI and the paperclip apocalypse



Cambridge, some time after the end of term. Demob-happy undergraduates, dressed for punting and swigging wine from the bottle, seem not so much to be enjoying themselves as determinedly following rites of passage on the way to a privileged future. I am heading towards the biggest, richest and arguably most beautiful college: Trinity. Of the 90 Nobel prizes won by members of Cambridge University in the 20th century, 32 were won by members of Trinity. Its alumni include Isaac Newton, Wittgenstein, Bertrand Russell and six prime ministers.

The porter's lodge is like an airlock, apparently sealed from the tribulations of everyday life. But inside the college, pacing the flagstones of what is called – all modesty aside – Great Court, are four men who do not take it for granted that those undergraduates actually have a future. They are the four founders of the Centre for the Study of Existential Risk (CSER), and they are in the business of "horizon scanning". Together, they are on alert for what they sometimes call "low-probability-but-high-consequence events", and sometimes – when they forget to be reassuring – "catastrophe".

At their head is a 72-year-old cosmologist, Martin Rees. The honorifics jostle at the start of his name: he is Professor Martin Rees, Baron Rees of Ludlow, OM FRS. He is the Astronomer Royal, a fellow of Trinity, formerly a master of the college and a president of the Royal Society. In newspaper articles, he is often described simply as Britain's "top scientist". In 2003, Rees published a book called Our Final Century. He likes to joke that the reason his book was published in the US as Our Final Hour is because "Americans like instant gratification". In the book, he rates the chances of a "serious setback" for humanity over the next 100 years at "50-50". There is an asteroid named after him – 4587 Rees. I can't help thinking, in light of his apocalyptic concerns, that it would be ironic if 4587 Rees crashed into the Earth.

But these four men are less concerned with acts of God than those we have created ourselves: the consequences of being too clever for our own good. They believe there is a risk that artificial intelligence (AI) will challenge our own. In a talk at a TED conference, Rees invoked another danger: that "in our interconnected world, novel technology could empower just one fanatic, or some weirdo with the mindset of those who now design computer viruses, to trigger some kind of disaster. Or catastrophe could arise from some technical misadventure – error rather than terror."

Rees proudly introduces his colleagues. There is Jaan Tallinn, a meditative Estonian computer programmer and one of five co-founders of Skype. There is a courtly Indian economic theorist, Professor Sir Partha Dasgupta ("Partha's very concerned with inequalities across time," Rees says). And there is Huw Price, a laid-back philosophy don – specifically, the Bertrand Russell professor of philosophy at Cambridge.

The group originated in 2011, when Price and Tallinn met at a conference on time in Copenhagen. Two weeks later Price, who had just taken up his philosophy post, invited Tallinn to Cambridge to meet his new colleague, Rees; all three shared a concern about near-term risks to humanity. "Fate," Price recalls, "was offering me a remarkable opportunity." After a two-year gestation, the CSER gets properly up and running next month. The first of a dozen post-doctoral researchers will be taken on, some of whom will be embedded with science and technology firms. There will be seminars on synthetic biology, decision theory and AI. Already there have been meetings with the Cabinet Office, the Ministry of Defence and the Foreign Office.

As the salutary clock of the Great Court looms behind them, the irresistible image of our leading brains uniting to save the planet: X-Men: The Last Stand, The Four Just Men, Guardians Of The Galaxy. Between photographs, Rees and Dasgupta chat about the relationship between facts and prejudice in global warming forecasts, and I wonder if they ever talk of anything other than the end of the world.

Before we met, I was sent a vast amount of reading material, including a paper touchingly described by Dasgupta as "somewhat informal", but still containing much algebra. Most strikingly, the material included four worst case possibilities:

1 The disaffected lab worker
In which an unhappy biotech employee makes minor modifications to the genome of a virus – for example, avian flu H5N1. A batch of live virus is created that can be released via aerosol. The lab worker takes a round-the-world flight, stopping off at airports to release the virus. The plausibility of this scenario is rated as "high", and "technologically possible in the near term". As the CSER men note: "No professional psychological evaluation of biotech lab staff takes place." A similar leakage might also happen accidentally, and I was sent, as a matter of urgency, an article from the Guardian about how researchers at the University of Wisconsin-Madison had modified strains of bird flu to create a virus similar to the 1918 Spanish flu that killed 50m people. The project was condemned as "absolutely crazy" by the respected epidemiologist Lord May.

2 Termination risk
In which pressure to stop climate change results in the adoption of stratospheric aerosol geo-engineering. Global warming is checked, but CO₂ levels continue to rise. The geo-engineering then ceases, perhaps as a result of some other catastrophe, such as world war. This triggers what is called "termination risk": the sticking plaster removed, the warming gets much worse, quickly. Half the Earth's population is wiped out. I was advised that geo-engineering appears possible in the near term, but the scientific consensus is against adopting it.

3 Distributed manufacturing
3D printing is already used to make automatic weapons. These weapons can work, but are liable to explode in the user's hand. Still, the refinement of these techniques may allow nanoscale manufacture of military-grade missiles. "This would require a range of technological advances currently beyond us," I was told, "but believed by many scientists to be possible."

4 All of America is turned into paper clips
In which AI undergoes runaway improvement and "escapes into the internet". Imagine a computer swallowing all the information stored in Wikipedia in one gulp and generally gaining access to everything human-made. (The already-emergent "internet of things" means that, increasingly, devices can communicate between themselves; our homes are becoming more automated.) This rogue machine then uses human resources to develop new and better technologies to achieve its goal. I was given the for-instance of a paper clip making software that turns the whole of America, including the people, into paper clips. This is "not technologically possible in the next 20 years. Estimates range from 20 years to 300 years to never. But the potential negative consequences are too severe not to study the possibility."

This is what these four men are up against.



Martin Rees: ‘A few crazy pioneers might establish colonies in outer space. But nowhere out there is comfortable, so we have to solve the problems here.’ Photograph: Jon Tonks for the Guardian

more here: https://www.theguardian.com/technolog...men-save-world