Nate Soares Says AI Risk Led Him to Stop Retirement Saving

Nate Soares, president of the Machine Intelligence Research Institute, says he has not contributed to his Google-era 401(k) for about a decade because he believes advanced artificial intelligence could cause human extinction within the next 10 years. He estimates the chance is above 50 percent, a personal judgment rather than an established scientific forecast. Soares described his retirement decision and risk assessment in an interview with Nicholas Thompson, chief executive of The Atlantic. He said his account dates from his time working at Google. The estimate concerns hypothetical future AI systems; the reporting does not establish that current models can cause human extinction. Soares has spent more than a decade working on AI alignment, which seeks to ensure that powerful systems behave in ways compatible with human intentions. He argues that a highly capable system pursuing its objectives could treat people as obstacles if they tried to limit its actions or shut it down. In his scenario, such a system might evade oversight, improve its capabilities and acquire resources, with potentially catastrophic consequences if its objectives conflicted with human survival. He has...