Former Anthropic Researcher Warns Council on AI Control

Former Anthropic researcher Jacob Coxon told the New York City Council on October 5 that humanity is more likely than not to lose control of advanced artificial intelligence if development continues on its current path, and that the outcome could be human extinction. His warning came as council members considered proposals to add safety and accountability requirements for some AI systems. Coxon, who also worked at OpenAI before leaving Anthropic in September, said short-term measures such as human-operated shutdown controls would not address longer-term risks. He urged a slowdown in the development of frontier AI models, arguing that the ability of systems to improve themselves could make them harder to control. His assessment of the likelihood of losing control was a personal judgment, not a measured probability. Former OpenAI researcher Daniel Kokotajlo also testified, saying AI developers may not recognize when their safety work has failed. He described efforts to detect misalignment, or behavior that diverges from developers’ intentions, as inadequate and potentially worsening.