OpenAI o1 Outperforms Physicians in ER Diagnosis Study

A new study published Thursday in the journal Science reveals that OpenAI's latest reasoning model, o1 preview, outperformed physicians in diagnostic accuracy and clinical reasoning tasks using real-world emergency department cases from a Boston hospital.

Researchers from Harvard Medical School and Beth Israel Deaconess Medical Center tested the model on a variety of challenges, including classic medical training scenarios, simulated cases and electronic health records from 76 actual patients treated at the facility. The AI included the correct diagnosis, or a close alternative, in its responses nearly 80 percent of the time across these evaluations. In contrast, two expert attending physicians reviewing the same initial triage data from emergency visits achieved correct diagnoses about half the time, while medical residents and older models like GPT-4 performed lower.

The study addressed longstanding benchmarks for clinical decision support, some tracing back to criteria outlined in a 1959 Science paper on surpassing human diagnostic ability. Co-senior author Adam Rodman, an internist at Beth Israel Deaconess, highlighted a case of an immunosuppressed organ transplant patient...