Microsoft's MAI-DxO beats generalist doctors on hard cases in Science study

Microsoft's AI diagnostic system has passed peer review. A paper published on 22 October in Science reports that MAI-DxO reached the right diagnosis in complex medical cases far more often than generalist physicians did, while ordering cheaper sets of tests. Mustafa Suleyman, the head of Microsoft AI, told TechCrunch he sees it as a step towards "medical superintelligence".

How MAI-DxO works

Most AI medical tests hand a model a tidy case summary and ask for an answer. Microsoft's team built a benchmark called SDBench that works more like a real consultation. The system, or a doctor, starts with only a short description of the patient's complaint and has to decide which questions to ask and which tests to order, and each test has an estimated cost.

MAI-DxO is an orchestrator that sits on top of large language models. Microsoft describes it as a panel of virtual physicians: several agents propose diagnoses, pick useful questions or tests and challenge each other over several rounds. The system then weighs each test's likely value against its cost.

The results

• Complex cases: on 304 clinicopathological cases from the New England Journal of Medicine, MAI-DxO scored 83.2%. Generalist physicians managed 36% with textbooks and the internet and 22% without them.
• Everyday emergencies: on emergency-department cases from the MIMIC-IV-ED dataset, it scored 80.3% against 46.8% for physicians, about 1.7 times better.
• Cost: a cost-conscious setting still reached 80.9% with an average estimated test bill of $2,241, compared with $5,431 for the most accurate unconstrained setting.

Microsoft says the approach is not tied to one model. The paper's main runs used GPT-5, and Microsoft says the results held up and improved with newer models, including GPT-5.5. It is also open-sourcing parts of SDBench for selected NEJM cases so other researchers can check the results.

The caveats

The authors are clear about the limits. Doctors in the study worked alone and were generalists, not the specialists or teams that normally handle cases this hard. The benchmark is text only, with no imaging or pathology slides, and every case had a disease, so the mix does not reflect a normal clinic. Costs use standard US prices. Most importantly, the study only measured diagnosis, not treatment, safety or patient outcomes.

Our take

This is a stronger result than a chatbot passing a multiple-choice exam, because the system had to choose its own tests. It is still a benchmark, not a clinical trial. The next real milestone is a prospective study in hospitals, with doctors in charge. Until then, keep using AI chatbots for health questions with care, and check what they do with your data. For more on Microsoft's own models, see our piece on MAI-Base-2, and for how independent testers audit AI systems, read about METR's access to Anthropic's models.