Microsoft's Diagnostic AI Wins a Test
Microsoft says its AI diagnoses up to four times more accurately than doctors, but the evidence is a case benchmark, not a patient trial.
Microsoft's claim that its diagnostic tool beats doctors is plausible but unproven for clinical use. MAI-DxO was reported to solve far more published NEJM case puzzles than physicians barred from colleagues and reference books, yet that benchmark is not a prospective patient trial, and Microsoft AI has not shown real-world superiority.
The Weights Desk · 3 min read- The 'up to four times more accurate' headline comes from a retrospective benchmark of published case records, not from treating patients.
- The human comparison group, as described in Microsoft's research, worked without colleagues, books or other tools, which is not how clinicians practise.
- Microsoft AI describes MAI-DxO as an orchestration layer over existing models, so the result depends on the underlying model and the prompting setup.
- No prospective trial, regulatory clearance or deployment outcome data was found in the sources reviewed.
- Treat the result as a research signal worth replicating, not as evidence that the tool should drive clinical decisions.
Microsoft AI says its diagnostic orchestrator can diagnose conditions up to four times more accurately than human doctors, and chief executive Mustafa Suleyman told The Verge that the promise of AI in healthcare 'is finally here'. The claim rests on a research benchmark, not a clinical deployment. On that benchmark the result is striking. As a statement about medicine, it is unproven, and the gap between the two is where the story sits.
What the benchmark actually measured
The figure comes from Microsoft's research on sequential diagnosis, which turns published NEJM case records into step-by-step puzzles. The system, MAI-DxO, asks questions and orders tests before committing to a diagnosis. Microsoft reports roughly 85 percent accuracy against about 20 percent for a group of physicians, which is the source of the 'four times' framing. These are the authors' own numbers on a retrospective test set, and independent replication has not been shown in the sources we reviewed.
Why the human baseline is weak
The comparison group was reportedly stripped of what clinicians normally use: colleagues, textbooks and search. That makes the 20 percent figure a measure of unaided recall on rare, deliberately difficult cases, not of everyday practice. A fair baseline would let physicians consult peers and references, and would use routine presentations as well as puzzle cases. Until that comparison exists, 'better than doctors' overstates what the data can support.
What the tool is, and is not
MAI-DxO is described by Microsoft AI as an orchestration layer that coordinates existing language models, not a new standalone medical model. That matters for reproducibility: results depend on the underlying model version and on the prompting and cost-control setup. It also means a hospital cannot assume the same performance after a model update. We found no regulatory clearance or prospective patient outcome data in the sources reviewed, so deployment claims would be speculation.
Verdict: a signal worth replicating
The result is a credible research signal that structured, multi-step AI reasoning can handle hard diagnostic puzzles, and the authors deserve credit for publishing a method. It is not evidence that the tool is safe or effective for patients. Health systems should wait for prospective trials against a realistic clinician baseline, with errors, costs and failure modes reported. Until then, treat the headline as vendor-led research, not a clinical result.
- Does Microsoft's AI really diagnose better than doctors?
- Only on a specific benchmark. Microsoft reports that MAI-DxO outperformed a group of physicians on sequentially presented NEJM case records. That does not show it outperforms clinicians treating real patients.
- What is the main limitation of the comparison?
- The physicians were reportedly denied the resources they normally use, such as colleagues and reference materials, and the cases are published puzzles, not routine presentations. Both choices favour the software.
- Is MAI-DxO approved for clinical use?
- We found no regulatory clearance in the sources reviewed. The tool is presented as research, and any clinical use would need prospective validation first.
- Microsoft says its AI tool is better than doctors at diagnosing conditions — The Verge
- Sequential Diagnosis with Language Models — arXiv (Microsoft AI authors)
- The Path to Medical Superintelligence — Microsoft AI