Microsoft's Diagnosis AI Is Now in Science. Let the Doctors Use References and Its Lead Falls From More Than 4 Times to 2.3.

On Thursday, October 22, 2026, Microsoft AI's diagnostic system made it through peer review. The paper behind MAI-DxO, the Microsoft AI Diagnostic Orchestrator, is now published in Science, and Microsoft marked it with a post titled "Building Medical Superintelligence."

The headline number is 83 percent. That is how often MAI-DxO reached the right diagnosis on 304 hard cases drawn from the New England Journal of Medicine's case records. Generalist physicians on the same cases scored 36 percent.

That is a big gap. It is also a much smaller one than Microsoft announced 479 days earlier, and the reason is worth understanding.

What Changed Since the Preprint

On Monday, June 30, 2025, Microsoft first showed MAI-DxO alongside an arXiv preprint. Paired with OpenAI's o3, it solved 85.5 percent of the same 304 NEJM cases. Twenty-one practicing physicians from the US and UK, with 5 to 20 years of experience, averaged 20 percent. Those doctors worked without colleagues, textbooks or generative AI, which Microsoft said was to keep the comparison fair. Few doctors work a hard case that way.

The Science version adds the arm that was missing. Microsoft now reports physicians at 22 percent without reference resources and 36 percent with them. Its own post puts MAI-DxO at "about 3.7 and 2.3 times more accurate, respectively."

In other words, letting the doctors look things up moved them from about one right answer in five to more than one in three. The AI's score barely moved. The ratio did.

Ratios are TensorFeed's arithmetic from the published percentages, except 2.3x, which Microsoft also states. Microsoft's 2026 post does not say how many physicians took part.

Where the 1.7x Comes From

Microsoft's post leads with a different multiple: MAI-DxO "is 1.7 times more accurate than physicians." That figure is from the second, new test set, typical emergency department cases from the MIMIC-IV-ED database, where MAI-DxO scored 80 percent and physicians 47 percent.

That is the more useful test in one way. Most patients are not NEJM puzzles. It is also the test where the orchestration layer seems to matter least. TechCrunch, reading the paper's data, reports that on the emergency cases GPT-5.5 scored about 85 percent both with and without MAI-DxO, at a similar cost.

The Model Is Doing Most of the Work

MAI-DxO is not a model. It is a harness that has a frontier model play a panel of doctors: ask questions, order tests, argue over the differential, then commit. Microsoft says it works across model families and that results held and improved with newer models, "including GPT-5.5, without changing the orchestration method."

The numbers TechCrunch pulled from the paper show how much depends on the model underneath:

• GPT-5 alone, with a short, basic prompt, scored 78.9 percent on the complex cases at about $8,400 per case. MAI-DxO cut that cost by about $3,000 and added about four points.
• GPT-5.5 went from 72 percent alone to nearly 87 percent with MAI-DxO, at slightly lower cost.
• Gemini 2.5 Flash went from 46.4 percent alone to nearly 69 percent with MAI-DxO.

So the harness helps most where the model alone scores lowest and the case is hardest. On a strong model and an ordinary emergency case, it adds little. Microsoft's own claim on cost is stated carefully: "Across both datasets, no other human or AI approach was both more accurate and less costly than MAI-DxO." Those costs are estimates from the simulator, not hospital bills.

What the Comparison Still Leaves Out

The physicians were primary care doctors and general internists working alone. As TechCrunch notes, cases this complex are usually handled by specialists or physician teams, who were not part of the comparison. The cases are historical and text-based, with findings revealed one request at a time by a gatekeeper model. No patient was in the room.

Microsoft is moving toward that. It says a new partnership with Beth Israel Deaconess Medical Center in Boston will run a prospective study in which patients interact with the system directly, measured against physicians' independent assessments. Dominic King, Microsoft AI's vice president of health, told TechCrunch: "This technology is ready for real-world evaluations."

Beth Israel is a busy test site. On Thursday, October 8, Google described a Lancet study run with the same hospital, in which 98 patients chatted with its AMIE research system before urgent care visits. Google reports that AMIE's differential matched the final diagnosis 90 percent of the time and that no conversation had to be stopped under its safety criteria.

Microsoft is also open-sourcing SDBench, the benchmark, gatekeeper and judge code, for select NEJM cases. The NEJM cases themselves must be licensed from NEJM, and the MIMIC data requires PhysioNet credentialing. MAI-DxO itself has no product, price or release date.

Our Take

Credit where it is due. Microsoft took a preprint with an easy-to-question physician comparison, put it through peer review at Science, and added the comparison that made its own result look smaller. The 36 percent arm is the single most important change in the paper, and Microsoft printed it.

But read the marketing against the data. "Medical superintelligence" is the frame. The evidence is a simulator, generalists working alone, and an orchestration layer whose gains swing from about four points to more than 20 depending on the model underneath. On ordinary emergency cases, TechCrunch's reading is that a frontier model alone gets about the same score. That is not an argument against MAI-DxO. It is an argument that the real asset here is the benchmark and the evaluation discipline, not the harness.

The number that will matter is the one the Beth Israel study produces, measured against doctors who can use references, consult colleagues and see the patient. If MAI-DxO still clears 2x there, Microsoft will have earned the headline. Until then, the honest version is narrower: a strong model with good scaffolding beats a lone generalist on hard paper cases, at lower modeled cost.

Three things to watch in the next 60 days: whether Microsoft names a start date and enrollment size for the Beth Israel study, whether it says how many physicians sat the Science comparison, and whether any outside group runs SDBench against specialists rather than generalists.

Sources: Microsoft AI, Science, Microsoft AI (June 2025), arXiv, SDBench, TechCrunch, The Verge, Google.