Key Findings
Current frontier AI agents complete less than half of the work when tasked with realistic clinical workflows. While large language models achieve near-perfect scores on multiple-choice medical licensing exams, they struggle with “long-horizon” tasks that require moving across different documents, software systems, and timeframes. Specifically, models consistently underperform in clinical decision support and administration and workflow—the areas where health systems most need to reduce administrative costs, which exceed $1 trillion annually in the U.S.
Technical Details
The research utilized three progressively harder benchmarks to measure AI capabilities:
- MedHELM: A task-based evaluation featuring a taxonomy of 5 categories, 22 subcategories, and 121 tasks. Win rates for frontier models varied by category: Clinical note generation (0.74 to 0.85), Patient communication and education (0.76 to 0.89), Medical research assistance (0.65 to 0.75), Clinical decision support (0.63 to 0.77), and Administration and workflow (0.53 to 0.63).
- MedAgentBench: Consists of 300 patient-specific tasks across 10 categories using a virtual electronic health record (EHR) environment. Completion rates were below 70% in early 2025, improving to over 90% by 2026.
- PhysicianBench: Tests 100 multi-step tasks adapted from real consultations across 21 specialties to evaluate end-to-end execution within a functioning EHR.
Background & Context
The shift toward agentic benchmarks was driven by the limitation of single-response grading. Traditional tests measure if a model can produce a correct text output, but real clinical work—such as prior authorization—requires a sequence of actions: reading an EHR, checking payer criteria, pulling documentation, filling forms, and recording reference numbers. A model’s ability to perform one step does not guarantee the completion of the entire process.
Significance & Outlook
The data highlights a “generative AI chasm” between a model’s stated capabilities and its actual utility in a health system. To close this gap, evaluation must move beyond text generation to environment-based testing where agents are given goals and tools, and success is measured by actual changes in the clinical environment.
Source: https://hai.stanford.edu/news/measuring-what-medical-ai-can-actually-do
Get our weekly technology intelligence — free
Receive an infographic that lets you judge at a glance whether each field’s analysis report is worth reading.
Subscribe Free — Weekly Tech Intelligence
By subscribing, you’ll receive Troy-Technical’s weekly technology intelligence newsletter.
- Your email and selected fields are used only to deliver the newsletter.
- We never share your information with third parties.
- You can unsubscribe anytime via the link in each email.
See our Privacy Policy for details.
Takes about a minute · Unsubscribe anytime
