What Does a 94.3% Medical Agent Score Mean? Breaking Down the Metric Differences Between OpenAI and Stanford Evaluations

94.3% doesn’t mean 94.3% of healthcare administrative tasks were completed in full. The 94.3% for GPT-6 Astra that OpenAI announced on November 16, 2026, is the percentage of rubric points earned on HealthAdminBench; the 36.3% for Claude Opus 4.6 CUA reported in a Stanford Shah Lab paper is the full-task success rate. The first measures “how many rubric points were satisfied,” while the second measures “how many tasks met all success conditions.” They’re not the same metric, and they can’t be placed on the same leaderboard.

A healthcare agent can perform well on individual operations such as information retrieval, document processing, and form filling, yet still fail to reliably complete an end-to-end cross-system business process. To judge whether an agent has production value, you need to look at subtask performance, end-to-end success rate, error types, execution cost, and human intervention together. No single score is enough.

What will you learn from this article?

  • What 94.3% and 36.3% each measure, and why they can’t be compared directly
  • How HealthAdminBench is structured, where its rubric points come from, and its two validation mechanisms
  • A set of evaluation dimensions built for production environments that you can adapt to other enterprise agents

Contents: The facts section walks through, in order, why the 94.3% result drew attention, what HealthAdminBench actually measures, the difference between subtask success and full-task success, where healthcare agents are most likely to fail, and why the same model can perform differently under different execution frameworks; the analysis section then discusses how to build a comprehensive evaluation framework, how to account for costs, and what trade-offs enterprises should make when putting these systems to use.

medical-agent-94-3-score-openai-stanford-healthadminbench-metrics-en-image-0


What to Know: 7 Concepts for Understanding Medical Agent Evaluation

Concept Brief explanation
Computer Use The ability to complete tasks by observing software interfaces and simulating clicks, typing, scrolling, and other operations
EHR Electronic health record system, the core software hospitals use to store patient diagnosis and treatment information
MCP A standard protocol for discovering and invoking external tools
RPA An automation approach that executes repetitive processes through preconfigured rules or scripts
Rubric points Scoring points earned according to evaluation rules, used to measure the achievement of partial requirements
End-to-end success rate The proportion of complete tasks that meet all success conditions; failure of any key condition counts as failure
Trace Trajectory evidence recording the execution process, operational actions, and interface states

Medical Agent Evaluation Impact at a Glance

  • Developers and technical leads: You can’t just rely on a single vendor-published score for selection. You need to validate the execution mechanism, end-to-end completion rate, and full cost—including screenshots, retries, and runtime environment—against your own tasks.
  • Healthcare organizations and workflow owners: You need to distinguish four types of evidence—demos in synthetic environments, benchmark evaluations, institutional pilots, and production cases—and implement human approval, least privilege, and operational auditing during deployment.

Objective Facts: The Medical Agent Evaluation Behind 94.3% and 36.3%

Why did the 94.3% score attract attention?

On November 16, 2026, OpenAI announced progress on a computer-use agent for medical teams and disclosed several sets of evaluation data. GPT-6 Astra scored 94.3% on HealthAdminBench, and GPT-6 Luna scored 91.8%; both were evaluated at maximum reasoning effort. OpenAI also reported that Luna’s cost per task was $0.085, 98% lower than Astra’s.

These numbers drew attention because Stanford’s previously published HealthAdminBench results were notably lower. It’s important to note that the 94.3% and 91.8% OpenAI published are the share of rubric points earned, not standalone end-to-end task success rates. The other data points in the same announcement also come from different evaluation stages, each with a different denominator:

Metric Value Denominator/scope What it doesn’t represent
HealthAdminBench score (Astra) 94.3% Share of rubric points End-to-end task success rate
HealthAdminBench score (Luna) 91.8% Share of rubric points End-to-end task success rate
Clinical task safety (Astra) 99% 325 reviewed tasks Medical diagnostic accuracy
Strong physician approval rate (Astra) 93% 646 physician evaluations Task success rate, and it doesn’t equal 646 physicians either
Luna cost per task $0.085 Cost per task under specific evaluation conditions Production deployment pricing

These three sets of data have different denominators and can’t be combined into a single “overall accuracy rate.” In addition, OpenAI’s announcement text did not disclose per-task results, the full experimental setup, confidence intervals, or all failure cases, so outsiders currently can’t independently verify how the 94.3% score was constructed. Astra’s cost per task is about $4.25; this is an estimate back-calculated from the 98% reduction, not a price OpenAI directly published.

medical-agent-94-3-score-openai-stanford-healthadminbench-metrics-en-image-1

What Does HealthAdminBench Actually Measure?

HealthAdminBench was proposed by the Shah Lab at Stanford University and is designed specifically to evaluate how computer-use agents perform on healthcare administrative tasks. Its public-facing versions have changed over time: the early research webpage described 70 tasks, 947 scoring points, and 6 agents, while the paper version 2604.09937v1 expands this to 135 tasks, 1,698 verifiable subtasks, and 7 model configurations. This article follows the paper version; the early webpage data is used only for historical comparison.

The four simulated GUI environments in the paper version are EHR, insurance payment portal A, insurance payment portal B, and a fax portal. The 135 tasks fall into three categories of administrative workflows:

Task Type Count Main Content
Prior Authorization 60 Insurance eligibility verification, medical record collection, and prior authorization submission
Appeals and Denials Management 60 Reviewing denials, collecting supporting evidence, preparing and submitting appeals
DME Order Processing 15 Collecting durable medical equipment documentation, submitting orders, and recording results

The 1,698 scoring points come from two mechanisms. A total of 1,177 use deterministic checks, relying on structured methods such as JMESPath to verify system state—for example, whether the correct patient was found, whether the specified fields were filled in, and whether the correct order was generated. The approach is close to traditional software testing. The other 521 are judged by an LLM Judge, mainly assessing whether document content is sufficient and whether submitted materials meet requirements—areas that are hard to verify directly with rules.

All tasks use synthetic patient data, not real patient records. That means the benchmark provides a reproducible test environment; it does not test in a real hospital production system, nor does it directly measure patient care outcomes.

What’s the difference between subtask success and full-task success?

A prior authorization request workflow makes the difference easiest to see. According to the research materials, a prior authorization task may require an Agent to complete the following in sequence:

  1. Find the correct patient in the EHR
  2. Review the patient’s medical records
  3. Locate imaging studies and prior treatment materials
  4. Determine what documentation the insurance request requires
  5. Extract the required information from the medical record
  6. Go into the insurance payment portal
  7. Select the correct request type
  8. Fill out the request form
  9. Attach the required documents
  10. Submit the request and confirm the status

These ten steps have strong dependencies. Get the patient wrong at step 1, and no matter how well you do every later step, it’s still wrong. Even if the first nine steps are all correct, a missed attachment at step 9 or a failure to actually submit at step 10 still makes the whole task a failure. Subtask success rate measures “how many of the ten steps were done correctly”; full-task success rate measures “whether all ten steps were done correctly.”

medical-agent-94-3-score-openai-stanford-healthadminbench-metrics-en-image-2

The data in the paper clearly captures this gap: GPT-5.4 CUA has a subtask success rate of 82.8%, while Claude Opus 4.6 CUA has a full-task success rate of 36.3%. These two numbers come from different models, so again they can’t be compared directly, but together they make one point: completing local actions and completing an entire business process are two different capabilities.

An earlier research web page reported a similar gap: Claude Opus 4.5’s best full-task success rate was 22.9%, and Gemini 3’s subtask success rate was 76.4%. Because task scale and versions differ, these numbers are only useful for spotting trends; they can’t be put into the same ranking as the paper’s version or OpenAI’s results.

Where do medical agents fail most often?

The paper reports results for two CUA configurations by subtask category:

Subtask category Claude Opus 4.6 CUA GPT-5.4 CUA
Information retrieval 96.5% 96.1%
Documentation 74.3% 84.4%
Form filling 88.9% 84.9%
Task solving 70.7% 66.2%
Document Handling 92.3% 94.3%
Clinical reasoning 61.3% 77.0%

One thing worth flagging: GPT-5.4 CUA’s 94.3% on Document Handling in this table being the same as the 94.3% overall rubric points OpenAI published for GPT-6 Astra is just a numerical coincidence. The models, metrics, and sources are completely different.

Looking at the breakdown, information retrieval is already near saturation, while clinical reasoning, task solving, and documentation are relatively weak. The paper also points out that Document Handling and Cross-system Data Transfer are the main issues causing end-to-end failures. An earlier web version reported that five out of six models had accuracy below 20% on the document-handling subtask, but that was a different version of the task setup and can’t be directly compared with the table above.

Why is cross-system transfer especially error-prone? In the prior authorization workflow, the same piece of information has to be read from the EHR, stored in the agent’s context, and then written into a different form structure in the insurance portal; attachments may also pass through a fax portal. Every transfer is another chance for something to go wrong, and an error upstream gets copied as-is into the downstream system. The risks listed in OpenAI’s materials also include “propagating incorrect information from one system to another” and “incorrectly reporting task completion when system state is unclear.”

Why does the same model perform differently under different execution frameworks?

The paper ran ablation experiments on prompting styles and observation modes, and came away with three main findings:

  • Providing Portal Guidance usually performs better than giving only a task description
  • Observing through the Accessibility Tree generally beats pure screenshot observation
  • Combining portal guidance with structured page information tends to yield better end-to-end performance

The paper also points out that real production environments may not always expose a complete Accessibility Tree. In other words, an Agent’s results depend not only on the underlying model, but also on the harness, the way the page is observed, the prompt, and the task description.

OpenAI’s developer documentation reflects the same variation in execution paths. The official guide separately covers ComputerToolset, BrowserToolset, and the hosted browser entry point; for GPT-6 Astra, the current docs recommend the Code Execution approach, where the model generates Playwright or similar code to drive the interface, with the Computer Tool still available as an option. The docs also emphasize that a model generating an action instruction doesn’t mean the action actually succeeded — the application has to execute it and then verify the result with a fresh screenshot or some other method.

CVS Health’s open-source CUA Agent Kit (Apache-2.0) takes a different engineering angle. Its README describes the mechanism: first, a Computer-use Agent explores the workflow and saves screenshots, actions, and traces as evidence; then a Codex Skill turns the successful trajectory into deterministic code, which engineers review; subsequent deterministic runs don’t need a model key. This splits “figuring out how to do it” and “doing it reliably, over and over” into two separate problems.


Analysis and judgment: How should we evaluate a truly usable medical Agent?

How should we evaluate a truly usable Agent?

The following is this article’s judgment, not an official standard from any organization. We believe scores like 94.3% on rubric points are good for answering “how much has the model improved on a narrow capability,” but not enough to answer “can it go into a production workflow.” The latter question requires a set of mutually constraining metrics:

Evaluation dimension What it measures Why it’s needed Data source
End-to-end success rate The share of complete tasks that meet all success conditions The business only cares about the final outcome Final system state validation
Key-step accuracy Correctness rate at key points such as selecting a patient, submitting, and uploading attachments One error in a key step breaks the whole task Deterministic checks
Error severity Whether a failure is a recoverable omission or involves writing wrong data or sending a message to the wrong recipient The same failure rate can carry very different levels of risk Manual review + severity labels
Human intervention rate The share of cases that require human takeover or approval-based correction Determines the actual labor saved Approval logs
Execution time Per-task latency and long-tail latency Affects throughput and queueing Trace timestamps
Full cost per task The sum of model, screenshot, retry, runtime environment, and maintenance costs Token unit price doesn’t represent the real cost Billing + run records
Exception recovery capability Whether it can stop or roll back when it encounters ambiguity, timeouts, or page changes In medical scenarios, it must “know when to stop” Fault injection testing

The core of this framework is to report “how many steps were done right” and “how many tasks were actually completed” separately, then adjust both using error severity and human intervention rate. The approach Ambience demonstrated reflects exactly the “exception recovery” dimension: when a medical record is ambiguous, the Agent stops and asks the doctor, and only submits the prior authorization request after the doctor approves. This is just the governance approach the company showcased; it doesn’t mean every medical Agent has the same configuration by default.

medical-agent-94-3-score-openai-stanford-healthadminbench-metrics-en-image-3

Sample size and number of repetitions also need to be disclosed. A study comparing RPA and Agents (DOI: 10.1080/23311975.2026.2724760) ran each technology 10 times on two workflows, P2 and P3, but on P1 the Agent ran only once and failed, so that result was excluded from later statistics. The study used a 2025 tool configuration, so it can’t be treated as a direct score for 2026 models. This example shows that when reading evaluation conclusions, you should first check how many runs sit behind each number and which results were excluded.

A Practical Approach to Medical Agent Evaluation: Run It Yourself Under the Same Harness

For development teams, the most practical approach is to fix the execution framework and, on your own task set, track both subtask-level and end-to-end metrics at the same time. The example below uses an LLM Judge to score the materials an Agent submits, with an interface compatible with the OpenAI SDK:

import openai

client = openai.OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.apiyi.com/v1"  # APIYI unified interface, lets you switch between different judge models
)

resp = client.chat.completions.create(
    model="YOUR_JUDGE_MODEL",
    messages=[
        {"role": "system", "content": "你是评测裁判,只输出 PASS 或 FAIL 及一句理由。"},
        {"role": "user", "content": "要求:附上影像报告与既往治疗记录。\n提交内容:<Agent 提交的材料摘要>"}
    ]
)
print(resp.choices[0].message.content)
View the full implementation: compute subtask success rate and end-to-end success rate together
import openai
from dataclasses import dataclass, field

client = openai.OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.apiyi.com/v1"
)

@dataclass
class Subtask:
    name: str
    critical: bool          # whether this is a critical step (a failure here fails the whole task)
    passed: bool = False

@dataclass
class TaskResult:
    task_id: str
    subtasks: list = field(default_factory=list)
    human_intervened: bool = False
    cost_usd: float = 0.0   # already includes retry tokens, don't add them again

def llm_judge(requirement: str, submission: str, model: str) -> bool:
    """Use an LLM Judge for subtasks that are hard to formalize as rules"""
    resp = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "只输出 PASS 或 FAIL。"},
            {"role": "user", "content": f"要求:{requirement}\n提交:{submission}"}
        ]
    )
    return resp.choices[0].message.content.strip().upper().startswith("PASS")

def summarize(results: list) -> dict:
    total_sub = sum(len(r.subtasks) for r in results)
    passed_sub = sum(s.passed for r in results for s in r.subtasks)
    full_success = sum(all(s.passed for s in r.subtasks) for r in results)
    critical_fail = sum(
        any(s.critical and not s.passed for s in r.subtasks) for r in results
    )
    n = len(results)
    return {
        "subtask_rate": passed_sub / total_sub,
        "end_to_end_rate": full_success / n,
        "critical_failure_rate": critical_fail / n,
        "human_intervention_rate": sum(r.human_intervened for r in results) / n,
        "avg_cost_usd": sum(r.cost_usd for r in results) / n,
    }

🎯 Technical tip: It’s best to report subtask scores and end-to-end success rates side by side, on the same harness and the same batch of tasks. With the OpenAI-compatible interface from APIYI apiyi.com, a single API key lets you switch among OpenAI, Claude, Gemini, and other models — whether as the execution model or the judge model — cutting down on the variables that come from different integration methods.

Medical Agent Cost Accounting and Model Tiering

Per-token pricing is only the starting point. Based on the Standard short-context pricing OpenAI published on November 17, 2026, GPT-6 Astra runs $10, $50, and $1 per million tokens for input, output, and cached input respectively, while GPT-6 Luna comes in at $0.10, $0.50, and $0.01. Processing speed, context length, and region all affect what you actually get billed. Computer Use is a multi-turn invocation, and every screenshot counts as vision input, so when you’re doing the math, tally the vision/text input and billed output for each turn separately, then add tools, runtime environment, and engineering maintenance on top. If retry tokens are already included in the total, don’t add them again.

Judging by the gap between $0.085 and roughly $4.25, model tiering will likely become the norm: hand complex, reasoning-heavy tasks to high-capability models, route highly structured and high-frequency work to low-cost models, and harden fixed workflows that have already proven themselves into deterministic code — much like the CVS approach. One caveat: these per-task costs come from specific benchmark conditions and shouldn’t be treated as production quotes.

💰 Cost optimization: Before you decide on tiers, run a small batch of real tasks through each model to measure its end-to-end success rate and full per-task cost. APIYI apiyi.com bills pay-as-you-go and gives new users free test credits, which makes it a good fit for running these comparison experiments cheaply during the project kickoff phase.

From Healthcare Agents to Enterprise Agents: Transferable Evaluation Approaches

What makes healthcare administration special is its high stakes and strict regulation, but the problems it exposes also show up in enterprise workflows such as insurance claims, financial reconciliation, and supply chain documents: multiple systems, strong dependencies, and cross-system data transfer. Our view is that the following four principles transfer directly:

  1. Separate the reporting: Report subtask scores and end-to-end success rates separately—never use the former as a substitute for the latter.
  2. Label the evidence level: Synthetic-environment demos, vendor self-assessments, third-party benchmarks, enterprise pilots, and production cases each belong in their own category—don’t mix them.
  3. Fix the harness before comparing: Once the observation method, prompt, or portal guidance changes, results are no longer comparable.
  4. Governance before scores: Least privilege, human approval for critical actions, post-action state verification, full trace logging, and anomaly halt mechanisms should be acceptance criteria before launch.

When running a PoC inside an enterprise, you can tag task sets by “critical step,” first establish an end-to-end baseline, then gradually introduce portal guidance, structured observation, or fixes baked into code, and observe each change’s real contribution to the overall success rate. If you need to quickly reproduce the same batch of tasks across multiple models, you can access them through APIYI apiyi.com in a unified way, reducing the engineering cost of side-by-side comparison.

Developments worth watching include: whether OpenAI will disclose the per-task results and experimental configuration behind the 94.3%; whether later versions of HealthAdminBench will publish complete task success rates for new models under the same methodology; and whether more organizations can disclose human intervention rates and error classification data from real production environments. Until that information appears, any claim that “healthcare administration can be fully automated” is speculation.

94.3% demonstrates significant model progress on scoring points, while 36.3% reminds us that whether a business process can be completed reliably must ultimately be tested on complete tasks, real systems, and human oversight. As the English proverb goes, “The proof of the pudding is in the eating.” (You only know if the pudding is good by eating it; actual results must be verified in practice.)


References

  1. OpenAI: Advancing computer-use agents for healthcare teams: evaluation figures such as 94.3%, 91.8%, 99%, and 93%, plus partnership case studies
    • Link: openai.com/index/advancing-computer-use-for-healthcare-teams/
  2. HealthAdminBench paper 2604.09937v1: 135 tasks, 1,698 subtasks, per-category results, and ablation experiments
    • Link: arxiv.org/html/2604.09937v1
  3. HealthAdminBench project page and code repository: benchmark environment and evaluation implementation
    • Link: som-shahlab.github.io/health-admin-bench-website/
    • Link: github.com/som-shahlab/health-admin-bench
  4. Stanford Shah Lab early research page: historical version with 70 tasks and 947 scoring points
    • Link: shahlab.stanford.edu/doku.php?id=healthadminbench
  5. OpenAI Computer Use developer docs and API pricing: execution paths and Standard pricing
    • Link: developers.openai.com/api/docs/guides/tools-computer-use
    • Link: developers.openai.com/api/docs/pricing
  6. CVS Health CUA Agent Kit: exploration, evidence, code hardening, and engineer review mechanisms
    • Link: github.com/cvs-health/cua-agent-kit
  7. Comparative study of RPA and Agents: repetition counts and P1 exception notes
    • Link: doi.org/10.1080/23311975.2026.2724760
  8. Cambridge Dictionary: definition of the proverb “proof of the pudding is in the eating”
    • Link: dictionary.cambridge.org/us/dictionary/english/proof-of-the-pudding-is-in-the-eating

Author: APIYI Technical Team
Technical discussion: You’re welcome to visit APIYI at apiyi.com to talk with us about Agent evaluation practices. For more integration guidance, see the docs center at docs.apiyi.com

Leave a Comment