Raycaster/ Eval

Biopharma Bench V0.2

Biopharma Bench is an AI agent benchmark for pharmaceutical regulatory and CMC work. Agents occupy functional seats across 24 drug and device company environments, reconciling eCTD leaves, QMS/LIMS records, stability datasets, and health-authority correspondence to deliver submission-ready documents and workbooks.

Announcement & reportHugging Face datasetRun on Harbor
24Company environments
81Total tasks
908Scoring criteria

Benchmark results

Muse Wonder 1.0 leads with a 59.3% mean score across all 81 tasks (908 rubric criteria across 24 drug and device company environments). 4 of 12 evaluated models achieve 100% of criteria on at least one task.

Biopharma Bench V0.2 · 81 tasks and 908 rubric criteria across 24 drug and device company environments.

Benchmark results for 12 evaluated models
RankModel
1Cursor CLI · high59.3%2/812.5%$1.5632.531.26.2 min
2Claude Code · high58.2%1/811.2%$6.6626.942.913.7 min
3Codex · high57.1%0/81none$0.6729.923.912.8 min
4Pi · high56.5%1/811.2%$1.5232.451.812.2 min
5Codex · high56.0%0/81none$4.1829.423.48.8 min
6Claude Code · high53.0%2/812.5%$3.0246.548.29.5 min
7Claude Code · high50.3%0/81none$1.5152.655.79.7 min
8Pi · high47.0%0/81none$0.3056.365.98.4 min
9Cursor CLI · high41.8%0/81none$1.8632.755.112.0 min
10Pi · high40.8%0/81none$0.1139.442.25.0 min
11Cursor CLI · high35.5%0/81none$1.3025647.2 min
12Codex · high32.2%0/81none$0.0918.112.12.5 min

Mean score is the average percentage of rubric criteria passed across all 81 tasks. Pass@1 is the percentage of tasks completed with a perfect score on the first attempt. Cost, steps, tool calls, and time are per-seat means over each seat's completed agent trials (infrastructure-killed trials excluded), then equal-weighted across the 81-seat roster. Repeated runs of the same task typically differ by 15–40% in cost (up to 2–3× at the extremes), so each task averages all of its valid runs. Cost is agent-only at current list price; Grok and Muse Wonder are Cursor list-price estimates on token counts.

Score against cost, effort, and release date

  • Sol · Codex ★
  • GLM · Pi ★
  • DeepSeek · Pi ★
  • Terra · Codex ★
  • Grok 4.6 · Cursor CLI
  • Grok 4.7 · Cursor CLI
  • Fable · Claude Code ★
  • Opus 5.5 · Claude Code
  • Astra · Codex
35%45%55%65%$0.00$2.00$4.00$6.00Mean cost per task (USD)Mean scoreFable · Claude CodeSol · CodexDeepSeek · PiGLM · PiTerra · CodexAstra · CodexOpus 5.5 · Claude CodeGrok 4.7 · Cursor CLIGrok 4.6 · Cursor CLI
Mean score against cost, steps, tool calls, time, or release date.

Company environments across the regulatory lifecycle

The v0.2 benchmark evaluates 81 tasks across 24 confidential drug and device programs spanning IND-enabling development through commercial lifecycle management, shown below alongside two companion showcase environments (Havenor Therapeutics and the Broad Institute).

  1. 1Discovery & Lead Selection

    Target validation, candidate selection, and early construct design.

  2. 2IND-Enabling Nonclinical

    GLP toxicology, safety pharmacology, bioanalytical qualification, and tox-batch CMC.

  3. 3Clinical Development (IND / CTA)

    Phase 1–3 clinical conduct, IND clinical hold responses, GCP site operations, and eCTD sequence control.

    1 more
  4. 4Registration & Dossier Review

    NDA, BLA, MAA, ANDS, and PMA submissions, CTD Module 2/3 freezes, and health-authority deficiency responses.

    11 more
  5. 5Approval & Launch Readiness

    Marketing authorization close-out, prescribing information lock, and post-marketing commitment baselining.

  6. 6Commercial Lifecycle & GMP

    Post-approval CMC variations, PPQ/tech transfer, pharmacovigilance signal evaluation, and GMP inspection responses.

    4 more

Inside

The leaderboard covers the v0.2 freeze across 24 confidential drug and device company environments. Because partner product files remain sealed, we publish two companion showcase environments in full: Havenor Therapeutics (synthetic sterile-injectables CMC/MSAT) and the Broad Institute (public FDA IND 167326 record) for end-to-end inspection of workspaces, trajectories, deliverables, and rubric grades.

10 CMC, MSAT, and QA tasks at Havenor Therapeutics.Sterile parenteral fill-finish, ICH Q1E stability regression, sterilizing filtration validation, container-closure integrity, and supplier change control.

  1. Task 01Prepare the next AVI-401 stability commitment updateRegulatory CMC ManagerGPT-6.1 SolGPT-6 Astra: 7 / 137 / 13
  2. Task 02State the technical position on the alternate overwrap changeHead of CMC and MSATGPT-6.1 SolGPT-6 Astra: 6 / 127 / 12
  3. Task 03Review the AVI-401 sterile-filter validation studiesPrincipal MSAT EngineerClaude Opus 5.5GPT-6 Astra: 7 / 133 / 13
  4. Task 04Answer the regulator’s seven questionsRegulatory CMC ManagerGPT-6.1 SolGPT-6 Astra: 11 / 1411 / 14
  5. Task 05Pre-submission records integrity checkQA Product LeadClaude Opus 5.5GPT-6 Astra: 5 / 128 / 12
  6. Task 06Check the container-closure validation recordsValidation EngineerClaude Opus 5.5GPT-6 Astra: 5 / 125 / 12
  7. Task 07Supplier and change record integrity checkSupplier Quality ManagerGPT-6 AstraGPT-6 Astra: 6 / 126 / 12
  8. Task 08Confirm whether stability results meet the specification in forceQA Product LeadGPT-6.1 SolGPT-6 Astra: 5 / 126 / 12
  9. Task 09Review the 24-month stability-testing workbookQC Stability ScientistGPT-6.1 SolGPT-6 Astra: 7 / 158 / 15
  10. Task 10Decide which stability-testing operations can proceedQC Stability ScientistGPT-6.1 SolGPT-6 Astra: 11 / 1514 / 15
Open the Havenor Therapeutics showcase →

4 tasks from FDA CDER IND 167326 partial clinical hold (21 CFR 312.42).Reconstructed from the Broad Institute’s public divalent siRNA IND dossier, FDA’s April 2025 hold letter, ELISA plate exports, GLP toxicology records, and oligonucleotide stoichiometry.

  1. Task 01Plan the response to the FDA’s partial clinical holdRegulatory operations associateclaude-opus-510 / 12
  2. Task 02Defend what the prion-protein spinal-fluid test may claimBioanalytical scientistclaude-opus-510 / 12
  3. Task 03Audit the animal-study record behind the holdNonclinical toxicologistclaude-opus-511 / 13
  4. Task 04Check the manufacturing math the filing got wrongCMC technical writerclaude-opus-512 / 13
Open the Broad Institute showcase →

Results by domain

Score range across the 12 models, by health authority, product modality, and work family. Select a model on the leaderboard to highlight it.

DomainScore range (lowest → highest)
US FDA27 tasks
GPT-6.1 Terra (34.6%)Δ 28.0%Muse Wonder 1.0 (62.6%)
China NMPA / CDE12 tasks
Grok 4.6 (17.1%)Δ 38.2%Claude Opus 5.5 (55.3%)
Korea MFDS7 tasks
Grok 4.6 (29.9%)Δ 42.3%GPT-6 Astra (72.2%)
Health Canada4 tasks
GPT-6.1 Terra (25.2%)Δ 37.3%Muse Wonder 1.0 (62.5%)
Small molecules48 tasks
GPT-6.1 Terra (32.8%)Δ 26.8%Kimi K3.1 Pro (59.6%)
Biologics, cell & gene therapies19 tasks
Grok 4.6 (30.6%)Δ 33.1%GPT-6 Astra (63.7%)
Medical devices & IVD8 tasks
GPT-6.1 Terra (31.4%)Δ 30.9%GPT-6.1 Sol (62.3%)
Clinical operations & GCP13 tasks
Grok 4.6 (33.3%)Δ 24.1%Claude Sonnet 5.5 (57.4%)
GMP quality & manufacturing8 tasks
GPT-6.1 Terra (30.1%)Δ 29.3%Kimi K3.1 Pro (59.4%)

Where models fail

Failure modes from the open Havenor Therapeutics and Broad Institute showcases illustrating specific GxP, ICH, and statutory compliance errors in candidate deliverables.

Clinical hold

Authorizing Site Activity Under a Partial Clinical Hold (MKL-P-01-R03 / R07)

Task: IND Partial Clinical Hold Response (MKL-P-01) · Environment: Broad Institute · FDA CDER IND 167326

Task prompt

"Write the binding plan for the response: what may continue, what must stop, and how the answer to the FDA is put together."

What models did

Models followed an internal steering memo proposing to contest the hold while preparing clinical sites in parallel, advising that screening and site activation could proceed during rebuttal drafting.

What was required

Under 21 CFR 312.42(b)(1)(iv), the symptomatic cohort protocol was permitted to proceed, whereas the presymptomatic carrier protocol was placed on binding partial clinical hold. No screening, site contracting, IRB submission, or subject contact may occur under the held protocol until FDA issues written notification lifting the hold.

Why it matters

Initiating site contracting or screening under an active partial clinical hold violates 21 CFR 312.42 and exposes the sponsor and clinical investigators to regulatory enforcement action.

Submitted work

Candidate Word (`.docx`) and Excel (`.xlsx`) deliverables from the open showcases compared against baseline workbooks and expert rubric criteria.

XLSXHavenor Therapeutics · Parenteral BiologicsGPT-6.1 Sol · 8 / 15

STB-AVI401-002_24M_pull_and_update_control.xlsx

493 cells modified in the submitted workbook; the sheets below are selected from the full cell-level comparison.

Critical Rubric Check: Preserves the baseline and catches the chamber, reference-standard, and calibration boundaries, but leaves the lot-to-pack mapping and the combined regulatory timeline unresolved.
Open Full Task & Trace View →Download Candidate XLSXDownload Baseline XLSX

Cell-Level Workbook Delta (493 changed cells)

Showing sample modified cells before and after model execution:

Sheet: Control11 cell changes
CellBaseline StateCandidate Submission
C2(blank)Tomas Li baseline retained; corrections are requested in QC fields/comments and Review Actions.
Sheet: Pull Plan18 cell changes
CellBaseline StateCandidate Submission
G2(blank)HOLD / SEND BACK

What the benchmark covers

Regulators

  • US FDACDER (IND, NDA, BLA), CBER (vaccines and cellular/gene therapies), and CDRH (PMA and companion diagnostics).
  • EU EMACentralised Procedure MAAs, Type II variations, CHMP re-examinations, SmPC, and EU GMP Annex 1/16 compliance.
  • Health CanadaAbbreviated New Drug Submissions (ANDS), CPID-CE quality summaries, and comparative bioavailability / ISR packages.
  • Korea MFDSGlobal Innovative Products on Fast-Track (GIFT) designation, ethnic bridging, and local RMP / prescribing information.
  • China NMPA / CDECDE 80-day supplementary-data deficiency responses, provincial institute specification verification, and registration QMS inspections.

Products

  • mRNA vaccinesLNP formulation controls, ultracold stability, and strain-update supplements.
  • Antibody-drug conjugatesDAR and payload-linker controls, multi-indication oncology filings, and regional launch readiness.
  • Targeted small molecules / kinase inhibitorsBiomarker-selected oncology NDAs, sNDAs, Type II variations, and mutagenic impurity control (ICH M7).
  • Peptide injectablesSynthetic peptide pen-injector ANDS filings, API-to-drug-product impurity bridging, and aseptic cartridge filling.
  • IVD and medical devicesNGS companion diagnostic PMAs (analytical/clinical validation, bioinformatics pipelines) and Class II device registrations.
  • Oral fixed-dose combinationsBilayer/fixed-dose tablet formulation, biowaiver justification, multi-pH comparative dissolution, and API PSD controls.
  • Parenteral biologics and MSATSterilizing-grade filtration validation, container-closure integrity (CCIT), extractables/leachables, and ICH Q1E stability.
  • Cell and gene therapyAutologous CRISPR-edited cell therapies, AAV gene therapy BLAs, and post-approval comparability (ICH Q5E) commitments.

Documents

  • Common Technical Document (CTD)Module 2.3 Quality Overall Summary (QOS), 2.7.3/2.7.4 clinical summaries, and Module 3 (3.2.S / 3.2.P) quality leaves.
  • GxP quality recordsChange controls, OOS/OOT investigations, deviation dispositions, Warning Letter CAPA commitments, and supplier qualification.
  • Clinical and regulatory recordsProtocols, CSRs, SAPs, Investigator’s Brochures (IB), ICF version governance, central-IRB reliance, and Form FDA 1572.
  • Statutory prescribing informationUSPI, EU SmPC, Korean prescribing information, companion-diagnostic intended-use tables, and device IFU.

Jobs

  • Regulatory strategy and filingIND/NDA/BLA/MAA sequence freezes, clinical hold responses (21 CFR 312.42), expedited designations, and deficiency replies.
  • Chemistry, manufacturing, and controlsICH Q1A–Q1E stability regression, Q2 analytical validation, Q3A/B/M7 impurities, Q6A/B specifications, and PPQ / tech transfer.
  • Labeling and posology negotiationComplete-response resubmission USPI redlines, active-moiety strength expression, and companion-diagnostic claim alignment.
  • Quality management and inspection readinessFDA Warning Letter and EU GMP deficiency responses, Health Hazard Evaluations (HHE), and pre-approval inspection audits.
  • Clinical trial oversightSite activation gates, protocol-amendment re-consent audits, CTMS/coverage analysis, and GCP serious-breach dispositions.

How it works

What workspace and systems does an agent receive?

Each task places the agent in a functional biopharma seat with a dated company file system, eCTD dossier leaves, QMS/EDMS and LIMS registers, email threads, and workplace chat. The agent must produce a specific controlled deliverable (`.docx` or `.xlsx`).

How is temporal leakage prevented?

Every workspace is frozen at a specific historical cutoff date (`virtual_now`). Documents, stability pulls, and correspondence dated after the task cutoff are not part of the environment.

How are submissions graded?

Each deliverable is evaluated against a task-specific expert rubric verifying regulatory, CMC, and GxP judgment grounded in the governing source records.

What is published?

Leaderboard scores across all 24 benchmark environments, plus complete workspace files, candidate deliverables, rubric checklists, and execution traces for the two public showcase environments.

Can I run the benchmark?

Yes. Full task packages, environment containers, MCP tools, and the rubric verifier are available on Harbor and Hugging Face.

How do the Havenor Therapeutics and Broad Institute showcases relate to the 81-task benchmark?

The v0.2 leaderboard evaluates 81 tasks across 24 confidential drug and device company environments. Havenor Therapeutics (a full-scale synthetic sterile-injectables manufacturer) and the Broad Institute (reconstructed from the public IND 167326 partial clinical hold record) are companion showcase environments published in full so researchers can inspect source documents, coworker chats, deliverables, and rubric grades without confidentiality restrictions.