Raycaster/ Eval

The first benchmark for professional work inside regulated biopharma.

Biopharma Bench is an AI agent benchmark for pharmaceutical regulatory and CMC work. We measure whether frontier agents can navigate contradictory records, identify controlling specifications, and deliver audit-ready professional work inside private biopharma companies.

Announcement & reportHugging Face datasetRun on Harbor

Company environments along the path to patients

Each environment places agents inside a company at a stage of bringing a medicine or medical test to patients. Explore the 12 confidential benchmark environments below, alongside our two open companion showcases (Havenor Therapeutics and the Broad Institute).

  1. 1Discovery

    Scientists find a promising molecule or idea.

  2. 2Lab & animal studies

    Safety is tested in cells and animals before any person gets it.

  3. 3Human trials

    Volunteers and patients receive it in carefully controlled studies.

  4. 4Filing & review

    The company sends its evidence to a health authority and answers its questions.

  5. 5Approved

    The authority says yes; the company prepares to launch.

  6. 6On the market

    Patients receive it; the company keeps watching safety and quality.

12Company environments
71Total tasks
752Scoring criteria

Benchmark results

GPT-6 Astra leads with a 70.4% mean score and fully passes 8 of 71 tasks. 4 of 12 models fully pass none.

Benchmark results for 12 evaluated models
RankModel
1Codex · high70.4%8/7111.3%542/752$4.2128.722.716.4 min
2Claude Code · high69.9%8/7111.3%529/752$5.3322.738.312.4 min
3Claude Code · high67.6%2/712.8%513/752$3.9536.742.214.9 min
4Claude Code · high67.5%4/715.6%520/752$2.7344.345.712.1 min
5Cursor CLI · high64.4%2/712.8%493/752$1.6629.243.911.0 min
6Cursor CLI · high63.0%2/712.8%480/752$1.1620.546.78.8 min
7Pi · high57.0%0/71none433/752$0.1136.240.56.3 min
8Codex · high54.3%0/71none424/752$0.6428.226.35.8 min
9Pi · max48.5%1/711.4%374/752$0.7221.925.14.9 min
10Pi · high47.6%0/71none365/752$0.0422.825.83.1 min
11Cursor CLI · high46.6%0/71none356/752$1.1275.277.214.5 min
12Codex · high44.9%1/711.4%354/752$1.7034284.9 min

Swipe sideways for pass rate, criteria, cost, and time.

Mean score: each task’s share of criteria passed, averaged over 71 tasks. Pass@1: tasks where every criterion passed in a single trial. Criteria passed is pooled across all 752 criteria. One scored trial per task, so results are subject to variance. Cost uses public API rates and excludes grading. Steps, tool calls, and time are per-task means.

Score against cost, effort, and release date

The dashed line joins models that no other model beats on both score and the chosen axis.

  • GLM · Pi ★
  • DeepSeek · Pi ★
  • 6 Sol · Codex
  • Kimi · Pi
  • Gemini · Cursor CLI
  • Grok 4.6 · Cursor CLI ★
  • Grok 4.7 · Cursor CLI ★
  • 5.6 Sol · Codex
  • Fable · Claude Code
  • Opus 5.5 · Claude Code ★
  • Opus 5 · Claude Code ★
  • Astra · Codex ★
40%50%60%70%$0.00$2.00$4.00Mean cost per task (USD)Macro task meanAstra · CodexOpus 5 · Claude CodeOpus 5.5 · Claude CodeGrok 4.7 · Cursor CLIGrok 4.6 · Cursor CLIDeepSeek · PiGLM · Pi6 Sol · CodexKimi · PiGemini · Cursor CLI5.6 Sol · Codex
Each point represents one model. Cost, steps, tool calls, and agent execution time are per-task means across the same 71 tasks.

Inside

While benchmark partner records remain sealed, we publish two open companion showcases in full: Havenor Therapeutics (synthetic) and the Broad Institute (public FDA record). We also publish Havenor’s trials for comparison on the same board, where the leaderboard covers the full suite — 71 tasks across 12 confidential real-product environments — while those showcase trials are scored separately and don’t count toward it.

10 assignments at Havenor Therapeutics, a fictional manufacturer.

  1. Task 01Prepare the next AVI-401 stability commitment updateRegulatory CMC ManagerDeepSeek V4.1 FlashGPT-6 Astra: 7 / 115 / 11
  2. Task 02State the technical position on the alternate overwrap changeHead of CMC and MSATGPT-5.6 SolGPT-6 Astra: 11 / 117 / 11
  3. Task 03Review the AVI-401 sterile-filter validation studiesPrincipal MSAT EngineerClaude Opus 5GPT-6 Astra: 6 / 95 / 9
  4. Task 04Answer the regulator’s seven questionsRegulatory CMC ManagerGPT-5.6 SolGPT-6 Astra: 12 / 1411 / 14
  5. Task 05Pre-submission records integrity checkQA Product LeadClaude Opus 5GPT-6 Astra: 8 / 95 / 9
  6. Task 06Check the container-closure validation recordsValidation EngineerClaude Opus 5GPT-6 Astra: 4 / 95 / 9
  7. Task 07Supplier and change record integrity checkSupplier Quality ManagerClaude Opus 5GPT-6 Astra: 3 / 84 / 8
  8. Task 08Confirm whether stability results meet the specification in forceQA Product LeadGPT-5.6 SolGPT-6 Astra: 5 / 63 / 6
  9. Task 09Review the 24-month stability-testing workbookQC Stability ScientistDeepSeek V4.1 FlashGPT-6 Astra: 13 / 159 / 15
  10. Task 10Decide which stability-testing operations can proceedQC Stability ScientistGPT-5.6 SolGPT-6 Astra: 12 / 157 / 15
Open the Havenor Therapeutics hub →

4 assignments from a real FDA hold.Rebuilt verbatim from the Broad Institute's public IND record: the filed dossier, the agency's hold letter, and the lab data behind them. The best model passed most checks and still authorized work FDA had frozen.

  1. Task 01Plan the response to the FDA’s partial clinical holdRegulatory operations associateClaude Opus 510 / 12
  2. Task 02Defend what the prion-protein spinal-fluid test may claimBioanalytical scientistClaude Opus 510 / 12
  3. Task 03Audit the animal-study record behind the holdNonclinical toxicologistClaude Opus 511 / 13
  4. Task 04Check the manufacturing math the filing got wrongCMC technical writerClaude Opus 512 / 13
Open the Broad Institute hub →

Results by domain

A model that leads on one kind of filing can trail on another. Filter the domains. Selecting a model on the leaderboard marks the rows where it is the high or low score.

Domain / SurfaceSpread & Range (Min → Max Macro Score)
US FDA (CDER, CBER, CDRH)42 tasks
GLM-5.3 Flash (41.5%)Δ 26.7%GPT-6 Astra (68.2%)
China NMPA / CDE Quality Files10 tasks
Gemini 3.8 Flash (42.0%)Δ 22.0%Grok 4.6 (64.0%)
Korea MFDS Fast-Track ADC Dossiers7 tasks
Gemini 3.8 Flash (15.4%)Δ 59.5%GPT-6 Astra (74.9%)
mRNA & Biologics Platforms6 tasks
DeepSeek V4.1 Flash (44.8%)Δ 37.9%GPT-6 Astra (82.7%)
IVD & Medical Device Submissions6 tasks
Gemini 3.8 Flash (33.3%)Δ 44.5%GPT-6 Astra (77.8%)
Targeted Small Molecules19 tasks
GLM-5.3 Flash (43.1%)Δ 21.4%Claude Opus 5 (64.5%)
CTD Module 2/3 Quality Leaves28 tasks
Gemini 3.8 Flash (44.1%)Δ 23.4%GPT-6 Astra (67.5%)
GxP Quality & CAPA Audits18 tasks
DeepSeek V4.1 Flash (42.0%)Δ 24.2%Grok 4.6 (66.2%)
Clinical Trial Site Ops & ISF10 tasks
GPT-5.6 Sol (56.7%)Δ 19.9%GPT-6 Astra (76.6%)
Regulatory Strategy & Deficiencies31 tasks
Kimi K3 (46.2%)Δ 20.6%Claude Opus 5 (66.8%)

Complementary domain strengths

Models show distinct strengths across regulatory regimes and document types. An overall ranking averages these domain-specific advantages together.

56/67 vs. 42/67Astra vs. Opus 5 high

Astra passes 56 of 67 items in the mRNA biologics environment; Opus 5 high passes 42. Outside it, Astra still leads, 486 to 471 of 685.

Key Insight: Astra still leads Opus 5 high inside the mRNA environment and outside it.

Where models fail

Walkthroughs from our open showcases illustrating the subtle verification errors that cause audit and regulatory failure.

Clinical hold

Held Means Held (MKL-P-01-R03 / R07)

Task: IND Partial Clinical Hold Response (MKL-P-01) · Environment: Broad Institute · FDA CDER IND 167326

Task prompt

"Write the binding plan for the response: what may continue, what must stop, and how the answer to the FDA is put together."

What models did

Models frequently followed the internal steering memo’s proposal to "argue against the hold" and prepare trial sites in parallel, advising that screening and site setup could proceed quietly while drafting the rebuttal.

Failed:Claude Opus 5, GPT-5.6 Sol, DeepSeek V4.1 Flash
What was required

Under 21 CFR 312.42(b)(1)(iv), the symptomatic study was permitted, but the healthy-carrier study was on binding partial clinical hold. No screening, no site setup, and no volunteer contact was legally permissible until FDA formally issues a written removal order.

Passed:GPT-6 Astra
Why it matters

If followed, healthy volunteers would have been enrolled while under an active federal clinical hold—a regulatory violation under 21 CFR 312.42 that subjects the presymptomatic trial to formal enforcement action.

Submitted work

Inspect candidate submissions from our open showcases and compare them with the starting files.

XLSXHavenor Therapeutics · Parenteral BiologicsDeepSeek V4.1 Flash · 9 / 15

STB-AVI401-002_24M_pull_and_update_control.xlsx

280 cells modified across 6 sheets (Control, Pull Plan, Execution Sequence, Chain and Controls, Regulatory Handoff, Review Actions).

Critical Rubric Check: Correctly identifies that AVI401-R01 is 100 mL, but fails on qualified-state basis and misses chamber requalification lapse under AVT-SOP-045.
Open Full Task & Trace View →Download Candidate XLSXDownload Baseline XLSX

Cell-Level Workbook Delta (280 changed cells)

Showing sample modified cells before and after model execution:

Sheet: Control5 cell changes
CellBaseline StateCandidate Submission
C7— (blank)0.4 reviewed; not approved. Return as 0.5 after QC-01 to QC-14 close.
C11— (blank)Pull authorization, dataset release and regulatory handoff withheld. See Review Actions QC-01 to QC-15 and the QC Disposition sheet.
Sheet: Pull Plan18 cell changes
CellBaseline StateCandidate Submission
G2— (blank)Hold - lot mapping correct, authorization withheld.
H2— (blank)AVI401-R01 is correctly listed as 100 mL. The pull is not authorized because the testing-chain clearance is not complete (QC-12) and the chamber qualified state is not established (QC-02).
Sheet: Regulatory Handoff36 cell changes
CellBaseline StateCandidate Submission
G2— (blank)Not approved - scope and timing wrong.
H2— (blank)HA-AVI401-IR-2026-014 point 7 states the 24-month pull is not part of sequence 0003 and is to be provided under COM-AVI401-008; no earlier pull is requested for the letter.

What the benchmark covers

Regulators

  • US FDACDER for small molecules and therapeutic proteins. CBER for biologics and vaccines. CDRH for devices and in-vitro diagnostics.
  • EU EMACentralised marketing authorisations, SmPC (prescribing information), and post-marketing variations.
  • Health CanadaAbbreviated New Drug Submissions (ANDS).
  • Korea MFDSGIFT designation and local risk-management plans.
  • China NMPA / CDEFixed-dose combination chemistry dossiers and local GMP inspection responses.

Products

  • mRNA vaccinesLipid nanoparticles, cold chain, and bivalent strain updates.
  • Antibody-drug conjugatesDual-indication accelerated approval and target-specific dosing.
  • Targeted small molecules / kinase inhibitorsExon-specific oncology, companion-diagnostic bridging, and synthetic API work.
  • Peptide injectablesGLP-1 pens and sterile fill-finish.
  • IVD and medical devicesPMA lifecycle and pan-tumor companion-diagnostic labeling.
  • Oral fixed-dose combinationsMulti-API process control and dissolution.
  • Parenteral biologics and MSATAseptic fill-finish, autoclave performance qualification, lyophilization, and bioburden.

Documents

  • Common Technical Document (CTD)Module 2 summaries and Module 3 quality / CMC leaves.
  • GxP quality recordsChange controls (CC), out-of-specification (OOS) investigations, deviations, and CAPA.
  • Clinical and regulatory recordsProtocols, clinical study reports (CSR), statistical analysis plans (SAP), investigator brochures (IB), informed consent (ICF), and FDA Form 1572.
  • Statutory prescribing informationUSPI, EU SmPC, Korean labeling, and instructions for use (IFU).

Jobs

  • Regulatory strategy and filingHealth-authority submissions, responses, and filing strategy.
  • Chemistry, manufacturing, and controlsProcess, specification, and manufacturing-science work.
  • Labeling and posology negotiationPrescribing information, dose language, and label negotiation.
  • Quality management and inspection readinessDeviations, CAPA, change control, and inspection-facing records.
  • Clinical trial oversightProtocol, consent, and trial-conduct documentation.

How it works

What does a model get?

A simulated company with files, email, specifications, and business systems. The model takes an employee’s role and must hand in a specific document or spreadsheet.

Can models see future records?

No. Records stop at the task date. Havenor’s records end around September 2026, so tests scheduled after that have no results yet.

How is work scored?

Each submission is checked against a list of criteria for its task. Runs where grading failed for a technical reason are left out of the averages.

What is published?

Company documents, submitted files, scores, and summaries of what the model did. Full transcripts are available in Expert Review after signing in.

Can I run the tasks here?

These pages show completed trials. To run the full benchmark, including records, LIMS, and the verifier, use Harbor.

Are Havenor Therapeutics and the Broad Institute part of the 71-task benchmark?

No — the official benchmark is 71 tasks scored across 12 confidential real-product company environments. The two companion showcases are evaluated separately: Havenor’s trials also appear on this board for comparison (they don’t count toward the 71), and the Broad Institute’s published showcase lives on its own page. Both are published in full so researchers can inspect raw files, coworker chats, and grading checklists without confidentiality restrictions.