Raycaster / evalsBack to AI Agents for FDA and Pharma Promotional Compliance

APEX-Agents · Law

World 420 LB_05

2/2Pass

APEX-Agents task World 420 LB_05 in AI Agents for FDA and Pharma Promotional Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for FDA and Pharma Promotional ComplianceLaw World 420Dual harnessGrader: rubric
Task IDtask_fa15dfc96f4b4a289fac616a6f3b4a1a
WorldLaw World 420
Expected outputmessage_in_console
Published runs7 models · dual config

Task prompt

What the agent was asked to do

For purposes of a training presentation for sales representatives giving background on what off-label promotion is and its legal status, please indicate whether it would be accurate to incorporate the following items from Livyra_handbook.pdf: (1) The definition of "misbranding" (2) The language in section 2.3 Please note that you should not consider the above items inaccurate solely based on being incomplete since additional language will be added for purposes of the training session. Also, there is no need to explain your reasoning; I just need a "would be accurate" or "would not be accurate" answer for each item. Provide your response in a message to the console. Consider the following additional sources: 1. US v Facteau.pdf 2. 21 USC 331.pdf 3. 21 USC 352.pdf

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual2/2PassShare pagePublic trace
fireworks models Kimi K2dual0/2FailShare pagePublic trace
Gemini 3 Flashdual2/2PassShare pagePublic trace
Gemini 3.1 Produal2/2PassShare pagePublic trace
GPT-5.4dual2/2PassShare pagePublic trace
GPT-5.4 minidual1/2FailShare pagePublic trace
GPT-5.4 nanodual0/2FailShare pagePublic trace

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that it would not be accurate to incorporate the definition of misbranding from the Livyra handbook

    Pass

    Evidence: TEXT_RESPONSE states “(1) The definition of ‘misbranding’: would not be accurate.” Assessment: Criterion asks whether it states that it would not be accurate to incorporate the definition of misbranding from the Livyra handbook; pass because the response directly says this.

  2. States that it would not be accurate to incorporate the language from section 2.3 of the Livyra handbook

    Pass

    Evidence: TEXT_RESPONSE states “(2) The language in section 2.3: would not be accurate.” Assessment: Criterion asks whether it states that it would not be accurate to incorporate the language from section 2.3 of the Livyra handbook; pass because the response directly says this.