Raycaster / evalsBack to AI Agents for SEC Disclosure Analysis

APEX-Agents · Law

Task iv36a08a

1/7Fail

APEX-Agents task Task iv36a08a in AI Agents for SEC Disclosure Analysis. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for SEC Disclosure AnalysisLaw World 415Dual harnessGrader: rubric
Task IDtask_7fe79f6934f64f159f8a6c27901e3db8
WorldLaw World 415
Expected outputmessage_in_console
Published runs6 models · dual config

Task prompt

What the agent was asked to do

Analyze whether Counts 1-7 of Delta's Complaint against CrowdStrike fall within the limitation of liability clause in Section 10.1 of CrowdStrike's standard MSA, indicating "Covered" or "Not Covered" for each count. Reply back to me here with your assessments.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual1/7FailShare pagePublic trace
fireworks models Kimi K2dual3/7FailShare pagePublic trace
Gemini 3.1 Produal6/7FailShare pagePublic trace
GPT-5.4dual1/7FailShare pagePublic trace
GPT-5.4 minidual1/7FailShare pagePublic trace
GPT-5.4 nanodual1/7FailShare pagePublic trace

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that Count 1 is Not Covered

    Fail

    Evidence: The response table lists Count I / Computer Trespass with Assessment: “Covered,” and bottom line says “Counts 1–7 are all Covered.” Assessment: Criterion requires “States that Count 1 is Not Covered”; fail because it states the opposite.

  2. States that Count 2 is Not Covered

    Fail

    Evidence: The response table lists Count II / Trespass to Personalty with Assessment: “Covered,” and bottom line says “Counts 1–7 are all Covered.” Assessment: Criterion requires “States that Count 2 is Not Covered”; fail because it states the opposite.

  3. States that Count 3 is Not Covered

    Fail

    Evidence: The response table lists Count III / Breach of Contract with Assessment: “Covered,” and bottom line says “Counts 1–7 are all Covered.” Assessment: Criterion requires “States that Count 3 is Not Covered”; fail because it states the opposite.

  4. States that Count 4 is Not Covered

    Fail

    Evidence: The response table lists Count IV / Intentional Misrepresentation / Fraud by Omission with Assessment: “Covered,” while noting only a caveat about enforceability. Assessment: Criterion requires “States that Count 4 is Not Covered”; fail because the assessment is Covered.

  5. States that Count 5 is Covered

    Pass

    Evidence: The response table lists Count V / Strict Liability: Product Defect with Assessment: “Covered,” and bottom line says “Counts 1–7 are all Covered.” Assessment: Criterion requires “States that Count 5 is Covered”; pass.

  6. States that Count 6 is Not Covered

    Fail

    Evidence: The response table lists Count VI / Gross Negligence with Assessment: “Covered,” while noting it may be challenged on enforceability. Assessment: Criterion requires “States that Count 6 is Not Covered”; fail because it states Covered.

  7. States that Count 7 is Not Covered

    Fail

    Evidence: The response table lists Count VII / Deceptive and Unfair Business Practices with Assessment: “Covered,” and bottom line says “Counts 1–7 are all Covered.” Assessment: Criterion requires “States that Count 7 is Not Covered”; fail because it states the opposite.