Raycaster / evalsBack to AI Agents for Maritime and Environmental Liability

APEX-Agents · Law

World419_el_01

6/10Fail

APEX-Agents task World419_el_01 in AI Agents for Maritime and Environmental Liability. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Maritime and Environmental LiabilityLaw World 419Dual harnessGrader: rubric
Task IDtask_08b37d183c62457da31a347b40b767a8
WorldLaw World 419
Expected outputmessage_in_console
Published runs7 models · dual config

Task prompt

What the agent was asked to do

Review Articles 2-6 and 9-13 of the Charter Party Agreement and let me know which ones could be used to create joint liability with or shift liability to the Owner for any oil spills? Provide a yes/no assessment for each Article. Print your response back here.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual6/10FailShare pagePublic trace
fireworks models Kimi K2dual8/10FailShare pagePublic trace
Gemini 3 Flashdual8/10FailShare pagePublic trace
Gemini 3.1 Produal8/10FailShare pagePublic trace
GPT-5.4dual6/10FailShare pagePublic trace
GPT-5.4 minidual8/10FailShare pagePublic trace
GPT-5.4 nanodual8/10FailShare pagePublic trace

Grading rubric

Criteria and grader verdict (showcase run)

  1. States yes for Article 2

    Fail

    Evidence: TEXT_RESPONSE table states “Article 2 — Safe Berthing and Shifting | **No**.” Assessment: Criterion requires “States yes for Article 2”; fail because response states No.

  2. States no for Article 3

    Pass

    Evidence: TEXT_RESPONSE table states “Article 3 — Cargo Details | **No**.” Assessment: Criterion requires “States no for Article 3”; pass.

  3. States no for Article 4

    Pass

    Evidence: TEXT_RESPONSE table states “Article 4 — Freight and Payment | **No**.” Assessment: Criterion requires “States no for Article 4”; pass.

  4. States no for Article 5

    Pass

    Evidence: TEXT_RESPONSE table states “Article 5 — Laytime and Demurrage | **No**.” Assessment: Criterion requires “States no for Article 5”; pass.

  5. States yes for Article 6

    Pass

    Evidence: TEXT_RESPONSE table states “Article 6 — Owner’s Obligations and Warranties | **Yes**.” Assessment: Criterion requires “States yes for Article 6”; pass.

  6. States yes for Article 9

    Pass

    Evidence: TEXT_RESPONSE table states “Article 9 — Pollution and Environmental Compliance | **Yes**.” Assessment: Criterion requires “States yes for Article 9”; pass.

  7. States no for Article 10

    Fail

    Evidence: TEXT_RESPONSE table states “Article 10 — General Exceptions | **Yes**” and bottom line includes Article 10 among likely supports. Assessment: Criterion requires “States no for Article 10”; fail because response states Yes.

  8. States no for Article 11

    Fail

    Evidence: TEXT_RESPONSE table states “Article 11 — Indemnities | **Yes**” and calls it “Strongest direct contractual basis.” Assessment: Criterion requires “States no for Article 11”; fail because response states Yes.

  9. States no for Article 12

    Fail

    Evidence: TEXT_RESPONSE table states “Article 12 — Lien and Cesser | **Yes**.” Assessment: Criterion requires “States no for Article 12”; fail because response states Yes.

  10. States no for Article 13

    Pass

    Evidence: TEXT_RESPONSE table states “Article 13 — Dispute Resolution | **No**.” Assessment: Criterion requires “States no for Article 13”; pass.