Raycaster / evalsBack to AI Agents for Maritime and Environmental Liability

APEX-Agents · Law

World 419_UM_02

0/2Fail

APEX-Agents task World 419_UM_02 in AI Agents for Maritime and Environmental Liability. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Maritime and Environmental LiabilityLaw World 419Dual harnessGrader: rubric
task_ee25ae5557074bbaac24113f0412275d
Law World 419
message_in_console
7 models · dual config

Task prompt

What the agent was asked to do

I need your help determining how much will be covered by MARINEX under our current insurance policies, between the cleanup costs incurred after the incident and Jack Sparrow's claim. Please see if Captain Dee's imposed penalty of $128,375 will also be covered under both our policies. Reply back with your findings. Thank you!

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual0/2Fail
fireworks models Kimi K2dual0/2Fail
Gemini 3 Flashdual2/2Pass
Gemini 3.1 Produal0/2Fail
GPT-5.4dual0/2Fail
GPT-5.4 minidual0/2Fail
GPT-5.4 nanodual1/2Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that Yes, the imposed penalty of $128,375 will be covered under the policies

    Fail

    Evidence: The response states, “Captain Dee’s $128,375 penalty: not covered under both policies” and “It is potentially covered under P&I... It is not covered under H&M.” Assessment: The criterion requires stating that yes, the imposed penalty of $128,375 will be covered under the policies. The response does not state this; it says not under both and only potentially under one policy. Fail.

  2. States that the total amount payable by MARINEX to Cooper/Jeffries is payable under both of these policies: (1) the Protection and Indemnity Insurance Policy and (2) the Marine Pollution Legal Liability Insurance Policy

    Fail

    Evidence: The response states, “I did not find a policy or underwriter named MARINEX” and “I treated the two final current policies as... Hull & Machinery and Protection & Indemnity,” with cleanup “covered under the P&I policy, not the H&M policy.” Assessment: The criterion requires stating that the total amount payable by MARINEX is payable under both the Protection and Indemnity Insurance Policy and the Marine Pollution Legal Liability Insurance Policy. The response does not mention Marine Pollution Legal Liability as an applicable policy and instead says only P&I applies, not H&M. Fail.