Raycaster / evalsBack to AI Agents for Maritime and Environmental Liability

APEX-Agents · Law

World 419_UM_01

3/10Fail

APEX-Agents task World 419_UM_01 in AI Agents for Maritime and Environmental Liability. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Maritime and Environmental LiabilityLaw World 419Dual harnessGrader: rubric
task_ef71da92049944b4b9985c776b72d3bf
Law World 419
message_in_console
7 models · dual config

Task prompt

What the agent was asked to do

We are faced with a claim from one Jack Sparrow. He alleges that his fishing boat and equipment were damaged by the M/V Red Room's pollution incident and is demanding compensation from Cooper/Jeffries. Can you go over our insurance policies and see if such a claim is covered under any of our policies? If it is covered, state what sections would be relevant for the claim. Reply to me here with all of the info I requested.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual3/10Fail
fireworks models Kimi K2dual0/10Fail
Gemini 3 Flashdual2/10Fail
Gemini 3.1 Produal0/10Fail
GPT-5.4dual0/10Fail
GPT-5.4 minidual6/10Fail
GPT-5.4 nanodual2/10Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that the Marine Excess Liability Insurance Policy will cover Jack Sparrow's claim

    Fail

    Evidence: The response says “Marine Excess / Bumbershoot — Provides excess coverage over underlying liability...” but also qualifies BLPL policies as “not ‘C/J’s’ direct coverage unless C/J is an additional insured/indemnitee” and “not the primary policies for a claim made directly against C/J.” Assessment: For “States that the Marine Excess Liability Insurance Policy will cover Jack Sparrow's claim,” fail because the response only identifies it as a possible tender/recovery or excess source, not that it will cover Jack’s claim.

  2. States that the Marine Pollution Legal Liability Insurance Policy will cover Jack Sparrow's claim

    Pass

    Evidence: The response lists “Marine Pollution Legal Liability policy” among “strongest potentially responsive policies” and says it is “responsive if C/J or a relevant liable party is the assured” and has “Very strong wording for Jack’s alleged boat/equipment damage.” Assessment: For “States that the Marine Pollution Legal Liability Insurance Policy will cover Jack Sparrow's claim,” pass; the coverage statement is clear, though appropriately qualified on applicability/binding.

  3. States that the Marine General Liability Insurance Policy will not cover Jack Sparrow's claim

    Pass

    Evidence: Under “Marine General Liability,” the response says the form has a pollution exclusion and concludes: “Not a primary coverage source for Jack’s pollution claim.” Assessment: For “States that the Marine General Liability Insurance Policy will not cover Jack Sparrow's claim,” pass.

  4. States that Article I of the Marine Excess Liability Insurance Policy is relevant to Jack Sparrow's claim

    Fail

    Evidence: The Marine Excess discussion states “Marine Excess / Bumbershoot” covers “Third-party property damage,” “Pollution liability,” and OPA 90 liability, but it does not mention “Article I.” Assessment: For “States that Article I of the Marine Excess Liability Insurance Policy is relevant,” fail because the required article is not identified.

  5. States that Article IV of the Marine Excess Liability Insurance Policy is relevant to Jack Sparrow's claim

    Fail

    Evidence: The response’s Marine Excess section does not identify Article IV; it only gives general bullet points about excess coverage and a pollution-exclusion carveback. Assessment: For “States that Article IV of the Marine Excess Liability Insurance Policy is relevant,” fail because Article IV is not stated as relevant.

  6. States that Article VIII of the Marine Excess Liability Insurance Policy is relevant to Jack Sparrow's claim

    Fail

    Evidence: The response’s Marine Excess section contains no reference to “Article VIII.” Assessment: For “States that Article VIII of the Marine Excess Liability Insurance Policy is relevant,” fail because the article is not mentioned.

  7. States that Article XV of the Marine Excess Liability Insurance Policy is relevant to Jack Sparrow's claim

    Fail

    Evidence: The response’s Marine Excess section contains no reference to “Article XV.” Assessment: For “States that Article XV of the Marine Excess Liability Insurance Policy is relevant,” fail because the article is not mentioned.

  8. States that Article I of the Marine Pollution Legal Liability Insurance Policy is relevant to Jack Sparrow's claim

    Fail

    Evidence: Under “Marine Pollution Legal Liability policy,” the response lists “Article V — Coverage Section D,” plus “Defense” and “Claims-made/reporting,” but does not mention “Article I.” Assessment: For “States that Article I of the Marine Pollution Legal Liability Insurance Policy is relevant,” fail because Article I is not identified.

  9. States that Article II of the Marine Pollution Legal Liability Insurance Policy is relevant to Jack Sparrow's claim

    Fail

    Evidence: Under “Marine Pollution Legal Liability policy,” the response lists “Article V — Coverage Section D,” plus “Defense” and “Claims-made/reporting,” but does not mention “Article II.” Assessment: For “States that Article II of the Marine Pollution Legal Liability Insurance Policy is relevant,” fail because Article II is not identified.

  10. States that Article V of the Marine Pollution Legal Liability Insurance Policy is relevant to Jack Sparrow's claim

    Pass

    Evidence: The response explicitly lists “Article V — Coverage Section D, Third Party Damages” for the Marine Pollution Legal Liability policy and explains it covers damage to “vessels, boats, and marine equipment” and “fishing gear.” Assessment: For “States that Article V of the Marine Pollution Legal Liability Insurance Policy is relevant,” pass.