Raycaster / evalsBack to AI Agents for Maritime and Environmental Liability

APEX-Agents · Law

World419_AH_01

0/6Fail

APEX-Agents task World419_AH_01 in AI Agents for Maritime and Environmental Liability. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Maritime and Environmental LiabilityLaw World 419Dual harnessGrader: rubric
task_f948569c92d1495eb7cbaf75570f65c8
Law World 419
message_in_console
7 models · dual config

Task prompt

What the agent was asked to do

I don't have a copy of our policy for Marine Pollution Legal Liability insurance, but it is identical to the draft terms we have in our files. Our insurer assured us that the 180,000 gallon crude oil discharge from the M/V Red Room wreck was covered under Article II Section 5, but we haven't asked them about the 1,4-dioxane detergent leak. Immediately after the shipwreck, crew discovered that a 1,000 gallon container on deck had sprung a leak. The crew had put a patch on the leak immediately after discovering it. The crew and emergency staff that were sent to assist focused their full efforts on addressing the crude oil leak. After the oil leak was contained, 4 days after the wreck, the crew went back to re-inspect the container and discovered it was almost empty, as the patch had failed. Will the same section of the insurance policy also cover liability for the detergent discharge? Reply to me with your answer right here. Give me a yes, no, likely yes, or likely no.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual0/6Fail
fireworks models Kimi K2dual1/6Fail
Gemini 3 Flashdual0/6Fail
Gemini 3.1 Produal0/6Fail
GPT-5.4dual0/6Fail
GPT-5.4 minidual1/6Fail
GPT-5.4 nanodual0/6Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States "No"

    Fail

    Evidence: TEXT_RESPONSE states “**Likely no.**” and “So my answer is: **likely no**.” Assessment: Criterion is “States "No".” The response gives a qualified “likely no,” not an unqualified “No”; fail.

  2. States that the Material Safety Data Sheet for 1,4-dioxane indicates that it is not a marine pollutant

    Fail

    Evidence: TEXT_RESPONSE discusses policy language and exclusions, but does not mention any Material Safety Data Sheet or whether 1,4-dioxane is a marine pollutant. Assessment: Criterion requires stating the MSDS indicates it is not a marine pollutant; fail.

  3. States that the Material Safety Data Sheet for 1,4-dioxane indicates that it is a hazardous substance

    Fail

    Evidence: TEXT_RESPONSE says the policy defines pollutants to include “hazardous substances… chemicals, and waste materials,” but does not say the Material Safety Data Sheet for 1,4-dioxane indicates it is a hazardous substance. Assessment: Criterion specifically requires an MSDS-based statement; fail.

  4. States that the Marine Pollution Legal Liability Insurance Policy will cover any liability for the initial leak of 1,4-dioxane under Article II, § 2

    Fail

    Evidence: TEXT_RESPONSE analyzes Article II Section 5 and the gradual pollution exclusion; it does not mention Article II, § 2 or coverage for the initial leak. Assessment: Criterion requires stating coverage for initial 1,4-dioxane leak under Article II, § 2; fail.

  5. States that the Marine Pollution Legal Liability Insurance Policy will cover reasonable costs incurred in patching the container under Article IV, § 3b

    Fail

    Evidence: TEXT_RESPONSE does not mention Article IV, § 3b or reasonable costs incurred in patching the container. Assessment: Criterion requires stating such costs are covered under Article IV, § 3b; fail.

  6. States that the coverage for wreck-related pollution liability found in Article II, Section 5 is not relevant to a leak of 1,4-dioxane

    Fail

    Evidence: TEXT_RESPONSE says “Article II Section 5 is broad enough on its face” and covers pollution arising from a wreck, then relies on an exclusion. Assessment: Criterion requires stating Article II, Section 5 is not relevant to the 1,4-dioxane leak; the response does not state this and suggests the opposite; fail.