APEX-Agents · Law
World 419_UM_01
APEX-Agents task World 419_UM_01 in AI Agents for Maritime and Environmental Liability. Compare dual-harness agent runs across models, scores, and public traces.
Task prompt
What the agent was asked to do
We are faced with a claim from one Jack Sparrow. He alleges that his fishing boat and equipment were damaged by the M/V Red Room's pollution incident and is demanding compensation from Cooper/Jeffries. Can you go over our insurance policies and see if such a claim is covered under any of our policies? If it is covered, state what sections would be relevant for the claim. Reply to me here with all of the info I requested.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.4 mini | dual | 6/10 | Fail | Run detailsPublic trace |
| GPT-5.5 | dual | 3/10 | Fail | Run detailsPublic trace |
| Gemini 3 Flash | dual | 2/10 | Fail | Run detailsPublic trace |
| GPT-5.4 nano | dual | 2/10 | Fail | Run detailsPublic trace |
| fireworks models Kimi K2 | dual | 0/10 | Fail | Run detailsPublic trace |
| Gemini 3.1 Pro | dual | 0/10 | Fail | Run detailsPublic trace |
| GPT-5.4 | dual | 0/10 | Fail | Run detailsPublic trace |
Grading rubric
Rubric criteria
Runs are graded against these criteria. Open a run for model-specific verdicts.
States that the Marine Excess Liability Insurance Policy will cover Jack Sparrow's claim
States that the Marine Pollution Legal Liability Insurance Policy will cover Jack Sparrow's claim
States that the Marine General Liability Insurance Policy will not cover Jack Sparrow's claim
States that Article I of the Marine Excess Liability Insurance Policy is relevant to Jack Sparrow's claim
States that Article IV of the Marine Excess Liability Insurance Policy is relevant to Jack Sparrow's claim
States that Article VIII of the Marine Excess Liability Insurance Policy is relevant to Jack Sparrow's claim
States that Article XV of the Marine Excess Liability Insurance Policy is relevant to Jack Sparrow's claim
States that Article I of the Marine Pollution Legal Liability Insurance Policy is relevant to Jack Sparrow's claim
States that Article II of the Marine Pollution Legal Liability Insurance Policy is relevant to Jack Sparrow's claim
States that Article V of the Marine Pollution Legal Liability Insurance Policy is relevant to Jack Sparrow's claim