APEX-Agents · Law
World 419_UM_02
APEX-Agents task World 419_UM_02 in AI Agents for Maritime and Environmental Liability. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
I need your help determining how much will be covered by MARINEX under our current insurance policies, between the cleanup costs incurred after the incident and Jack Sparrow's claim. Please see if Captain Dee's imposed penalty of $128,375 will also be covered under both our policies. Reply back with your findings. Thank you!
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 0/2 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 0/2 | Fail | Share pagePublic trace |
| Gemini 3 Flash | dual | 2/2 | Pass | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 0/2 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 0/2 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 0/2 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 1/2 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that Yes, the imposed penalty of $128,375 will be covered under the policies
FailEvidence: The response states, “Captain Dee’s $128,375 penalty: not covered under both policies” and “It is potentially covered under P&I... It is not covered under H&M.” Assessment: The criterion requires stating that yes, the imposed penalty of $128,375 will be covered under the policies. The response does not state this; it says not under both and only potentially under one policy. Fail.
States that the total amount payable by MARINEX to Cooper/Jeffries is payable under both of these policies: (1) the Protection and Indemnity Insurance Policy and (2) the Marine Pollution Legal Liability Insurance Policy
FailEvidence: The response states, “I did not find a policy or underwriter named MARINEX” and “I treated the two final current policies as... Hull & Machinery and Protection & Indemnity,” with cleanup “covered under the P&I policy, not the H&M policy.” Assessment: The criterion requires stating that the total amount payable by MARINEX is payable under both the Protection and Indemnity Insurance Policy and the Marine Pollution Legal Liability Insurance Policy. The response does not mention Marine Pollution Legal Liability as an applicable policy and instead says only P&I applies, not H&M. Fail.