Raycaster / evalsBack to AI Agents for Maritime and Environmental Liability

APEX-Agents · Law

World418-TK-01

1/2Fail

APEX-Agents task World418-TK-01 in AI Agents for Maritime and Environmental Liability. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Maritime and Environmental LiabilityLaw World 418Dual harnessGrader: rubric
Task IDtask_3413464bd2b14df6bd2ee15c36d3abf1
WorldLaw World 418
Expected outputmessage_in_console
Published runs7 models · dual config

Task prompt

What the agent was asked to do

Can you please assess whether or not the new LNG carries built by Nakamura Heavy Industries can be used in Jones Act Trade? Additionally, BlueLNG JV purchased another LNG vessel and asks whether it can be used in Jones Act Trade as well. It was built in 1992 in France. Please give me your reply right back here.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual1/2FailShare pagePublic trace
fireworks models Kimi K2dual1/2FailShare pagePublic trace
Gemini 3 Flashdual2/2PassShare pagePublic trace
Gemini 3.1 Produal1/2FailShare pagePublic trace
GPT-5.4dual1/2FailShare pagePublic trace
GPT-5.4 minidual2/2PassShare pagePublic trace
GPT-5.4 nanodual1/2FailShare pagePublic trace

Grading rubric

Criteria and grader verdict (showcase run)

  1. States No, the LNG carriers built by Nakamura Heavy Industries cannot be used in Jones Act Trade

    Pass

    Evidence: TEXT_RESPONSE states, “No, neither category appears usable in Jones Act trade…” and for “Nakamura newbuild LNG carriers” says “They therefore should not be used for Jones Act trade.” Assessment: The criterion requires stating that the Nakamura-built LNG carriers cannot be used in Jones Act Trade; pass because the response clearly states this.

  2. States Yes, the LNG carrier built in 1992 in France can be used in Jones Act Trade solely between the U.S. and Puerto Rico

    Fail

    Evidence: TEXT_RESPONSE states for the “1992 France-built LNG vessel” that it “fails the U.S.-build requirement” and “It also should not be used for Jones Act trade.” Assessment: The criterion requires stating Yes, the 1992 France-built LNG carrier can be used in Jones Act Trade solely between the U.S. and Puerto Rico; fail because the response states the opposite and does not identify the U.S.-Puerto Rico use.