APEX-Agents · Law

World431_AVK_03

0/2Fail

APEX-Agents task World431_AVK_03 in AI Agents for Real Estate Lease Review. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Real Estate Lease ReviewLaw World 431Dual harnessGrader: rubric

Task IDtask_ccd303088b344ea8b81c13dc04309142

WorldLaw World 431

Expected outputmessage_in_console

Published runs7 models · dual config

Task prompt

What the agent was asked to do

Review the attached documents to determine the amounts to be distributed from AI Automation Group, LLC ("AIAG") to Shohei Yamamoto and Janet Swift. Get me your reply right back here.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

Model	Harness	Score	Result	Links
GPT-5.5showcase	dual	0/2	Fail	Share page Public trace
fireworks models Kimi K2	dual	0/2	Fail	Share page Public trace
Gemini 3 Flash	dual	2/2	Pass	Share page Public trace
Gemini 3.1 Pro	dual	2/2	Pass	Share page Public trace
GPT-5.4	dual	2/2	Pass	Share page Public trace
GPT-5.4 mini	dual	2/2	Pass	Share page Public trace
GPT-5.4 nano	dual	2/2	Pass	Share page Public trace

Grading rubric

Criteria and grader verdict (showcase run)

States that the amount to be distributed to Shohei Yamamoto is $20,000
Fail
Evidence: The response’s main table states “Shohei Yamamoto | 50,000 | 25.0% | $10,000” and says “Shohei: 50,000 × $0.20 = $10,000.” It only adds conditionally, “if AIAG does not give effect ... Shohei = $20,000.” Assessment: The criterion requires stating the amount to be distributed to Shohei is $20,000. The response presents $10,000 as the distribution and $20,000 only as a caveat/alternative, so it fails.
States that the amount to be distributed to Janet Swift is $5,000
Fail
Evidence: The response’s main table states “Janet Swift | 75,000 | 37.5% | $15,000” and says “Janet: 75,000 × $0.20 = $15,000.” It only adds conditionally, “if AIAG does not give effect ... Janet = $5,000.” Assessment: The criterion requires stating the amount to be distributed to Janet is $5,000. The response presents $15,000 as the distribution and $5,000 only as a caveat/alternative, so it fails.