APEX-Agents · Law
World 431 Task_EL_02
APEX-Agents task World 431 Task_EL_02 in AI Agents for Real Estate Lease Review. Compare dual-harness agent runs across models, scores, and public traces.
Task prompt
What the agent was asked to do
Go ahead and check the terms of the Letter of Intent ("LOI") and the final draft of the lease to see if there are any inconsistencies between the two documents. If there are, please list each inconsistency. Put all of your findings in a new document that you make, and get it back to me. Please highlight the main differences, if there are any.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.4 mini | dual | 4/5 | Fail | Run detailsPublic trace |
| GPT-5.5 | dual | 4/5 | Fail | Run detailsPublic trace |
| fireworks models Kimi K2 | dual | 2/5 | Fail | Run detailsPublic trace |
| Gemini 3 Flash | dual | 2/5 | Fail | Run detailsPublic trace |
| Gemini 3.1 Pro | dual | 2/5 | Fail | Run detailsPublic trace |
| GPT-5.4 | dual | 2/5 | Fail | Run detailsPublic trace |
| GPT-5.4 nano | dual | 2/5 | Fail | Run detailsPublic trace |
Grading rubric
Rubric criteria
Runs are graded against these criteria. Open a run for model-specific verdicts.
States both of the following: (1) the lease does not contain a personal guarantee clause, and (2) the LOI does contain a personal guarantee clause
States both of the following: (1) the lease does not contain a consideration clause, and (2) the LOI does contain a consideration clause
States both of the following: (1) the lease does not contain an exclusivity clause, and (2) the LOI does contain an exclusivity clause
States both of the following: (1) the lease does not contain a confidentiality clause, and (2) the LOI contains a confidentiality clause
States both of the following: (1) the lease contains a notice period for lease renewal of at least 9 months before the expiration of the current lease term, and (2) the LOI contains a notice period for lease renewal of at least 90 days before the expiration of the current lease term