APEX-Agents · Law
World434_AH_03
APEX-Agents task World434_AH_03 in AI Agents for Real Estate Lease Review. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
One of the Singapore properties we're acquiring (classified PG III) has an exit staircase that services the 1 level underground parking lot, located immediately under the exit staircase that services the above ground floors. The staircases are not continuous and are separated. The audit confirmed that exit distances in the basement to ground level doors comply with the "Determination of Exit Requirement" table. However, the auditor flagged that while the separation between the basement exit staircase and the above-ground exit staircase was fire-rated, the basement exit staircase did not have a fire-rated enclosure and that this was a deficiency that would need to be rectified. The cost will be huge, as the entire building will need to be shut down and major construction completed. Is the auditor's recommendation strictly necessary under Singapore's rules, or can we avoid this cost? Give your assessment to me here as a reply.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 2/2 | Pass | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 2/2 | Pass | Share pagePublic trace |
| Gemini 3 Flash | dual | 2/2 | Pass | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 2/2 | Pass | Share pagePublic trace |
| GPT-5.4 | dual | 2/2 | Pass | Share pagePublic trace |
| GPT-5.4 mini | dual | 2/2 | Pass | Share pagePublic trace |
| GPT-5.4 nano | dual | 1/2 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States the auditor's recommendation is incorrect
PassEvidence: The response says “the auditor’s recommendation does not appear strictly necessary” and “the auditor seems to be applying the general... rule without giving effect to the express... exception.” Assessment: Criterion “States the auditor's recommendation is incorrect” is met; the response clearly says to push back and treats it as not a mandatory deficiency, albeit with caveats.
States a fire-rated enclosure for the basement exit staircase is not required
PassEvidence: The response quotes/paraphrases that “Exit staircases serving a single storey basement... are not required to be protected with fire-rated enclosures” and concludes “not a mandatory deficiency.” Assessment: Criterion “States a fire-rated enclosure for the basement exit staircase is not required” is met; the response directly states the enclosure is not required on the described facts.