APEX-Agents · Law
World434_IG_02
APEX-Agents task World434_IG_02 in AI Agents for Real Estate Lease Review. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
One of the Hong Kong facilities is going to overhaul its fire safety after they found that some of the equipment from the supplier produced faulty sprinklers. This same supplier produced essentially all of the equipment that can't be used. The facility needs 50 fire doors to be installed by code. Most doors are very expensive, but we have identified three in our price point. Assuming the dimensions are proper, please state the cheapest one that we can purchase and use. Explain your reasoning. Reply to me here concisely.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 3/4 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 3/4 | Fail | Share pagePublic trace |
| Gemini 3 Flash | dual | 4/4 | Pass | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 3/4 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 3/4 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 1/4 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 0/4 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States the AegisCore door cannot be sourced for the Hong Kong facilities
PassEvidence: TEXT_RESPONSE says, “I would exclude **AegisCore** despite its USD 640 price because delivery is listed only for Europe/U.S./Australia/Canada/NZ, not Hong Kong.” Assessment: This satisfies “States the AegisCore door cannot be sourced for the Hong Kong facilities,” because the response explicitly says it is not deliverable to Hong Kong.
States the SentinelGuard does cannot be sourced for the Hong Kong facilities
FailEvidence: TEXT_RESPONSE says, “I would exclude **SentinelGuard** because its lock can maintain its last commanded state during an outage, which is inconsistent with the HK exit-door rule.” Assessment: This does not satisfy “States the SentinelGuard ... cannot be sourced for the Hong Kong facilities”; the response says SentinelGuard should be excluded for regulatory/lock-compliance reasons, not that it cannot be sourced for Hong Kong.
States that the Fireline Veritas doors comply with Hong Kong fire safety regulations
PassEvidence: TEXT_RESPONSE says, “Fireline meets this, including automatic unlocking on power/control failure or fire alarm” and identifies “Hong Kong materials require 60-minute fire-rated stairwell doors.” Assessment: This satisfies “States that the Fireline Veritas doors comply with Hong Kong fire safety regulations,” because it explicitly asserts Fireline meets the relevant HK requirements.
States the Fireline Veritas doors are the cheapest doors which should be sourced for the Hong Kong facilities
PassEvidence: TEXT_RESPONSE opens with “Cheapest usable choice: **Fireline Veritas FR60 Escape-Safe Doorset** at **USD 690/unit**” and excludes AegisCore and SentinelGuard. Assessment: This satisfies “States the Fireline Veritas doors are the cheapest doors which should be sourced for the Hong Kong facilities.”