APEX-Agents · Law
LawWorld433_NAF_02
APEX-Agents task LawWorld433_NAF_02 in AI Agents for Healthcare and Senior Living Legal Risk. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
The attorney for Thomas Whitaker wants to submit this matter for arbitration in Texas. Reply to me in here, drafting an output explaining whether that would be consistent with Mr. Stoker's Litigation Plan and the Residency Agreement for Thomas Whitaker.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 1/3 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 0/3 | Fail | Share pagePublic trace |
| Gemini 3 Flash | dual | 1/3 | Fail | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 2/3 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 0/3 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 2/3 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 2/3 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that going to arbitration with the Whitakers is consistent with Mr. Stoker's Litigation Plan
FailEvidence: The response says arbitration is “only conditionally consistent” and that “Mr. Stoker’s Litigation Plan does **not** identify arbitration as the recommended path,” instead recommending “direct negotiations or mediation.” Assessment: For criterion “States that going to arbitration with the Whitakers is consistent with Mr. Stoker's Litigation Plan,” this does not clearly state consistency; it emphasizes arbitration is not the Plan’s recommended path. Fail.
States that the Residency Agreement requires mandatory arbitration in Los Angeles, California
PassEvidence: The response states, “The Whitaker agreement in the litigation file contains a mandatory arbitration clause, but it specifies that arbitration is to occur in **Los Angeles, California**,” and quotes “The arbitration shall be conducted in Los Angeles, California.” Assessment: This directly satisfies “States that the Residency Agreement requires mandatory arbitration in Los Angeles, California.” Pass.
States that participating in arbitration in Texas is not consistent with the Residency Agreement
FailEvidence: The response says Texas arbitration would be consistent “only if Magnolia also agrees,” but also says a “separate Texas-specific Whitaker agreement” provides arbitration in “Dallas, Texas” and concludes Texas arbitration is “contractually consistent” if that version is operative. Assessment: For criterion “States that participating in arbitration in Texas is not consistent with the Residency Agreement,” the response does not clearly state Texas arbitration is not consistent; it qualifies and identifies scenarios where it would be consistent. Fail.