APEX-Agents · Law
LawWorld433_ANB_04
APEX-Agents task LawWorld433_ANB_04 in AI Agents for Healthcare and Senior Living Legal Risk. Compare dual-harness agent runs across models, scores, and public traces.
Task prompt
What the agent was asked to do
Magnolia Gardens is looking to renew its 2025 coverage through Valiant Assurance Ltd. for 2026, in Texas. Valiant has asked for a list including the number of patients they serve that utilize Medicaid. Does Magnolia Gardens have to provide this information? State your view as a Yes or No, with a two-sentence explanation. Put it in a NEW documents document for me.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5 | dual | 3/3 | Pass | Run detailsPublic trace |
| fireworks models Kimi K2 | dual | 2/3 | Fail | Run detailsPublic trace |
| Gemini 3.1 Pro | dual | 2/3 | Fail | Run detailsPublic trace |
| GPT-5.4 nano | dual | 2/3 | Fail | Run detailsPublic trace |
| GPT-5.4 mini | dual | 1/3 | Fail | Run detailsPublic trace |
| GPT-5.4 | dual | 0/3 | Fail | Run detailsPublic trace |
Grading rubric
Rubric criteria
Runs are graded against these criteria. Open a run for model-specific verdicts.
States that for purposes of writing professional liability insurance for healthcare providers, an insurance company may not consider the extent to which a health care provider provides services to individuals who are recipients of Medicaid
States that for purposes of Title 10 §1901 of the Texas Insurance Code, an assisted living facility is considered a medical provider
States “No”, Magnolia Gardens does not have to provide documentation on its number of Medicaid patients to its insurer