APEX-Agents · Law
Law 433_AC_02
APEX-Agents task Law 433_AC_02 in AI Agents for Healthcare and Senior Living Legal Risk. Compare dual-harness agent runs across models, scores, and public traces.
Task prompt
What the agent was asked to do
Please review Grove’s 2023 and 2025 operations manuals. Let me know if there are any changes in the 2025 manual that may present issues for Grove’s regulatory compliance. For any regulatory issues identified, tell me what changed in the 2025 manual and why it presents an issue. Reply back just with one or two sentences please.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.4 mini | dual | 5/10 | Fail | Run detailsPublic trace |
| GPT-5.5 | dual | 4/10 | Fail | Run detailsPublic trace |
| fireworks models Kimi K2 | dual | 3/10 | Fail | Run detailsPublic trace |
| GPT-5.4 nano | dual | 3/10 | Fail | Run detailsPublic trace |
| Gemini 3 Flash | dual | 2/10 | Fail | Run detailsPublic trace |
| GPT-5.4 | dual | 2/10 | Fail | Run detailsPublic trace |
| Gemini 3.1 Pro | dual | 0/10 | Fail | Run detailsPublic trace |
Grading rubric
Rubric criteria
Runs are graded against these criteria. Open a run for model-specific verdicts.
States that the Resident Rights and Dignity section was removed from the 2025 manual
States that the 2025 manual may violate the regulatory requirement for Grove to advise residents of their rights
States that the 2025 manual only requires a registered nurse to be on-call at night
States that the 2025 manual may violate the regulatory requirement for Grove to staff registered nurses on-site to conduct overnight duties
States that the 2025 manual removed dietician approval for meals offered to residents
States that the 2025 manual may violate the regulatory requirement for Grove to maintain quality control over dietary services
States that the 2025 manual allows for non-urgent complaints to be acknowledged within 14 days
States that the 2025 manual may violate the regulatory requirement for Grove to respond promptly to complaints
States that the Emergency Preparedness section was removed in the 2025 manual
States that 2025 manual may violate the regulatory requirement for Grove to develop and maintain a written emergency preparedness and response plan