APEX-Agents · Law

Law 433_AC_02

4/10Fail

APEX-Agents task Law 433_AC_02 in AI Agents for Healthcare and Senior Living Legal Risk. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Healthcare and Senior Living Legal RiskLaw World 433Dual harnessGrader: rubric

Task IDtask_1d7f4e0be1274b069ad86be4ed813c46

WorldLaw World 433

Expected outputmessage_in_console

Published runs7 models · dual config

Task prompt

What the agent was asked to do

Please review Grove’s 2023 and 2025 operations manuals. Let me know if there are any changes in the 2025 manual that may present issues for Grove’s regulatory compliance. For any regulatory issues identified, tell me what changed in the 2025 manual and why it presents an issue. Reply back just with one or two sentences please.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

Model	Harness	Score	Result	Links
GPT-5.5showcase	dual	4/10	Fail	Share page Public trace
fireworks models Kimi K2	dual	3/10	Fail	Share page Public trace
Gemini 3 Flash	dual	2/10	Fail	Share page Public trace
Gemini 3.1 Pro	dual	0/10	Fail	Share page Public trace
GPT-5.4	dual	2/10	Fail	Share page Public trace
GPT-5.4 mini	dual	5/10	Fail	Share page Public trace
GPT-5.4 nano	dual	3/10	Fail	Share page Public trace

Grading rubric

Criteria and grader verdict (showcase run)

States that the Resident Rights and Dignity section was removed from the 2025 manual
Fail
Evidence: The response lists changes including “staffing, reassessment, meal, complaint, and emergency-preparedness safeguards” and specific items, but does not mention Resident Rights and Dignity. Assessment: Fails criterion “States that the Resident Rights and Dignity section was removed from the 2025 manual” because that removal is not stated.
States that the 2025 manual may violate the regulatory requirement for Grove to advise residents of their rights
Fail
Evidence: The response says compliance risk relates to “staffing based on resident service-plan needs,” meals/dietitian, complaints, and emergency preparedness. It does not discuss advising residents of their rights. Assessment: Fails criterion “States that the 2025 manual may violate the regulatory requirement for Grove to advise residents of their rights.”
States that the 2025 manual only requires a registered nurse to be on-call at night
Fail
Evidence: The response mentions staffing variances and counting volunteers/interns/family visitors toward coverage, but contains no statement that “only” a registered nurse is on-call at night. Assessment: Fails criterion “States that the 2025 manual only requires a registered nurse to be on-call at night.”
States that the 2025 manual may violate the regulatory requirement for Grove to staff registered nurses on-site to conduct overnight duties
Fail
Evidence: The response says there may be staffing compliance risk because rules require “staffing based on resident service-plan needs,” but it does not mention registered nurses being required on-site for overnight duties. Assessment: Fails criterion “States that the 2025 manual may violate the regulatory requirement for Grove to staff registered nurses on-site to conduct overnight duties.”
States that the 2025 manual removed dietician approval for meals offered to residents
Pass
Evidence: The response states the 2025 manual makes “dietitian oversight no longer required.” Assessment: Passes criterion “States that the 2025 manual removed dietician approval for meals offered to residents,” because removal of dietitian oversight/approval for meals is clearly conveyed.
States that the 2025 manual may violate the regulatory requirement for Grove to maintain quality control over dietary services
Fail
Evidence: The response says the change may create risk because rules require “qualified-dietician calculation for certain therapeutic diets,” but it does not state a requirement to “maintain quality control over dietary services.” Assessment: Fails criterion “States that the 2025 manual may violate the regulatory requirement for Grove to maintain quality control over dietary services.”
States that the 2025 manual allows for non-urgent complaints to be acknowledged within 14 days
Pass
Evidence: The response states the 2025 manual gives “only 14 business days to acknowledge complaints.” Assessment: Passes criterion “States that the 2025 manual allows for non-urgent complaints to be acknowledged within 14 days”; although it omits “non-urgent,” it clearly identifies the 14-day complaint-acknowledgment change.
States that the 2025 manual may violate the regulatory requirement for Grove to respond promptly to complaints
Pass
Evidence: The response states the complaint change “may create Texas ALF compliance risk” because rules require “prompt complaint resolution.” Assessment: Passes criterion “States that the 2025 manual may violate the regulatory requirement for Grove to respond promptly to complaints.”
States that the Emergency Preparedness section was removed in the 2025 manual
Fail
Evidence: The response says the 2025 manual “loosens” emergency-preparedness safeguards, but does not say the Emergency Preparedness section was removed. Assessment: Fails criterion “States that the Emergency Preparedness section was removed in the 2025 manual.”
States that 2025 manual may violate the regulatory requirement for Grove to develop and maintain a written emergency preparedness and response plan
Pass
Evidence: The response states the changes may create compliance risk because rules require “a written emergency-preparedness plan.” Assessment: Passes criterion “States that 2025 manual may violate the regulatory requirement for Grove to develop and maintain a written emergency preparedness and response plan.”