Raycaster / evalsBack to AI Agents for Privacy and GDPR Compliance

APEX-Agents · Law

World423_JS_03

2/2Pass

APEX-Agents task World423_JS_03 in AI Agents for Privacy and GDPR Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Privacy and GDPR ComplianceLaw World 423Dual harnessGrader: rubric
task_28f0924227374bcea8d59b13e605be90
Law World 423
message_in_console
7 models · dual config

Task prompt

What the agent was asked to do

Northstar is reviewing its policies for notifying American data subjects in case there is another unauthorized data transfer. Specifically, a lot of Northstar's data subjects reside in the State of New York. Would Northstar be required, under New York Law, to notify affected New York residents, if the data contained in the Northstar health-related product identifiers were transferred to an unauthorized person? Respond to me me with a yes or no answer. Also, give a single explanation.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual2/2Pass
fireworks models Kimi K2dual0/2Fail
Gemini 3 Flashdual2/2Pass
Gemini 3.1 Produal0/2Fail
GPT-5.4dual2/2Pass
GPT-5.4 minidual2/2Pass
GPT-5.4 nanodual2/2Pass

Grading rubric

Criteria and grader verdict (showcase run)

  1. States Yes, Northstar is required, under New York Law, to notify affected New York residents, if the data contained in the Northstar health-related product identifiers were transferred to an unauthorized person

    Pass

    Evidence: The response begins “**Yes.**” and explains that “§ 899-aa requires notice to any New York resident whose private information ‘was, or is reasonably believed to have been, accessed or acquired by a person without valid authorization,’” so “an unauthorized transfer ... would require notice to affected New York residents.” Assessment: The criterion asks whether it states Yes, Northstar is required under New York law to notify affected New York residents if the health-related product identifier data were transferred to an unauthorized person; pass.

  2. States that the data contained in the Northstar health-related product identifiers is private information

    Pass

    Evidence: The response says New York law treats “private information” to include personal information combined with “medical information,” and concludes that the Northstar health/vital-sign records are “identifiable health-related records” covered by the notice requirement. Assessment: The criterion asks whether it states the data contained in the Northstar health-related product identifiers is private information; pass, as the response clearly characterizes the data as private information under the law.