APEX-Agents · Law
World423_JS_02
APEX-Agents task World423_JS_02 in AI Agents for Privacy and GDPR Compliance. Compare dual-harness agent runs across models, scores, and public traces.
Task prompt
What the agent was asked to do
It has come to our attention that some of the data transferred by the "Diagnostics Analytics Module" related to residents of Colorado. Does Colorado Law require us to notify Colorado residents of this data transfer? Please respond to me here as a memo that outlines the requirements under the relevant laws and analyzes Northstar's situation in reference to the incident documentation.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.4 mini | dual | 8/9 | Fail | Run detailsPublic trace |
| GPT-5.4 | dual | 5/9 | Fail | Run detailsPublic trace |
| GPT-5.4 nano | dual | 5/9 | Fail | Run detailsPublic trace |
| GPT-5.5 | dual | 5/9 | Fail | Run detailsPublic trace |
| fireworks models Kimi K2 | dual | 3/9 | Fail | Run detailsPublic trace |
| Gemini 3.1 Pro | dual | 2/9 | Fail | Run detailsPublic trace |
Grading rubric
Rubric criteria
Runs are graded against these criteria. Open a run for model-specific verdicts.
States that a disclosure of a security breach to affected Colorado residents is required unless a good faith investigation determines that a misuse of personal information is not reasonably likely to occur
States that a security breach means the unauthorized acquisition of unencrypted computerized data that compromises the integrity of personal data maintained by a covered entity
States that Northstar maintains personal information
States that Northstar is a covered entity
States that the "Diagnostics Analytics Module" transmitted metadata that did not include personally identifying information
States that there is no evidence that BlueQuill can identify any individual using the metadata it received from the "Diagnostics Analytics Module."
States that the personal information maintained by Northstar was not compromised
States that transmission of the data by the "Diagnostics Analytics Module" did not constitute a security breach under Colorado law
States that Northstar is not required to notify affected Colorado residents of the data transferred by the "Diagnostics Analytics Module"