Raycaster / evalsBack to AI Agents for Privacy and GDPR Compliance

APEX-Agents · Law

Law_World_423_DM_05

2/9Fail

APEX-Agents task Law_World_423_DM_05 in AI Agents for Privacy and GDPR Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Privacy and GDPR ComplianceLaw World 423Dual harnessGrader: rubric
task_58e8668c0a7c47808ff26e7b1e5105ae
Law World 423
make_new_doc
7 models · dual config

Task prompt

What the agent was asked to do

Northstar's US customer, Zellwerk, reported a system-wide outage that delayed access to shipment-tracking data containing health-related product identifiers and customer account IDs. During the outage investigation, Northstar's internal team discovered an unapproved third-party analytics module embedded in the US and European instances of the platform for "temporary performance monitoring." The General Counsel has reached out asking if Northstar's data practices would be considered unfair under the Federal Trade Commission Act. Make a NEW document, and prepare a short memorandum with a summary of the relevant legal authority, analysis, and a conclusion.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual2/9Fail
fireworks models Kimi K2dual2/9Fail
Gemini 3 Flashdual3/9Fail
Gemini 3.1 Produal1/9Fail
GPT-5.4dual2/9Fail
GPT-5.4 minidual3/9Fail
GPT-5.4 nanodual2/9Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that consumer injury could not reasonably have avoided is a required element for a finding of unfairness under the FTC Act

    Pass

    Evidence: In /root/workspace/filesystem/Incident Documentation/Deliverables/Northstar FTC Unfairness Memorandum.docx, Relevant Legal Authority states unfairness requires injury that “cannot be reasonably avoided by consumers themselves.” Analysis section 2 says customers/downstream individuals “could not reasonably avoid the injury.” Assessment: The criterion asks whether it states reasonable avoidability is a required element; pass.

  2. States that a violation of public policy is a required element for a finding of unfairness under the FTC Act

    Fail

    Evidence: The memo’s Relevant Legal Authority lists the three unfairness elements and then says, “Public policy may be considered with the other evidence, but it is not the primary basis for unfairness.” Analysis section 4 similarly says public policy “cannot be the primary basis.” Assessment: The criterion asks whether it states a violation of public policy is a required element; it states the opposite/only ancillary role, so fail.

  3. States that unethical or unscrupulous activity is a required element for a finding of unfairness under the FTC Act

    Fail

    Evidence: The memo’s Relevant Legal Authority defines unfairness as substantial injury, reasonable avoidability, and countervailing-benefits balancing. It does not list “unethical or unscrupulous activity” as an element. Assessment: The criterion asks whether it states unethical or unscrupulous activity is required; the document does not state that, so fail.

  4. States that substantial consumer injury must have occurred in its net effects in order for a practice to be deemed unfair under the FTC Act

    Fail

    Evidence: The memo states an act is unfair when it “causes or is likely to cause substantial injury,” and concludes substantial injury is “supportable because the practices created a likely risk” of harm. It also says “unwarranted health and safety risks may constitute substantial injury even before monetary loss is proven.” Assessment: The criterion requires stating substantial consumer injury must have occurred in net effects; the memo allows likely injury/risk rather than occurrence, so fail.

  5. States that an exemption to "substantial injury" involves analyzing whether the injury is offset by consumer or competitive benefits of the activity

    Pass

    Evidence: Relevant Legal Authority states unfairness requires injury “not outweighed by countervailing benefits to consumers or competition.” It further says the practice must be injurious “in its net effects after weighing benefits and remedy costs.” Assessment: The criterion asks whether it states the offset/benefits analysis for substantial injury; pass.

  6. States that Northstar did not cause a substantial injury

    Fail

    Evidence: The memo states “Northstar faces a credible and material risk” of unfairness and that substantial injury is “supportable” based on likely privacy and health/safety harm. Although it calls substantial injury “the closest issue,” it does not conclude Northstar did not cause substantial injury. Assessment: The criterion asks for a statement that Northstar did not cause substantial injury; fail.

  7. States that Northstar did not violate public policy

    Fail

    Evidence: Analysis section 4 says GDPR, internal policies, and FTC guidance show “the bypassed controls were not optional hygiene” and “the risk was foreseeable.” The memo does not state Northstar did not violate public policy. Assessment: The criterion asks whether it states no public-policy violation; fail.

  8. States that Northstar's actions do not constitute unethical or unscrupulous practices

    Fail

    Evidence: The memo characterizes “governance failures,” “unapproved” analytics, “bypassed or absent” controls, and a plausible unfairness claim. It contains no statement that the actions do not constitute unethical or unscrupulous practices. Assessment: The criterion asks for that exculpatory statement; fail.

  9. States that Northstar did not engage in unfair practices in violation of Section 5 of the FTC Act

    Fail

    Evidence: The Conclusion says “the FTC could plausibly view the practices as unfair under Section 5” and recommends the legal position: “potentially unfair; high remediation priority.” Assessment: The criterion asks whether the memo states Northstar did not engage in unfair practices in violation of Section 5; it states the opposite risk-based conclusion, so fail.