Raycaster / evalsBack to AI Agents for Privacy and GDPR Compliance

APEX-Agents · Law

world415_aeu_01

4/10Fail

APEX-Agents task world415_aeu_01 in AI Agents for Privacy and GDPR Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Privacy and GDPR ComplianceLaw World 415Dual harnessGrader: rubric
task_7952b3923473458ab7c415da7be74810
Law World 415
message_in_console
7 models · dual config

Task prompt

What the agent was asked to do

We want to get ahead of preparing a settlement agreement for the Delta matter. Can you let me know which of Delta’s original causes of actions are no longer live as we head into the pre-trial conference in March? You can ignore the derivative claims, though I would like to know if punitive fees are likely to apply and whether there is a limit to them based on CrowdStrike’s litigation case file against Delta. And, assuming that IronPeak agrees to insure us during mediation next week, please also estimate our budget as we head into trial. Reply to me back in here with your view.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual4/10Fail
fireworks models Kimi K2dual6/10Fail
Gemini 3 Flashdual6/10Fail
Gemini 3.1 Produal4/10Fail
GPT-5.4dual4/10Fail
GPT-5.4 minidual6/10Fail
GPT-5.4 nanodual4/10Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States any of the following is one of Delta's original causes of action that is no longer live: (1) strict liability for a defective product and (2) Count V from Delta's complaint;

    Pass

    Evidence: The response states “Count V — strict-liability/product-defect is out” and says Delta withdrew “claims for product liability.” Assessment: The criterion asks whether the response states strict liability for defective product / Count V is no longer live. Pass.

  2. States any of the following is one of Delta's original causes of action that is no longer live: (1) that CrowdStrike's business practices were deceptive and unfair; (2) a claim pursuant O.C.G.A. § 10-1-391; and (3) Count VIII from Delta's complaint.

    Pass

    Evidence: The response states “Count VII — Georgia Fair Business Practices Act/deceptive and unfair business practices is also out” and refers to “FBPA withdrawal.” Assessment: The criterion accepts stating that deceptive/unfair business practices (or the O.C.G.A. § 10-1-391 claim / Count VIII) is no longer live. Despite using Count VII rather than Count VIII, it clearly identifies the deceptive/unfair business practices claim as out. Pass.

  3. States all of the following requirements to claim punitive damages under Georgia law: (1) that punitive damages must be tied to a tort action, and (2) that the punished party acted with willful intent.

    Fail

    Evidence: The response states punitive damages require “clear and convincing proof of willful misconduct, malice, fraud, wantonness, oppression, or an entire want of care raising conscious indifference.” It also discusses surviving tort/statutory theories, but does not state punitive damages must be tied to a tort action. Assessment: The criterion requires both tort-action tie and willful intent. Only the intent-related requirement is clearly covered. Fail.

  4. States that Delta's claim on the basis of fraud was limited to misrepresentation in section 6.2 of the services agreement between Delta and CrowdStrike.

    Fail

    Evidence: The response says the fraud claim remains “principally the alleged SSA § 6.2 ‘back door’ misrepresentation / present-intent theory and the omission theory requiring fact development.” Assessment: The criterion requires stating Delta’s fraud claim was limited to misrepresentation in section 6.2. The response does not say it is limited solely to § 6.2; it also says an omission theory remains. Fail.

  5. States that any punitive damages are likely tied to Delta's claim of gross negligence.

    Fail

    Evidence: The response says punitive damages survive because “computer trespass, trespass to personalty, and fraud remain,” and later discusses negligence evidence, but it does not tie punitive damages to Delta’s gross-negligence claim. Assessment: The criterion asks for stating any punitive damages are likely tied to gross negligence. Fail.

  6. States that establishing the willful intent required for a successful claim of gross negligence likely meets the requisite intent when determining whether punitive damages should be applied.

    Fail

    Evidence: The response describes the punitive standard and says Delta has some negligence evidence, but also says punitive damages are “possible rather than probable” and cites defenses. It does not state that willful intent required for gross negligence likely satisfies punitive-damages intent. Assessment: The required statement is absent. Fail.

  7. States that punitive damages are likely to apply if Delta's claim of gross negligence is successful.

    Fail

    Evidence: The response says “punitive damages as possible rather than probable” and does not say they are likely if Delta succeeds on gross negligence. Assessment: The criterion requires stating punitive damages are likely to apply if gross negligence succeeds. The response says the opposite likelihood posture and omits that conditional. Fail.

  8. States that punitive damages are generally limited to $250,000 unless a statutory exception applies.

    Pass

    Evidence: The response states “the likely statutory punitive cap is $250,000 if punitive damages are awarded” and quotes that the amount “shall be limited to a maximum of $250,000.00,” with caveats about product liability/specific intent. Assessment: This satisfies the criterion that punitive damages are generally limited to $250,000 unless an exception applies. Pass.

  9. States that any punitive damages awarded against CrowdStrike are likely limited to $250,000.

    Pass

    Evidence: The response states “the likely statutory punitive cap is $250,000 if punitive damages are awarded.” Assessment: The criterion asks whether it states punitive damages against CrowdStrike are likely limited to $250,000. Pass.

  10. States that the working budget for CrowdStrike's litigation against Delta is approximately $37M.

    Fail

    Evidence: The response’s budget bottom line is “$30 million as the gross litigation budget/reserve” and “net CrowdStrike-funded budget should be roughly $5 million.” Assessment: The criterion requires stating the working budget is approximately $37M. The response does not; it gives $30M/$5M instead. Fail.