Raycaster / evalsBack to AI Agents for Privacy and GDPR Compliance

APEX-Agents · Law

Task acg14a6c

5/6Fail

APEX-Agents task Task acg14a6c in AI Agents for Privacy and GDPR Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Privacy and GDPR ComplianceLaw World 415Dual harnessGrader: rubric
task_5c14b248c88842b1a3442152671cbb18
Law World 415
make_new_doc
7 models · dual config

Task prompt

What the agent was asked to do

Can you review Sections 8.4, 9.1, 9.3,10.1,14.7 in our MSA, against the below outage scenario? A faulty update causes endpoint failures. The customer sends a breach notice by email to a support inbox, seeks lost revenue and lost data costs and asserts that CrowdStrike must indemnify all losses. For each of the clauses that applies to the facts above, state whether CrowdStrike is liable or not, and provide a one-sentence reason for the conclusion. Put it in a new doc file you create.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual5/6Fail
fireworks models Kimi K2dual4/6Fail
Gemini 3 Flashdual6/6Pass
Gemini 3.1 Produal6/6Pass
GPT-5.4dual5/6Fail
GPT-5.4 minidual6/6Pass
GPT-5.4 nanodual5/6Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that section 8.4 of the MSA does not protect CrowdStrike from their own mistakes

    Pass

    Evidence: In the created doc at /root/workspace/filesystem/Commercial Counsel/CrowdStrike MSA Outage Scenario Clause Review.docx, Section 8.4 row says “Section 8.4 does not appear to exclude express warranty coverage” and the reason says the issue “stems from CrowdStrike's own faulty update rather than” excluded customer/non-CrowdStrike causes. Assessment: The criterion asks whether it states 8.4 does not protect CrowdStrike from their own mistakes; pass, as the document clearly says 8.4 exclusions do not apply to CrowdStrike’s own faulty update.

  2. States that Section 10.1 of the MSA limits liability for all of the following: (1) lost revenue and (2) lost data damages

    Pass

    Evidence: The Section 10.1 row says “CrowdStrike is not liable for lost revenue or lost-data damages” and explains that Section 10.1 “expressly excludes lost profits, revenue… lost data.” Assessment: The criterion requires stating Section 10.1 limits liability for both lost revenue and lost data damages; pass, both categories are specifically addressed.

  3. States that the customer's email does not meet the contractual delivery requirements in Section 14.7 of the MSA

    Pass

    Evidence: The Section 14.7 row states “The support-inbox email does not appear to be an effective legal notice or breach-notice trigger,” and says Section 14.7 requires delivery by personal delivery, courier, or registered/certified mail, “so an email to a support inbox does not satisfy the stated notice mechanism.” Assessment: The criterion asks whether it states the customer email does not meet Section 14.7 delivery requirements; pass.

  4. States that CrowdStrike is liable under section 8.4 of the MSA

    Fail

    Evidence: The Section 8.4 liability conclusion says “No standalone liability under Section 8.4; however, Section 8.4 does not appear to exclude express warranty coverage on the assumed facts.” Assessment: The criterion requires stating CrowdStrike is liable under Section 8.4; fail, because the document expressly says there is no standalone liability under Section 8.4 rather than stating CrowdStrike is liable under it.

  5. States that CrowdStrike is not liable under Section 10.1 of the MSA

    Pass

    Evidence: The Section 10.1 liability conclusion states “CrowdStrike is not liable for lost revenue or lost-data damages, and any non-excluded direct damages would be capped.” Assessment: The criterion requires stating CrowdStrike is not liable under Section 10.1; pass.

  6. States that CrowdStrike is not liable under Section 14.7 of the MSA

    Pass

    Evidence: The Section 14.7 liability conclusion states “The support-inbox email does not appear to be an effective legal notice or breach-notice trigger under Section 14.7,” with the reason that email to support “does not satisfy the stated notice mechanism.” Assessment: The criterion requires stating CrowdStrike is not liable under Section 14.7; pass, the document’s clause-by-clause liability conclusion indicates no liability/trigger from the defective notice under that section.