APEX-Agents · Law
Task acg14a6c
APEX-Agents task Task acg14a6c in AI Agents for Privacy and GDPR Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
Can you review Sections 8.4, 9.1, 9.3,10.1,14.7 in our MSA, against the below outage scenario? A faulty update causes endpoint failures. The customer sends a breach notice by email to a support inbox, seeks lost revenue and lost data costs and asserts that CrowdStrike must indemnify all losses. For each of the clauses that applies to the facts above, state whether CrowdStrike is liable or not, and provide a one-sentence reason for the conclusion. Put it in a new doc file you create.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 5/6 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 4/6 | Fail | Share pagePublic trace |
| Gemini 3 Flash | dual | 6/6 | Pass | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 6/6 | Pass | Share pagePublic trace |
| GPT-5.4 | dual | 5/6 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 6/6 | Pass | Share pagePublic trace |
| GPT-5.4 nano | dual | 5/6 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that section 8.4 of the MSA does not protect CrowdStrike from their own mistakes
PassEvidence: In the created doc at /root/workspace/filesystem/Commercial Counsel/CrowdStrike MSA Outage Scenario Clause Review.docx, Section 8.4 row says “Section 8.4 does not appear to exclude express warranty coverage” and the reason says the issue “stems from CrowdStrike's own faulty update rather than” excluded customer/non-CrowdStrike causes. Assessment: The criterion asks whether it states 8.4 does not protect CrowdStrike from their own mistakes; pass, as the document clearly says 8.4 exclusions do not apply to CrowdStrike’s own faulty update.
States that Section 10.1 of the MSA limits liability for all of the following: (1) lost revenue and (2) lost data damages
PassEvidence: The Section 10.1 row says “CrowdStrike is not liable for lost revenue or lost-data damages” and explains that Section 10.1 “expressly excludes lost profits, revenue… lost data.” Assessment: The criterion requires stating Section 10.1 limits liability for both lost revenue and lost data damages; pass, both categories are specifically addressed.
States that the customer's email does not meet the contractual delivery requirements in Section 14.7 of the MSA
PassEvidence: The Section 14.7 row states “The support-inbox email does not appear to be an effective legal notice or breach-notice trigger,” and says Section 14.7 requires delivery by personal delivery, courier, or registered/certified mail, “so an email to a support inbox does not satisfy the stated notice mechanism.” Assessment: The criterion asks whether it states the customer email does not meet Section 14.7 delivery requirements; pass.
States that CrowdStrike is liable under section 8.4 of the MSA
FailEvidence: The Section 8.4 liability conclusion says “No standalone liability under Section 8.4; however, Section 8.4 does not appear to exclude express warranty coverage on the assumed facts.” Assessment: The criterion requires stating CrowdStrike is liable under Section 8.4; fail, because the document expressly says there is no standalone liability under Section 8.4 rather than stating CrowdStrike is liable under it.
States that CrowdStrike is not liable under Section 10.1 of the MSA
PassEvidence: The Section 10.1 liability conclusion states “CrowdStrike is not liable for lost revenue or lost-data damages, and any non-excluded direct damages would be capped.” Assessment: The criterion requires stating CrowdStrike is not liable under Section 10.1; pass.
States that CrowdStrike is not liable under Section 14.7 of the MSA
PassEvidence: The Section 14.7 liability conclusion states “The support-inbox email does not appear to be an effective legal notice or breach-notice trigger under Section 14.7,” with the reason that email to support “does not satisfy the stated notice mechanism.” Assessment: The criterion requires stating CrowdStrike is not liable under Section 14.7; pass, the document’s clause-by-clause liability conclusion indicates no liability/trigger from the defective notice under that section.