APEX-Agents · Law
Task acg14a6c
APEX-Agents task Task acg14a6c in AI Agents for Privacy and GDPR Compliance. Compare dual-harness agent runs across models, scores, and public traces.
Task prompt
What the agent was asked to do
Can you review Sections 8.4, 9.1, 9.3,10.1,14.7 in our MSA, against the below outage scenario? A faulty update causes endpoint failures. The customer sends a breach notice by email to a support inbox, seeks lost revenue and lost data costs and asserts that CrowdStrike must indemnify all losses. For each of the clauses that applies to the facts above, state whether CrowdStrike is liable or not, and provide a one-sentence reason for the conclusion. Put it in a new doc file you create.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| Gemini 3 Flash | dual | 6/6 | Pass | Run detailsPublic trace |
| Gemini 3.1 Pro | dual | 6/6 | Pass | Run detailsPublic trace |
| GPT-5.4 mini | dual | 6/6 | Pass | Run detailsPublic trace |
| GPT-5.4 | dual | 5/6 | Fail | Run detailsPublic trace |
| GPT-5.4 nano | dual | 5/6 | Fail | Run detailsPublic trace |
| GPT-5.5 | dual | 5/6 | Fail | Run detailsPublic trace |
| fireworks models Kimi K2 | dual | 4/6 | Fail | Run detailsPublic trace |
Grading rubric
Rubric criteria
Runs are graded against these criteria. Open a run for model-specific verdicts.
States that section 8.4 of the MSA does not protect CrowdStrike from their own mistakes
States that Section 10.1 of the MSA limits liability for all of the following: (1) lost revenue and (2) lost data damages
States that the customer's email does not meet the contractual delivery requirements in Section 14.7 of the MSA
States that CrowdStrike is liable under section 8.4 of the MSA
States that CrowdStrike is not liable under Section 10.1 of the MSA
States that CrowdStrike is not liable under Section 14.7 of the MSA