APEX-Agents · Law
Task jhp9ccb4
APEX-Agents task Task jhp9ccb4 in AI Agents for SEC Disclosure Analysis. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
A real-time Rapid Response Content (RRC) update is automatically delivered and it causes performance degradation in a customer’s environment. The customer did not stage deployments, had no rollback plan, and used the system in a critical operational setting. Can you review the board memo, along with Crowdstrike's standard and proposed MSA and let me know if CrowdStrike bears the risk for service interruption caused by the RRC? Explain your response and tell me what documents inform your assessment. Write our your findings to me here. Thanks!
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 2/3 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 3/3 | Pass | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 0/3 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 0/3 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 2/3 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 1/3 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that CrowdStrike bears the RRC risk under one of the following: "CrowdStrike MSA.pdf"; "CrowdStrike MSA.docx"; and the standard MSA
FailEvidence: The response says, “Under the current standard MSA, CrowdStrike has a strong contractual argument that it does *not* bear the customer’s business-interruption risk” and concludes “CrowdStrike likely does not bear the contractual risk.” Assessment: The criterion requires the response to state that CrowdStrike bears the RRC risk under the standard/CrowdStrike MSA. It states the opposite, so fail.
States that there is no provision in the MSA that explicitly allocates the RRC risk to the customer
PassEvidence: The response says “the current MSA has **no specific RRC clause**” and “That gap is why the proposed MSA adds explicit RRC risk allocation.” Assessment: This satisfies the criterion that it state there is no MSA provision explicitly allocating RRC risk to the customer, via a clear paraphrase that the standard MSA lacks a specific RRC/risk-allocation clause. Pass.
States that there is no provision in the MSA that addresses content deployment
PassEvidence: The response says “the current MSA appears to impose only generic obligations” and “does not clearly require staged deployment or rollback planning for RRC,” and also notes “no specific RRC clause.” Assessment: The criterion requires stating no MSA provision addresses content deployment. The response conveys that the standard MSA lacks specific provisions for RRC deployment/staging/rollback, which is a reasonable paraphrase of content deployment in this context. Pass.