APEX-Agents · Law
Task iv36a08a
APEX-Agents task Task iv36a08a in AI Agents for SEC Disclosure Analysis. Compare dual-harness agent runs across models, scores, and public traces.
Task prompt
What the agent was asked to do
Analyze whether Counts 1-7 of Delta's Complaint against CrowdStrike fall within the limitation of liability clause in Section 10.1 of CrowdStrike's standard MSA, indicating "Covered" or "Not Covered" for each count. Reply back to me here with your assessments.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| Gemini 3.1 Pro | dual | 6/7 | Fail | Run detailsPublic trace |
| fireworks models Kimi K2 | dual | 3/7 | Fail | Run detailsPublic trace |
| GPT-5.4 | dual | 1/7 | Fail | Run detailsPublic trace |
| GPT-5.4 mini | dual | 1/7 | Fail | Run detailsPublic trace |
| GPT-5.4 nano | dual | 1/7 | Fail | Run detailsPublic trace |
| GPT-5.5 | dual | 1/7 | Fail | Run detailsPublic trace |
Grading rubric
Rubric criteria
Runs are graded against these criteria. Open a run for model-specific verdicts.
States that Count 1 is Not Covered
States that Count 2 is Not Covered
States that Count 3 is Not Covered
States that Count 4 is Not Covered
States that Count 5 is Covered
States that Count 6 is Not Covered
States that Count 7 is Not Covered