APEX-Agents · Management Consulting

Task 14

6/9Fail

APEX-Agents task Task 14 in AI Agents for SaaS Due Diligence. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for SaaS Due DiligenceManagement Consulting World 129Dual harnessGrader: rubric

Task IDtask_0cc6381ab1db4902bdac1c95b3ffcc45

WorldManagement Consulting World 129

Expected outputmessage_in_console

Published runs5 models · dual config

Task prompt

What the agent was asked to do

Can you use the feature usage file and identify the average adoption rate percentage, average monthly usage, and average retention impact score for each tier except Team? Print the output for me here.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

Model	Harness	Score	Result	Links
GPT-5.5showcase	dual	6/9	Fail	Share page Public trace
Gemini 3.1 Pro	dual	6/9	Fail	Share page Public trace
GPT-5.4	dual	6/9	Fail	Share page Public trace
GPT-5.4 mini	dual	6/9	Fail	Share page Public trace
GPT-5.4 nano	dual	6/9	Fail	Share page Public trace

Grading rubric

Criteria and grader verdict (showcase run)

States the average adoption rate of Business is 30.0%
Fail
Evidence: <TEXT_RESPONSE> table lists Business Average Adoption Rate % as “39.69%.” Assessment: Criterion requires stating Business average adoption rate is 30.0%; the response gives a different value, so fail.
States the average adoption rate of Growth is 42.1%
Fail
Evidence: <TEXT_RESPONSE> table lists Growth Average Adoption Rate % as “59.95%.” Assessment: Criterion requires stating Growth average adoption rate is 42.1%; the response gives a different value, so fail.
States the average adoption rate of Enterprise is 44.9%
Fail
Evidence: <TEXT_RESPONSE> table lists Enterprise Average Adoption Rate % as “59.61%.” Assessment: Criterion requires stating Enterprise average adoption rate is 44.9%; the response gives a different value, so fail.
States the average monthly usage of Business is 19.1 hours per month
Pass
Evidence: <TEXT_RESPONSE> table lists Business Average Monthly Usage as “19.11.” Assessment: Criterion requires Business average monthly usage is 19.1 hours per month; 19.11 rounds to 19.1 and is presented as monthly usage, so pass.
States the average monthly usage of Growth is 50.3 hours per month
Pass
Evidence: <TEXT_RESPONSE> table lists Growth Average Monthly Usage as “50.25.” Assessment: Criterion requires Growth average monthly usage is 50.3 hours per month; 50.25 rounds to 50.3 to one decimal and is presented as monthly usage, so pass.
States the average monthly usage of Enterprise is 51.1 hours per month
Pass
Evidence: <TEXT_RESPONSE> table lists Enterprise Average Monthly Usage as “51.14.” Assessment: Criterion requires Enterprise average monthly usage is 51.1 hours per month; 51.14 rounds to 51.1 and is presented as monthly usage, so pass.
States the average retention impact of Business is 1.46
Pass
Evidence: <TEXT_RESPONSE> table lists Business Average Retention Impact Score as “1.46.” Assessment: Criterion requires Business average retention impact is 1.46; the stated value matches, so pass.
States the average retention impact of Growth is 1.59
Pass
Evidence: <TEXT_RESPONSE> table lists Growth Average Retention Impact Score as “1.59.” Assessment: Criterion requires Growth average retention impact is 1.59; the stated value matches, so pass.
States the average retention impact of Enterprise is 1.50
Pass
Evidence: <TEXT_RESPONSE> table lists Enterprise Average Retention Impact Score as “1.50.” Assessment: Criterion requires Enterprise average retention impact is 1.50; the stated value matches, so pass.