APEX-Agents · Management Consulting

World112-1_Task02_TB

0/3Fail

APEX-Agents task World112-1_Task02_TB in AI Agents for Cross-Border Regulatory Review. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Cross-Border Regulatory ReviewManagement Consulting World 112.1Dual harnessGrader: rubric

Task IDtask_87dd4d824ff84835b0ed505d2499c15a

WorldManagement Consulting World 112.1

Expected outputmessage_in_console

Published runs5 models · dual config

Task prompt

What the agent was asked to do

I want to inform client discussions around what action to take regarding sourcing of materials and the speculated tariffs. Can you summarize averages of the following 3 data points pertaining to tariffs from the latest group of supply chain expert witness interviews: 1. % of competitors that are shifting sourcing to domestic US material suppliers. 2. % of competitors that are exploring international diversification. 3. Average % increase in material costs for companies that moved sourcing exclusively to the USA. Take an average of the values identified for each data point, and adjust for outliers by removing any values that are more than 1.5 population standards deviation from the mean. Also only consider data from expert witnesses with a seniority level of Director, VP, or Chief in their job titles. Write your reply straight here. Your final outputs should be rounded to the nearest .01%.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

Model	Harness	Score	Result	Links
GPT-5.5showcase	dual	0/3	Fail	Share page Public trace
Gemini 3.1 Pro	dual	0/3	Fail	Share page Public trace
GPT-5.4	dual	0/3	Fail	Share page Public trace
GPT-5.4 mini	dual	0/3	Fail	Share page Public trace
GPT-5.4 nano	dual	0/3	Fail	Share page Public trace

Grading rubric

Criteria and grader verdict (showcase run)

States the outlier-adjusted % of competitors that are shifting sourcing to domestic US material suppliers is 39.06%
Fail
Evidence: TEXT_RESPONSE states “Competitors shifting sourcing to domestic U.S. material suppliers | 38.11%”. Assessment: Criterion requires stating the outlier-adjusted percentage is 39.06%; the response gives 38.11%, so fail.
States the outlier-adjusted % of competitors that are exploring international diversification is 62.81%
Fail
Evidence: TEXT_RESPONSE states “Competitors exploring international diversification | 60.29%”. Assessment: Criterion requires 62.81%; the response gives 60.29%, so fail.
States the outlier-adjusted average % increase in material costs for companies that moved sourcing exclusively to the USA is 10.40%
Fail
Evidence: TEXT_RESPONSE states “Material cost increase for companies that moved sourcing exclusively to the USA | 13.00%”. Assessment: Criterion requires 10.40%; the response gives 13.00%, so fail.