Raycaster / evalsBack to AI Agents for Comparable Company Analysis

APEX-Agents · Management Consulting

Task_World130_CamilleMoingeon_3

2/6Fail

APEX-Agents task Task_World130_CamilleMoingeon_3 in AI Agents for Comparable Company Analysis. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Comparable Company AnalysisManagement Consulting World 130Dual harnessGrader: rubric
Task IDtask_9363f85adf864ccaa8dbff72d38225cb
WorldManagement Consulting World 130
Expected outputmessage_in_console
Published runs5 models · dual config

Task prompt

What the agent was asked to do

Using HarFeast's baseline diagnostic dataset, assess the impact of predictive maintenance on HarFeast's scrap rate. We will pilot predictive maintenance only on equipment a) whose scheduled hours per year are at or above that equipment type's median scheduled hours and b) whose labor hours are at or above its plant's median labor hours. For all equipment qualifying for the pilot, apply the improvement assumptions from the Gogo Food case study. 1. Calculate the new overall scrap rates per product family, rounded to one decimal point place. 2. Calculate the total scrap units each product family avoids per year after these improvements, rounded to the nearest whole number. Please return all results to me in your reply.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual2/6FailShare pagePublic trace
Gemini 3.1 Produal6/6PassShare pagePublic trace
GPT-5.4dual4/6FailShare pagePublic trace
GPT-5.4 minidual4/6FailShare pagePublic trace
GPT-5.4 nanodual1/6FailShare pagePublic trace

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that the new overall scrap rate for the Canned Vegetables product family is 6.7%

    Fail

    Evidence: TEXT_RESPONSE table lists “Canned Vegetables | 6.3% | 589,283.” Assessment: Criterion requires stating the new overall scrap rate for Canned Vegetables is 6.7%; the response states 6.3%, so fail.

  2. States that the new overall scrap rate for the Condiments product family is 6.6%

    Fail

    Evidence: TEXT_RESPONSE table lists “Condiments | 6.5% | 379,583.” Assessment: Criterion requires stating the new overall scrap rate for Condiments is 6.6%; the response states 6.5%, so fail.

  3. States that the new overall scrap rate for the Sauces product family is 6.6%

    Pass

    Evidence: TEXT_RESPONSE table lists “Sauces | 6.6% | 283,526.” Assessment: Criterion requires stating the new overall scrap rate for Sauces is 6.6%; the response matches, so pass.

  4. States that the total number of scrap units the Canned Vegetables product family avoids every year as a result of these changes is 290,497

    Fail

    Evidence: TEXT_RESPONSE table lists “Canned Vegetables | 6.3% | 589,283.” Assessment: Criterion requires stating Canned Vegetables avoids 290,497 scrap units per year; the response states 589,283, so fail.

  5. States that the total number of scrap units the Condiments product family avoids every year as a result of these changes is 329,588

    Fail

    Evidence: TEXT_RESPONSE table lists “Condiments | 6.5% | 379,583.” Assessment: Criterion requires stating Condiments avoids 329,588 scrap units per year; the response states 379,583, so fail.

  6. States that the total number of scrap units the Sauces product family avoids every year as a result of these changes is 283,526

    Pass

    Evidence: TEXT_RESPONSE table lists “Sauces | 6.6% | 283,526.” Assessment: Criterion requires stating Sauces avoids 283,526 scrap units per year; the response matches, so pass.