Raycaster / evalsBack to AI Agents for Tax Due Diligence

APEX-Agents · Law

World425_jrf_01

3/3Pass

APEX-Agents task World425_jrf_01 in AI Agents for Tax Due Diligence. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Tax Due DiligenceLaw World 425Dual harnessGrader: rubric
Task IDtask_11893dcabbe34b0aa991516dfe7edcba
WorldLaw World 425
Expected outputmessage_in_console
Published runs7 models · dual config

Task prompt

What the agent was asked to do

We need to review Summit's historical distributions. Write me a memo analyzing whether Summit's shareholder distributions are in accordance with US tax code. Cite the exact tax code in each instance (short citations are acceptable). Reply to me right in here please.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual3/3PassShare pagePublic trace
fireworks models Kimi K2dual2/3FailShare pagePublic trace
Gemini 3 Flashdual3/3PassShare pagePublic trace
Gemini 3.1 Produal3/3PassShare pagePublic trace
GPT-5.4dual3/3PassShare pagePublic trace
GPT-5.4 minidual2/3FailShare pagePublic trace
GPT-5.4 nanodual1/3FailShare pagePublic trace

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that Summit's shareholder distributions are disproportionate

    Pass

    Evidence: The response states, “The May 22, 2022 distribution is materially disproportionate” and “The February 9, 2024 distribution is also disproportionate,” and concludes “the 2022 and 2024 distributions are not pro rata as recorded.” Assessment: Criterion asks whether it states that Summit's shareholder distributions are disproportionate. Pass; the response clearly identifies disproportionate distributions.

  2. States that Summit's disproportionate distributions violate US tax code

    Pass

    Evidence: The response states Summit’s distributions are “not fully in accordance with the U.S. S-corporation tax rules,” and for 2022 says the disproportionate distribution is “not consistent with the identical distribution rights required by IRC §1361(b)(1)(D) and Treas. Reg. §1.1361-1(l)(1)-(2).” It also says the 2024 issue is “real under IRC §1361(b)(1)(D).” Assessment: Criterion asks whether it states that disproportionate distributions violate US tax code. Pass; it explicitly ties disproportionate/non-pro-rata distributions to violations/noncompliance under cited tax provisions.

  3. States that a nonresident alien receiving shares violates tax code

    Pass

    Evidence: The response states, “if Carrie was a nonresident alien, Summit would have ceased to qualify as an S corporation,” and “IRC §1361(b)(1)(C) prohibits an S corporation from having a nonresident alien shareholder.” It recommends confirming her status and seeking relief if she was a nonresident alien. Assessment: Criterion asks whether it states that a nonresident alien receiving shares violates tax code. Pass; the response clearly states that nonresident alien shareholder ownership would violate the S-corporation eligibility rule.