Raycaster / evalsBack to AI Agents for Employment Law Analysis

APEX-Agents · Law

LawWorld417_AS_02

5/10Fail

APEX-Agents task LawWorld417_AS_02 in AI Agents for Employment Law Analysis. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Employment Law AnalysisLaw World 417Dual harnessGrader: rubric
task_43de2db73b5a4d899660f13394e80085
Law World 417
message_in_console
7 models · dual config

Task prompt

What the agent was asked to do

As you know, the new artist montage reel is a hit. Unfortunately, one of the artists featured is not a fan of the wardrobe upgrade - Mara Sings sent a takedown notice, and we initially complied, but we'd like to keep the reel in production and on air. Can you draft a letter to Mara that outlines Streams' IP policies and her legal obligations under her licensing agreement? You can cite these documents and California law to defend Streams' position when relevant, but the rationale shouldn't come off heavy-handed, more like lightly persuasive - a starting point for negotiations. Reply back to me here with the main body of the letter.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual5/10Fail
fireworks models Kimi K2dual5/10Fail
Gemini 3 Flashdual7/10Fail
Gemini 3.1 Produal2/10Fail
GPT-5.4dual5/10Fail
GPT-5.4 minidual2/10Fail
GPT-5.4 nanodual5/10Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that the promotional reel was a derivative work

    Pass

    Evidence: The response says Streams has a license to “adapt, modify, and create derivative works” and “Streams believes the edit falls within the licensed rights to adapt, modify, and create derivative promotional content.” Assessment: Criterion asks whether it states the promotional reel was a derivative work. Pass; it clearly characterizes the reel/edit as derivative promotional content.

  2. States that the promotional reel was within the permissible uses of the company's license with Mara

    Pass

    Evidence: The response states “Streams’ position is that the reel may remain in production and on air” and cites license rights to “use, distribute, monetize, adapt, modify, and create derivative works,” plus says “the edit falls within the licensed rights.” Assessment: Criterion asks whether it states the reel was within permissible uses of the company's license with Mara. Pass.

  3. States that Streams recognizes Mara's right of publicity concerns

    Pass

    Evidence: The letter says “To the extent your concern is framed as a likeness or publicity issue...” and later refers to “address your brand concerns.” Assessment: Criterion asks whether it states Streams recognizes Mara's right of publicity concerns. Pass; the response acknowledges the concern as a likeness/publicity issue, though it disputes legal effect.

  4. States that the use of AI to create derivative works of a person's likeness raises novel legal issues that are not fully addressed by existing copyright law

    Fail

    Evidence: The response does not mention AI, novel legal issues, or that existing copyright law does not fully address AI-generated derivative likeness works. Assessment: Criterion asks for that specific statement. Fail.

  5. States that Streams understood that using AI to modify Mara's image was within the broad consent of the talent agreement

    Fail

    Evidence: The response says “Streams believes the edit falls within the licensed rights to adapt, modify, and create derivative promotional content,” but does not mention AI or modifying Mara’s image using AI. Assessment: Criterion specifically asks whether it states Streams understood that using AI to modify Mara's image was within the broad consent of the talent agreement. Fail because the AI aspect is absent.

  6. States that the company complied with Mara's takedown notice per the licensing agreement(s)

    Fail

    Evidence: The letter says “Streams’ initial pause of the reel was intended as a good-faith accommodation” and the original task context in the response refers to reviewing “the takedown language,” but the response does not clearly state that Streams complied with Mara’s takedown notice per the licensing agreement(s). Assessment: Criterion asks for a statement of compliance with the takedown notice per the agreement. Fail; “initial pause” is not framed as compliance under the licensing agreement.

  7. States that Streams' compliance with the takedown notice demonstrates an attempt to facilitate dialogue

    Pass

    Evidence: The response states the “initial pause of the reel was intended as a good-faith accommodation while we reviewed the concern” and proposes a “prompt creative/legal check-in.” Assessment: Criterion asks whether Streams' compliance with the takedown notice demonstrates an attempt to facilitate dialogue. Pass; while it says initial pause rather than compliance, it clearly frames the action as good-faith accommodation to review and resolve the concern collaboratively.

  8. States that Streams requests permission to air the promotional reel

    Fail

    Evidence: The response says “Streams will keep the reel active and in production” and asks Mara to “suspend or withdraw the takedown notice,” but it does not request Mara’s permission to air the reel. Assessment: Criterion asks whether Streams requests permission to air the promotional reel. Fail; the letter asserts Streams will air it and seeks withdrawal of the takedown, not permission.

  9. States at least one potential option for compromise to begin negotiations with Mara to publish the reel

    Pass

    Evidence: The response offers “revising the wardrobe treatment in future cuts,” “adjusting thumbnails or key art,” “adding or refining credit language,” or “agreeing on a review process.” Assessment: Criterion asks for at least one potential compromise option to begin negotiations. Pass.

  10. States that a violation of California's rights of publicity requires demonstrable damages

    Fail

    Evidence: The response discusses California law generally and says the licensing framework supplies consent, but it does not state that a violation of California right of publicity requires demonstrable damages. Assessment: Criterion asks for that specific statement. Fail.