APEX-Agents · Law
World421_TG_03
APEX-Agents task World421_TG_03 in AI Agents for Cross-Border Regulatory Review. Compare dual-harness agent runs across models, scores, and public traces.
Task prompt
What the agent was asked to do
We've just received an MRA from the CFPB for our policies and procedures, so I need your help to draft our response. Can you let me know if our Complaint Response and Regulatory Investigation Protocols fail to meet any of the examination objectives for policies and procedures that are outlined by the CFPB’s compliance management review? For each objective identified, please include an explanation in one or two sentences and note if SLL has likely addressed the objective through another document. Write your draft here as a reply.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| Gemini 3 Flash | dual | 7/9 | Fail | Run detailsPublic trace |
| GPT-5.4 mini | dual | 7/9 | Fail | Run detailsPublic trace |
| GPT-5.5 | dual | 7/9 | Fail | Run detailsPublic trace |
| GPT-5.4 | dual | 5/9 | Fail | Run detailsPublic trace |
| fireworks models Kimi K2 | dual | 3/9 | Fail | Run detailsPublic trace |
| Gemini 3.1 Pro | dual | 3/9 | Fail | Run detailsPublic trace |
| GPT-5.4 nano | dual | 3/9 | Fail | Run detailsPublic trace |
Grading rubric
Rubric criteria
Runs are graded against these criteria. Open a run for model-specific verdicts.
States that the Complaint Response and Regulatory Investigation Protocols fail to sufficiently maintain compliance policies for employee reference
States that compliance policies should include at least one of the following timing requirements: (1) a mandatory period of time by which compliance policies should be reviewed or amended, (2) triggers for when the compliance policies should be reviewed outside of an established monitoring cycle, and (3) triggers for when compliance policies should be amended outside of an established monitoring cycle
States that the SLL Telemarketing Policies Manual (or "SLL Telemarketing Policies Manual.docx") is likely to meet the CFPB's examination objective regarding sufficiently maintaining compliance policies for employee reference
States that the Complaint Response and Regulatory Investigation Protocols fail to establish the existence of a board-approved compliance policy document
States that the Complaint Response and Regulatory Investigation Protocols fail to effectively manage compliance risk in all of the following: (1) products of Senior Living Lending, Inc. and (2) services of Senior Living Lending, Inc
States that the Complaint Response and Regulatory Investigation Protocols fail to identify all of the following: (1) the products offered by Senior Living Lending, Inc. and (2) the services offered by Senior Living Lending, Inc
States that the Complaint Response and Regulatory Investigation Protocols fail to cover the full life-cycle of all of the following: (1) products offered by Senior Living Lending, Inc. and (2) services offered by by Senior Living Lending, Inc
States that the Complaint Response and Regulatory Investigation Protocols fail to address all of the following purposes: (1) minimizing violations of applicable federal consumer financial laws, (2) detecting harm to consumers due to violations of applicable federal consumer financial laws, and (3) minimizing risks of consumer harm due to violations of applicable federal consumer financial laws
States that all of the following documents are likely to meet the CFPB's examination objective regarding sufficient compliance with applicable federal consumer financial laws: (1) the SLL Telemarketing Policies Manual (or "SLL Telemarketing Policies Manual.docx") and (2) the SLL CFPB Oversight of Senior-Focused Mortgage Marketing Memo (or "SLL CFPB Oversight of Senior-Focused Mortgage Marketing Memo.docx")