APEX-Agents · Law
World421_mc_08
APEX-Agents task World421_mc_08 in AI Agents for Cross-Border Regulatory Review. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
Can you create a GLBA-compliant privacy framework (and put it in a CONCISE documents document that you create new) for SLL's text messaging program that addresses initial privacy notice delivery via SMS, opt-out rights for non-public personal information sharing, annual notice requirements, and simplified notice eligibility?
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 9/10 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 6/10 | Fail | Share pagePublic trace |
| Gemini 3 Flash | dual | 8/10 | Fail | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 6/10 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 7/10 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 7/10 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 9/10 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that the initial consumer privacy notice must be provided not later than when Senior Living Lending, Inc. establishes a customer relationship
PassEvidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 4: “Timing. For customers, provide the initial notice no later than the time the customer relationship is established...” Assessment: The criterion asks whether it states the initial consumer privacy notice must be provided not later than when SLL establishes a customer relationship. Pass; the document states this timing rule for customers.
States that SLL should use a hybrid delivery approach that includes all of the following: (1) SMS notification and (2) a URL link to the full privacy notice on a webpage
PassEvidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, bottom line: “Use SMS as a delivery and acknowledgment channel, not as the full GLBA privacy notice”; section 4: “The link must take the recipient directly to a printable/downloadable, version-controlled privacy notice”; sample SMS includes “SLL Privacy Notice: sll.com/privacy.” Assessment: The criterion requires a hybrid approach including SMS notification and URL link to the full notice webpage. Pass.
States at least one of the following regarding SLL’s full privacy notice: (1) contains the complete Model Privacy Form and (2) complies with 12 C.F.R. Part 1016
PassEvidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 2: “Use the current GLBA model privacy notice, or counsel-approved equivalent, as the controlling notice”; source basis: “Regulation P, 12 CFR Part 1016...” Assessment: The criterion requires stating the full notice contains the complete Model Privacy Form or complies with 12 C.F.R. Part 1016. Pass; it directs use of the GLBA model privacy notice.
States that SLL should obtain consumer consent for electronic delivery prior to delivering the privacy notices
PassEvidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 4: “Send the GLBA notice link by SMS only after the consumer has consented to electronic notices...” and section 2: “Electronic delivery through SMS requires the consumer/customer to agree to electronic delivery...” Assessment: The criterion requires consumer consent for electronic delivery before notices. Pass.
States that SLL must provide consumers with a reasonable means to exercise an opt-out right
PassEvidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 5: “SLL must provide an initial privacy notice, a clear opt-out notice, a reasonable means to opt out...” and “Reasonable means. Offer at least two practical channels...” Assessment: The criterion requires a reasonable means to exercise opt-out. Pass.
States at least two of the following mechanisms for opting out of non-public personal information sharing: (1) Designate check-off boxes in a prominent position on the relevant forms with the opt out notice; (2) Include a reply form together with the opt out notice that includes the address to which the form should be mailed; (3) Provide an electronic means to opt out; and (4) Provide a toll-free telephone number that consumers may call to opt out
PassEvidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 5: “Offer at least two practical channels, such as toll-free telephone, web form, in-app or landing-page control, reply form, or an agreed electronic method.” Assessment: The criterion requires at least two enumerated opt-out mechanisms. Pass; it includes toll-free telephone, electronic/web methods, and reply form.
States that all of the following situations are exceptions to SLL’s obligation to provide consumers with opt-out rights when disclosing non-public personal information: (1) to a non-affiliated third party to perform services for SLL; (2) to effect a transaction that a consumer requests or authorizes; and (3) to comply with the law
PassEvidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 3 table: “Section 1016.13 service providers / joint marketing” with “No opt-out notice required”; “Processing/servicing a requested loan... legal requests, regulators... No consumer opt-out right.” Assessment: The criterion requires exceptions for service providers, consumer-requested/authorized transactions, and law compliance. Pass; all three categories are included.
States that SLL must provide annual notice to consumers reflecting its privacy policies
PassEvidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 6: “Baseline rule. Provide each customer a clear and conspicuous annual privacy notice at least once in any 12 consecutive months during the customer relationship...” Assessment: The criterion requires stating SLL must provide annual notice reflecting privacy policies. Pass.
States that both of the following conditions must be met for SLL to be excepted from the requirement to provide annual privacy notice: (1) SLL only discloses a consumer’s non-public personal information to a non-affiliate third party in situations falling under the exceptions to SLL’s obligation to provide consumers with opt-out rights; and (2) SLL’s policies with regard to disclosing non-public personal information have not changed since the last privacy notice was provided
PassEvidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 6: “SLL may stop annual privacy notices only if both conditions remain true: (1) SLL shares NPI solely under... exceptions...; and (2) SLL has not changed its NPI disclosure policies and practices since the most recent GLBA privacy disclosure.” Assessment: The criterion requires both conditions for the annual-notice exception. Pass.
States that SLL should notify consumers of changes to its privacy practices within 100 days of the change
FailEvidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 6 states revised notices are required “before disclosing a new category of NPI or sharing with a new category...” but no inspected text states a 100-day deadline. Assessment: The criterion requires stating SLL should notify consumers of changes within 100 days. Fail; the document does not include the 100-day timing requirement.