Control 3.6: Supervision and Oversight (FINRA 3110 / SEC Reg BI) — Verification & Testing
Test cases and evidence collection procedures for validating supervisory controls over Copilot-assisted activities.
Regulatory applicability
Regulatory citations identify commonly referenced requirements. Applicability, record classification, and retention periods are a firm and counsel determination; this framework does not assert that any regulation applies or is satisfied. See the Regulatory Applicability Questions appendix.
Test Cases
Test 1: Supervisory Review Workflow
- Objective: Confirm that Copilot-assisted communications flow through the supervisory review process
- Steps:
- Have a test registered representative draft an investment recommendation using Copilot.
- Send the recommendation to a test client account.
- Verify the communication appears in the assigned supervisor's review queue.
- Complete the review (approve, reject, or escalate) and confirm audit trail.
- Expected Result: Communication flows through supervisory review with complete audit trail of actions taken.
- Evidence: Review queue screenshot and audit log entry showing review completion.
Test 2: Pre-Send Hold for Investment Recommendations
- Objective: Validate that Copilot-drafted investment recommendations are held pending supervisory approval
- Steps:
- Configure pre-send hold for investment recommendation communications.
- Have a test user draft a recommendation via Copilot and attempt to send.
- Verify the message is held and not delivered until supervisor approves.
- Have the supervisor approve the message and confirm delivery.
- Expected Result: Message is held, supervisor reviews and approves, and the message is then delivered with approval timestamp.
- Evidence: Message trace showing hold status, approval action, and delivery confirmation.
Test 3: Supervisor Capacity Validation
- Objective: Verify that supervisory ratios are within acceptable limits for effective oversight
- Steps:
- Run the supervisor-to-representative ratio script.
- Verify no supervisor oversees more than 50 Copilot-enabled representatives unless compliance/legal approves a firm-specific threshold.
- Review each supervisor's review queue backlog.
- Confirm all supervisors are completing reviews within the defined SLA.
- Expected Result: All supervisory ratios are within policy limits and review SLAs are being met.
- Evidence: Ratio report and SLA compliance metrics.
Test 4: Reg BI Documentation Completeness
- Objective: Confirm that Copilot-assisted recommendations capture required Reg BI documentation elements
- Steps:
- Review a sample of 10 Copilot-drafted recommendations that were supervisory approved.
- Verify each recommendation includes: client suitability basis, cost disclosure, conflict of interest disclosure, and alternatives considered.
- Confirm the supervisory review log captures the reviewer's assessment of each element.
- Expected Result: All sampled recommendations contain required Reg BI elements and supervisory attestation.
- Evidence: Sampled review records showing Reg BI element completeness.
Test 5: Agent Audit Trail Capture (FINRA 3110(a) Agent Supervision)
- Objective: Verify that the audit trail correctly captures agent-specific interactions for supervisory review when a Teams channel agent or declarative agent is used
- Steps:
- Deploy a test Teams channel agent or use an existing declarative agent in a non-production channel.
- Have a test registered representative interact with the agent (e.g., ask it to summarize account information or draft a communication).
- Wait for audit events to propagate to the Purview audit log using the tenant's documented audit-ingestion expectation; allow up to 24 hours before treating delayed events as missing.
- Run Script 5 (Agent Interaction Audit) from the PowerShell setup guide to retrieve agent-specific CopilotInteraction events.
- Verify the returned records contain:
AgentId,AgentName, the interacting user's identity, and the interaction timestamp. - Confirm the
AccessedResources.XPIADetectedflag is absent orfalsefor normal interactions (no cross-prompt injection attempt detected on referenced resources). - Expected Result: Agent interactions appear in the audit log with correct AgentId, AgentName, user identity, and timestamp. The
AccessedResources.XPIADetectedflag is captured when a detection occurs. The records are exportable for supervisory review evidence. - Evidence: CSV export from Script 5 showing agent interaction records; Purview audit log screenshot showing CopilotInteraction records with AgentId populated.
Test 6: WSP Coverage Verification for Deployed Agents
- Objective: Confirm that the firm's written supervisory procedures (WSPs) address every currently deployed Copilot agent
- Steps:
- Generate a list of deployed Teams channel agents and declarative agents from the Microsoft 365 Admin Center (Admin Center > Agents > All agents / Registry).
- Cross-reference each deployed agent against the agent inventory section of the firm's WSP Copilot addendum.
- For any agent not listed in the WSP, flag as a gap requiring immediate documentation.
- For listed agents, verify the WSP entry includes: agent scope, authorized actions, supervisory review cadence, and the person responsible for oversight.
- Expected Result: All deployed agents are listed in the WSP with complete supervisory documentation. Zero undocumented agents are found.
- Evidence: Agent inventory from Admin Center (screenshot or export) cross-referenced against WSP agent list; any gaps documented with remediation date.
Evidence Collection
| Evidence Item | Source | Format | Retention |
|---|---|---|---|
| Supervisory review logs | Purview audit log | CSV export | 7 years; retention schedule: confirm |
| Pre-send hold records | Message trace | CSV | 7 years; retention schedule: confirm |
| Supervisor ratio report | PowerShell | Text export | With control documentation |
| Reg BI documentation samples | Review records | Redacted copies | 7 years; retention schedule: confirm |
| Agent interaction audit records | Script 5 output | CSV export | 7 years; retention schedule: confirm |
| WSP agent coverage gap report | Test 6 results | Document | With control documentation |
Compliance Mapping
| Regulation | Requirement | How This Control Helps |
|---|---|---|
| FINRA 3110 | Supervisory system and WSP citation | Provides evidence for supervisory review of AI-assisted activities; applicability: confirm |
| FINRA 3110(a) | Supervisory system citation for tools used by associated persons | Agent audit trail capture and WSP coverage verification provide evidence for counsel/compliance review of agent supervision |
| SEC Reg BI | Care, disclosure, and conflict citation | Provides evidence for best-interest documentation review; applicability: confirm |
| FINRA 3120 | Supervisory control system testing | Supports annual testing of supervisory effectiveness including agent supervision |
Next Steps
- Review Troubleshooting for supervisory workflow issues
- Proceed to Control 3.7 for regulatory reporting
- Back to Control 3.6