Skip to content

Control 2.18: Automated Conflict of Interest Testing

Control ID: 2.18
Pillar: Management
Regulatory Reference: SEC Reg BI, SEC Rule 10b-5, FINRA 2111, FINRA 25-07, FINRA Rule 3110
Last UI Verified: April 2026
Governance Levels: Baseline / Recommended / Regulated

Objective

Implement automated testing to detect potential conflicts of interest in AI agent recommendations, particularly for agents providing product recommendations, investment guidance, or financial advice, supporting compliance with Regulation Best Interest.

Why This Matters for FSI

  • SEC Regulation Best Interest (Reg BI) / FINRA Rule 2111 (Suitability): Broker-dealers must act in the best interest of retail customers when making recommendations. The FINRA 2026 Annual Regulatory Oversight Report's dedicated GenAI section reinforces that suitability and best-interest obligations are technology-neutral — firms cannot outsource these obligations to AI systems, and human supervisors must validate that AI-generated recommendations are suitable for the specific customer.
  • SEC Rule 10b-5 (Anti-Fraud): Prohibits material misstatements, omissions, and self-serving recommendations; helps ensure AI agents do not steer clients toward proprietary or higher-fee products without a documented best-interest basis.
  • FINRA Rule 3110 (Supervision): Requires written supervisory procedures (WSPs) and reasonable supervision of recommendations. Pre-deployment and ongoing conflict testing supports the supervisory diligence FINRA expects for GenAI-assisted recommendations.
  • FINRA Notice 25-07 (AI Supervision): Reinforces that firms must implement testing, monitoring, and documentation of AI tools used in recommendations to retail customers.

Updated April 2026

The FINRA 2026 Annual Regulatory Oversight Report (December 2025) introduced a standalone GenAI section emphasizing pre-deployment testing, output monitoring, model risk management, and that obligations under Reg BI, Rule 2111, and Rule 3110 apply to AI-assisted recommendations on the same basis as human recommendations.

Control Description

This control establishes conflict testing through:

  1. Proprietary Bias Detection - Test for preferential recommendation of firm's own products
  2. Commission Bias Testing - Detect bias toward higher-compensation products
  3. Cross-Selling Analysis - Identify inappropriate bundling or upselling patterns
  4. Competitor Fairness - Ensure competitor products aren't unfairly excluded
  5. Prompt Audit - Review system prompts for prohibited bias instructions
  6. Scoring Validation - Verify product scoring algorithms are unbiased

Conflict Types to Test

Conflict Type Description Example
Employee vs Customer Agent recommendations favor firm over client Recommending proprietary products over better alternatives
Customer vs Customer Agent treats different customers unfairly Allocating limited resources to preferred customers
Related-Party Agent involved in transactions with affiliated entities Recommendations involving parent/subsidiary companies
Cross-Business Unit Information barriers not properly enforced Research influencing investment banking recommendations

Key Configuration Points

  • Define conflict-of-interest test scenarios relevant to agent use case
  • Create test datasets with comparable proprietary vs. competitor products
  • Establish bias thresholds (e.g., proprietary recommendations should not exceed market share)
  • Configure automated testing in pre-deployment pipeline
  • Schedule recurring conflict testing (quarterly minimum)
  • Audit system prompts for prohibited bias language
  • Document testing methodology and results for examination

Automation Available

See COI Testing Framework in FSI-AgentGov-Solutions for automated conflict of interest testing with 10 predefined scenarios covering proprietary bias, suitability, fee transparency, and cross-selling.

Copilot Studio Evaluation Framework

Copilot Studio's agent evaluation framework can complement automated COI testing with an 8-step methodology for evidence-based agent validation. Key capabilities include:

  • Set-level grading — Evaluate COI test scenarios across entire test sets with aggregate scoring, enabling statistically meaningful bias detection rather than individual response review
  • Multi-dimensional graders — Score responses across accuracy, groundedness, coherence, and relevance dimensions to assess recommendation quality and appropriateness
  • Classification grading — Supports detection of proprietary bias, commission bias, and suitability issues by classifying agent responses against expected categories
  • Capability verification — Helps validate that agents invoke the correct topics and tools for different recommendation scenarios
  • Import/export test sets — Import standardized COI test scenarios and export results for version control, supporting reproducible conflict testing across environments
  • Production data import — Import real-world recommendation interactions as test inputs to validate against actual customer scenarios
  • Enterprise audit trail — Purview integration captures evaluation activities and grading results, supporting examination readiness under FINRA Rule 3110 and Reg BI evidence retention requirements
  • Comparative monitoring — Enables sequential evaluation runs to track quality trends and detect regressions over time

The evaluation framework supports observable, repeatable, and explainable testing—key attributes for demonstrating supervisory diligence under FINRA Rule 3110. See the Verification & Testing playbook for evaluation methodology guidance and Agent Evaluation in Copilot Studio for platform documentation.

Zone-Specific Requirements

Zone Requirement Rationale
Zone 1 (Personal) Awareness only; no recommendation agents Personal agents typically don't make recommendations
Zone 2 (Team) Basic conflict testing for recommendation agents; documented methodology Team recommendation agents warrant validation
Zone 3 (Enterprise) Comprehensive Reg BI testing; automated monitoring; independent validation; quarterly review Customer-facing recommendations require rigorous conflict controls

Roles & Responsibilities

Role Responsibility
Compliance Officer Define testing requirements, validate Reg BI alignment
AI Governance Lead Configure testing, oversee methodology
Model Risk Manager Independent validation of testing approach
Agent Owner Remediate identified conflicts, update prompts
Control Relationship
2.11 - Bias Testing Complementary bias testing for fairness
2.6 - Model Risk Management Conflict testing is MRM component
2.12 - Supervision Supervisory review of recommendations
2.20 - Adversarial Testing Red team testing for hidden biases

Implementation Playbooks

Step-by-Step Implementation

This control has detailed playbooks for implementation, automation, testing, and troubleshooting:

Verification Criteria

Confirm control effectiveness by verifying:

  1. Test scenarios defined for proprietary, commission, and cross-selling bias
  2. Test datasets include comparable proprietary and competitor products
  3. Automated conflict testing executes in pre-deployment pipeline
  4. System prompts audited and free of prohibited bias instructions
  5. Conflict testing report generated with statistical analysis

Additional Resources

Updated: April 2026 | Version: v1.4.0 | UI Verification Status: Current