Skip to content

Control 2.6: Model Risk Management - Troubleshooting

This playbook provides troubleshooting guidance for Control 2.6.

Common Issues and Solutions

Issue Symptoms Root Cause Solution
Unclear model classification Difficulty determining if agent is a "model" Ambiguous use case Review OCC 2011-12 definition; when in doubt, treat as model
Limited validation resources Cannot perform independent validation Resource constraints Identify internal teams not involved in development; engage external validators
Performance data unavailable Cannot measure model performance Logging not enabled Enable conversation logging; configure Dataverse analytics
Manifest reconstruction fails Cannot reproduce historical configuration Version control gaps Implement comprehensive manifest versioning

Detailed Troubleshooting

Issue 1: Unclear Model Classification

Symptoms: Difficulty determining if agent qualifies as a "model" under OCC 2011-12

Resolution:

  1. Review OCC 2011-12 model definition:

    A model is a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.

  2. Assess agent characteristics:

  3. Does it provide quantitative estimates?
  4. Does it influence business decisions?

  5. Does it affect customer outcomes?

  6. Evaluate impact:

  7. What happens if the agent provides incorrect output?

  8. Could errors result in financial loss or regulatory violation?

  9. Consult Model Risk Management team:

  10. Discuss borderline cases with MRM committee

  11. Document rationale for classification

  12. When in doubt, treat as model:

  13. More governance is better than less
  14. Can reclassify later if justified

Issue 2: Limited Validation Resources

Symptoms: Cannot perform independent validation due to resource constraints

Resolution:

  1. Identify internal teams not involved in development:
  2. Internal audit
  3. Risk management
  4. Compliance team

  5. Other business units

  6. Consider second-line risk functions:

  7. Operational risk
  8. Model risk management

  9. Compliance

  10. Engage external validators for Tier 1:

  11. Third-party assessment firms
  12. Consulting firms with MRM expertise

  13. Consider Cohasset Associates for recordkeeping compliance

  14. Use automated validation tools:

  15. Copilot Studio built-in analytics
  16. Golden dataset regression testing

  17. Automated bias testing

  18. Document resource constraints:

  19. Note limitations in validation report
  20. Request additional resources if needed

Issue 3: Performance Data Unavailable

Symptoms: Cannot measure model performance

Resolution:

  1. Enable conversation logging:
  2. Copilot Studio > Settings > Analytics

  3. Ensure logging is enabled for all conversations

  4. Configure Dataverse analytics:

  5. Set up Dataverse tables for conversation data

  6. Configure retention policies

  7. Implement user feedback collection:

  8. Add thumbs up/down to conversations
  9. Configure CSAT surveys

  10. Track escalation rates

  11. Create manual sampling process:

  12. Random sample review of conversations
  13. SME quality assessment

  14. Document sampling methodology

  15. Document data limitations:

  16. Note in model documentation
  17. Identify gaps for future improvement

Issue 4: Manifest Reconstruction Fails

Symptoms: Cannot reproduce agent configuration for a specific date

Resolution:

  1. Review version control history:
  2. Check Git history for manifest files
  3. Look for SharePoint version history

  4. Check solution export archives

  5. Identify gap in versioning:

  6. Determine what changes were not captured

  7. Document missing time periods

  8. Implement comprehensive versioning:

  9. Export manifest before every change
  10. Use Git with branch protection

  11. Configure automated export pipelines

  12. Test reconstruction capability:

  13. Conduct reconstruction drill
  14. Verify can retrieve configuration for any date

  15. Document procedure

  16. Establish remediation plan:

  17. Create process to prevent future gaps
  18. Set up automated manifest export

Issue 5: Model Change Impact Assessment

Symptoms: Unclear if change requires revalidation

Resolution:

  1. Review change classification criteria:
  2. Material change = Full revalidation
  3. Non-material change = Abbreviated review

  4. Emergency change = Expedited process

  5. Assess output impact:

  6. Will outputs change significantly?
  7. Could customer decisions be affected?

  8. Is there regulatory impact?

  9. Compare performance baselines:

  10. Run golden dataset against new version
  11. Compare accuracy metrics

  12. Identify any degradation

  13. Consult MRM guidelines:

  14. Review organization's MRM policy

  15. Align with existing change criteria

  16. Document decision:

  17. Record rationale for classification
  18. Get MRM approval for material changes

Issue 6: Multi-Agent Validation Complexity

Symptoms: Unclear how to validate agent orchestration chains

Resolution:

  1. Map complete delegation chain:
  2. Identify all agents in the chain
  3. Document interaction points

  4. Note data flows

  5. Apply highest tier to chain:

  6. If any agent is Tier 1, treat chain as Tier 1

  7. Document risk inheritance

  8. Validate end-to-end:

  9. Test complete user journeys
  10. Validate combined outputs

  11. Test failure scenarios

  12. Configure circuit breakers:

  13. Limit delegation depth
  14. Implement fallback procedures

  15. Monitor for cascade failures

  16. Document as single model:

  17. Register orchestration as model
  18. Reference component agents
  19. Maintain combined documentation

Escalation Path

If issues cannot be resolved using this guide:

  1. Level 1: AI Governance Lead - Classification and policy questions
  2. Level 2: Model Risk Manager - Validation and MRM framework
  3. Level 3: Compliance Officer - Regulatory requirements
  4. Level 4: External consultants - Third-party validation

Updated: January 2026 | Version: v1.2