Skip to content

Control 2.5: Testing, Validation, and Quality Assurance - Troubleshooting

This playbook provides troubleshooting guidance for Control 2.5.

Common Issues and Solutions

Issue Symptoms Root Cause Solution
Test environment mismatch Tests pass in test but fail in production Configuration drift Compare and sync environment settings
Intermittent test failures Same test passes sometimes, fails other times Race conditions Add wait times, retry logic
UAT delays Business users not completing UAT Unclear expectations Provide clear scenarios, schedule dedicated time
Golden dataset outdated Regression tests failing on valid responses Knowledge sources updated Update golden dataset entries
Hallucination rate increase Groundedness scores declining Knowledge source gaps Review and expand knowledge base

Detailed Troubleshooting

Issue 1: Test Environment Not Matching Production

Symptoms: Tests pass in test but fail in production

Resolution:

  1. Compare environment configurations:
  2. DLP policies
  3. Security roles
  4. Connection references

  5. Environment variables

  6. Verify DLP policies match:

  7. Export DLP from both environments
  8. Compare connector allowances

  9. Sync any differences

  10. Check data source connectivity:

  11. Verify test environment can reach same data sources

  12. Check authentication methods match

  13. Review security role differences:

  14. Compare role assignments

  15. Verify service accounts exist in both

  16. Sync solution versions:

  17. Export from production
  18. Import to test environment

Issue 2: Automated Tests Failing Intermittently

Symptoms: Same test passes sometimes, fails other times

Resolution:

  1. Add appropriate wait times: 

    # Add delay between requests
Start-Sleep -Milliseconds 500

  2. Check for race conditions:

  3. Review async operations

  4. Add proper synchronization

  5. Review test data dependencies:

  6. Ensure test data is reset between runs

  7. Check for data state dependencies

  8. Increase timeout values: 

    $response = Invoke-RestMethod -Uri $endpoint -TimeoutSec 60

  9. Add retry logic: 

    $maxRetries = 3
for ($i = 0; $i -lt $maxRetries; $i++) {
    try {
        $response = Invoke-RestMethod -Uri $endpoint
        break
    }
    catch {
        if ($i -eq $maxRetries - 1) { throw }
        Start-Sleep -Seconds 2
    }
}

Issue 3: UAT Delays

Symptoms: Business users not completing UAT

Resolution:

  1. Provide clear test scenarios:
  2. Write step-by-step instructions
  3. Include expected outcomes

  4. Provide example inputs

  5. Schedule dedicated UAT time:

  6. Block calendars for testing
  7. Remove competing priorities

  8. Set clear deadlines

  9. Offer testing support:

  10. Assign QA resource to assist
  11. Provide FAQ document

  12. Hold office hours for questions

  13. Simplify test documentation:

  14. Use screenshots
  15. Create video walkthroughs

  16. Reduce required documentation

  17. Set firm deadlines with escalation:

  18. Communicate consequences of delay
  19. Escalate to management if needed

Issue 4: Golden Dataset Outdated

Symptoms: Valid agent responses failing regression tests

Resolution:

  1. Review failed test cases:
  2. Identify why expected response no longer matches

  3. Determine if agent response is actually correct

  4. Update golden dataset:

  5. Modify expected_answer_contains patterns

  6. Add new acceptable variations

  7. Establish update process:

  8. Schedule quarterly reviews
  9. Assign ownership for maintenance

  10. Track knowledge source updates

  11. Version golden datasets:

  12. Tag with date and version
  13. Maintain history of changes
  14. Document reasons for updates

Issue 5: Hallucination Rate Increasing

Symptoms: Groundedness scores declining, inaccurate responses

Resolution:

  1. Analyze failed responses:
  2. Review specific hallucinations

  3. Identify patterns or topics

  4. Check knowledge sources:

  5. Verify sources are current
  6. Look for gaps in coverage

  7. Check for conflicting information

  8. Review agent prompts:

  9. Check system prompts for clarity
  10. Ensure grounding instructions are clear

  11. Add constraints if needed

  12. Expand knowledge base:

  13. Add missing topics
  14. Update outdated content

  15. Improve source quality

  16. Adjust agent settings:

  17. Increase grounding strictness
  18. Add fallback behaviors for uncertain topics

Issue 6: Performance Degradation

Symptoms: Response times exceeding thresholds

Resolution:

  1. Check concurrent usage:
  2. Review usage patterns

  3. Identify peak times

  4. Review knowledge source size:

  5. Large knowledge bases slow responses

  6. Consider chunking or filtering

  7. Check connector performance:

  8. Test individual connectors

  9. Look for slow data sources

  10. Review agent complexity:

  11. Complex topic flows increase latency

  12. Simplify where possible

  13. Monitor environment resources:

  14. Check Dataverse performance
  15. Review environment limits

Issue 7: Security Tests Failing

Symptoms: Prompt injection or data leakage detected

Resolution:

  1. Review agent configuration:
  2. Check system prompt safeguards

  3. Verify content moderation settings

  4. Strengthen prompts:

  5. Add explicit security instructions

  6. Include boundary definitions

  7. Enable content filtering:

  8. Configure Azure AI Content Safety

  9. Block harmful patterns

  10. Test additional attack vectors:

  11. Expand security test cases

  12. Include new injection techniques

  13. Document and escalate:

  14. Log security findings
  15. Involve security team

Escalation Path

If issues cannot be resolved using this guide:

  1. Level 1: QA Lead - Test methodology and execution
  2. Level 2: AI Governance Lead - Policy and standards
  3. Level 3: Compliance Officer - Regulatory requirements
  4. Level 4: Microsoft Support - Product-level issues

Updated: January 2026 | Version: v1.2