Skip to content

Portal Walkthrough: Control 3.10 - Hallucination Feedback Loop

Last Updated: April 2026 Portal: Microsoft Copilot Studio, SharePoint Online, Power Automate, Power BI Estimated Time: 90-120 minutes

This playbook provides step-by-step portal configuration guidance for Control 3.10. It covers the full feedback loop: capture, triage, remediation, and trend reporting.

Detection Limitations (April 2026)

Microsoft Copilot Studio does not natively detect hallucinations. Every step in this playbook depends on human-submitted feedback (thumbs down, flag, or out-of-band escalation) and manual or workflow-driven review. Configure the controls below as a structured intake — not as an automated detector — and pair them with the proactive mitigations described in the parent control.

Prerequisites

  • Power Platform Admin role (for environment-level Copilot Studio settings)
  • Copilot Studio Agent Author access to each agent in scope (per-agent feedback toggle)
  • SharePoint Site Owner on the AI Governance site (to create the tracking list)
  • Power Automate premium license for the agent owner mailbox or service account that runs intake flows
  • Power BI Pro license for the AI Governance Lead (trend reporting)
  • Hallucination taxonomy and severity matrix approved by Compliance (see Step 2)
  • Service account or shared mailbox for SLA timer notifications (recommended for FINRA 3110 evidence integrity)
  • Decision recorded on integration with Control 3.4 (Incident Reporting) — direct flow vs. manual escalation

Step-by-Step Configuration

Part 1: Enable Feedback Capture in Copilot Studio

Step 1: Enable Customer Satisfaction (CSAT) on Each Agent

Portal Path: Copilot Studio → select agent → Settings → Customer satisfaction

  1. Open Copilot Studio and sign in with Copilot Studio Agent Author credentials
  2. Select the target agent from the agent list
  3. In the left navigation, expand Settings and choose Customer satisfaction
  4. Toggle Allow users to provide feedback to On
  5. Configure the survey:
  6. Survey type: CSAT survey after each conversation (recommended for Zone 2/3) or Thumbs only (acceptable for Zone 1)
  7. Comment box: Enable to capture free-text rationale (required for Zone 3 to meet SEC 17a-4 evidence quality)
  8. Trigger: After end-of-conversation and After each response (Zone 3 only — captures per-turn signal)
  9. Click Save
  10. Publish the agent so the configuration takes effect in deployed channels

Zone 3 Requirement: CSAT must be enabled with comment box on every published agent. Document the agent inventory and CSAT status as part of your supervisory procedures (FINRA Rule 3110 / Notice 25-07).

Portal Path: Copilot Studio → agent → Topics → + Add a topic → From blank

  1. Create a new topic named Report Inaccurate Response
  2. Trigger phrases: report a hallucination, that's wrong, incorrect answer, flag this response, escalate to human
  3. Add a Question node: "Help us improve. What category best describes the issue?" with multiple choice options matching the taxonomy in Step 4 of Part 2
  4. Add a Question node: "What would the correct answer have been?" (text input — store as variable CorrectAnswer)
  5. Add a Power Automate action node that calls the intake flow built in Part 3
  6. End with a Message node: "Thank you. Your report has been logged for review by our governance team."
  7. Save and Publish

Capturing the conversation ID and timestamp in the Power Automate call is essential — without these, you cannot reconstruct the conversation transcript from Dataverse for SEC 17a-4 evidence.

Part 2: Define the Hallucination Taxonomy

This taxonomy must be approved by Compliance before configuring tracking. Modifying categories later invalidates trend data.

Category Definition Default Severity FSI Example
Factual Error Verifiable fact stated incorrectly High Stated wrong APY for a product
Fabrication Information that does not exist in any source Critical Cited a non-existent SEC rule
Outdated Information Was correct at one time, now stale Medium Quoted last quarter's rates
Misattribution Cited the wrong source document Medium Linked to wrong policy PDF
Calculation Error Arithmetic or formula mistake High Wrong fee or interest calculation
Conflation Combined details of two distinct items Medium Mixed product features
Overconfidence Asserted certainty on uncertain matter Medium "Definitely" instead of "may"
Misleading Framing Technically true, materially misleading High Cherry-picked disclosures

Severity to SLA mapping (target — adjust per your firm's risk appetite):

Severity Definition Triage SLA Remediation SLA
Critical Customer harm, regulatory exposure, or supervisory escalation 1 hour 4 hours
High Material misinformation that could influence decisions 4 hours 24 hours
Medium Minor inaccuracy, low decision impact 1 business day 72 hours
Low Cosmetic, stylistic, or de minimis 2 business days 1 week

SLA targets are firm-defined. Document the chosen values in your Written Supervisory Procedures (WSPs) so supervisors can demonstrate consistent oversight under FINRA Rule 3110.

Part 3: Create the Hallucination Tracking List in SharePoint

Portal Path: SharePoint AI Governance site (https://<tenant>.sharepoint.com/sites/AI-Governance) → + New → List → Blank list

Step 1: Create the List

  1. From the AI Governance site home, click + New → List → Blank list
  2. Name: Hallucination Tracking
  3. Description: Intake and remediation log for AI agent hallucinations. Retain per SEC 17a-4 (6 years).
  4. Show in site navigation: Yes
  5. Click Create

Step 2: Add Required Columns

Add these columns via + Add column in list view. Match types exactly — Power BI and Power Automate flows depend on the schema.

Column Type Required Notes
Title Single line of text Yes Auto: <AgentName> - <Category>
IssueID Single line of text Yes Format HAL-YYYYMMDD-NNN (set by intake flow)
ReportDate Date and time Yes Default: Today
AgentName Single line of text Yes Pulled from conversation context
AgentEnvironment Single line of text Yes Power Platform environment ID
Zone Choice (1, 2, 3) Yes Auto-populated from environment metadata
Category Choice Yes All 8 taxonomy values from Part 2
Severity Choice (Critical, High, Medium, Low) Yes Reporter-suggested; reviewer-confirmed
ConfirmedSeverity Choice No Set by triage reviewer
UserQuery Multiple lines of text Yes Plain text
AgentResponse Multiple lines of text Yes Plain text
CorrectInformation Multiple lines of text No Captured at remediation
SourceOfTruth Hyperlink No Authoritative reference
ConversationId Single line of text Yes For Dataverse transcript lookup
ReportedBy Person or group Yes
AssignedTo Person or group No Set by triage
Status Choice (New, Triaged, In Remediation, In Validation, Closed, Won't Fix) Yes Default: New
RootCause Choice (Knowledge Gap, Prompt Issue, Source Conflict, Model Limitation, Configuration, Unknown) No Set at root cause analysis
RemediationActions Multiple lines of text No Free-text summary
ResolutionDate Date and time No Set when Status → Closed
TriageSLAMet Yes/No No Calculated by flow
RemediationSLAMet Yes/No No Calculated by flow
RelatedIncidentId Single line of text No Link to Control 3.4 incident

Step 3: Configure Retention

  1. Open List settings → Library settings → Information management policy settings (or use a Purview retention label)
  2. Apply a Purview retention label with 6 years retention from the ReportDate column (SEC 17a-4)
  3. Set the Disposition review to require approval by Purview Records Manager before deletion
  4. Verify the label applies by adding a test item and inspecting Compliance details

If your firm uses Records Management for broker-dealer evidence, apply the same retention label your existing surveillance records use. Do not rely on SharePoint's built-in retention as a stand-alone control for SEC 17a-4 evidence.

Part 4: Configure Power Automate Intake and SLA Flows

Portal Path: Power Automate → + Create

Flow 1: Hallucination Intake (HTTP-Triggered)

Type: Instant cloud flow with When an HTTP request is received trigger

  1. Create new flow HFL-Intake-3.10
  2. Trigger: When an HTTP request is received with JSON schema: 
    {
  "type": "object",
  "properties": {
    "agentName": {"type": "string"},
    "agentEnvironment": {"type": "string"},
    "conversationId": {"type": "string"},
    "userQuery": {"type": "string"},
    "agentResponse": {"type": "string"},
    "category": {"type": "string"},
    "severity": {"type": "string"},
    "reportedBy": {"type": "string"}
  }
}
  3. Action: Compose → generate IssueID = HAL- + formatDateTime(utcNow(),'yyyyMMdd') + - + rand(100,999)
  4. Action: Compose → derive Zone from environment lookup (call your environment registry)
  5. Action: Get user profile (V2) for reportedBy to resolve to person field
  6. Action: Create item in Hallucination Tracking with mapped fields, Status = New
  7. Action: Condition — if severity == Critical:
  8. HTTP call to Control 3.4 incident intake flow (or Create item in Incident Tracking list)
  9. Post message in Teams AI Governance — Critical channel tagging the AI Governance Lead and Compliance Officer
  10. Send email to agent owner and AI Administrator
  11. Action: Send email acknowledgment to reportedBy with the IssueID
  12. Save and copy the trigger URL into the Copilot Studio topic from Part 1, Step 2

Flow 2: SLA Monitor (Scheduled)

Type: Scheduled cloud flow — every 1 hour

  1. Trigger: Recurrence — every 1 hour
  2. Action: Get items from Hallucination Tracking filtered by Status ne 'Closed' and Status ne 'Won''t Fix'
  3. Action: Apply to each item:
  4. Compute elapsed hours since ReportDate
  5. Compare against SLA matrix for ConfirmedSeverity (fall back to Severity if not yet triaged)
  6. If breached and TriageSLAMet or RemediationSLAMet is empty/false:
    • Update item to set the appropriate SLA field to No
    • Post adaptive card in Teams to AssignedTo with escalation prompt
    • For Critical/High breaches, also notify the AI Governance Lead

Flow 3: Trend Detection (Scheduled Daily)

Type: Scheduled cloud flow — daily at 06:00 local

  1. Trigger: Recurrence — daily, 06:00
  2. Action: Get items from last 24 hours grouped by AgentName
  3. Action: For each agent, calculate hallucination rate using a separate query against your conversation count source (Application Insights, Copilot Studio Analytics export, or Dataverse session count)
  4. Action: Condition — if rate > firm-defined threshold (e.g., 2% for Zone 3):
  5. Post message in Teams AI Governance — Trends channel
  6. Create item in Trend Alerts list (separate list, similar schema)

Part 5: Build the Power BI Trend Dashboard

Portal Path: Power BI Service → + New → Dataset → SharePoint Online list

  1. Sign in with the AI Governance Lead account
  2. Create a new dataset connected to the Hallucination Tracking list
  3. (Optional) Add a second source for the Copilot Studio Analytics export (CSV) or Dataverse conversation transcripts to compute hallucination rate
  4. Build the report with these visuals:
Visual Field(s) Purpose
KPI: Total reports (30d) Count of IssueID Volume signal
KPI: Critical (30d) Count where ConfirmedSeverity = Critical Risk signal
KPI: MTTR (30d) Avg(ResolutionDate - ReportDate) Remediation velocity
KPI: SLA compliance % Count(TriageSLAMet = Yes) / Total Process health
Stacked column: Reports by category over time ReportDate × Category Pattern detection
Bar: Reports by agent AgentName × Count Identify problem agents
Donut: Root cause distribution RootCause Inform mitigation strategy
Table: Open critical issues Filtered to Status != Closed and ConfirmedSeverity = Critical Operational worklist
  1. Publish to a workspace named AI Governance — Pillar 3 Reporting
  2. Schedule refresh every 4 hours (Zone 3) or daily (Zone 2)
  3. Share the workspace with AI Governance Lead, Compliance Officer, and AI Administrator

Part 6: Document the Process

Create a one-page run-sheet stored alongside the SharePoint list, referencing:

  • The taxonomy and severity definitions from Part 2
  • The intake URL and flow ownership from Part 4
  • The dashboard URL from Part 5
  • Escalation contacts (AI Governance Lead, Compliance Officer, agent owners)
  • Pointer to the parent control and to Control 3.4 (incident escalation)

The run-sheet itself becomes evidence under FINRA Rule 4511 (books and records of supervisory procedures).

Validation

After completing the configuration, verify:

  1. CSAT is enabled and published on every agent in scope (capture screenshot per agent)
  2. Report Inaccurate Response topic is published and reachable from the agent
  3. Hallucination Tracking list exists with all schema columns and the Purview retention label applied
  4. Intake flow (HFL-Intake-3.10) succeeds end-to-end with a synthetic test report
  5. SLA monitor flow runs hourly and updates TriageSLAMet / RemediationSLAMet correctly
  6. Trend detection flow runs daily and posts to Teams when the threshold is exceeded
  7. Power BI dashboard loads with non-empty data after the synthetic test
  8. Critical-severity test triggers the Control 3.4 incident path

Expected Result: A user-submitted thumbs-down or report a hallucination utterance creates a tracked item within 60 seconds, applies the correct SLA, and (for Critical) escalates to incident response within the same workflow.

Sovereign Cloud Considerations

For GCC, GCC High, or DoD tenants:

  • Substitute the regional Copilot Studio, SharePoint, Power Automate, and Power BI URLs (*.gov.us, *.us, etc.)
  • Verify CSAT is available in your sovereign cloud — feature parity has historically lagged commercial; check the Copilot Studio release notes for your cloud
  • Power Automate connectors for Teams notifications must use the sovereign-cloud connector variant
  • Confirm the Purview retention label is published to the sovereign tenant before applying it

Next Steps

Back to Control 3.10

Updated: April 2026 | Version: v1.4.0 | UI Verification Status: Current