Skip to content

Portal Walkthrough: Control 2.9 - Agent Performance Monitoring and Optimization

Last Updated: April 2026 Portals: Power Platform Admin Center (PPAC), Microsoft Copilot Studio, Azure Portal (Application Insights / Log Analytics), Power BI Estimated Time: 4–6 hours for a full Zone 3 setup; ~90 min for Zone 1 baseline Audience: M365 administrators in US financial services

Prerequisites

  • Power Platform Admin role (tenant analytics, environment selection, data export)
  • AI Administrator role (per-agent analytics and Application Insights linkage in Copilot Studio)
  • Owner or Contributor on the target Azure subscription (to create Application Insights / Log Analytics workspace)
  • Power BI Pro or PPU license for any user authoring dashboards (Premium capacity recommended for Zone 3 datasets)
  • KPI thresholds documented per zone and approved by AI Governance Lead and (for Zone 3) Model Risk Manager

Step 1 — Confirm tenant analytics is enabled (Power Platform Admin Center)

  1. Open Power Platform Admin Center.
  2. Navigate to Analytics → Copilot Studio.
  3. Confirm the analytics dashboard renders for at least one environment (data may take 24–48 hours after first agent activity).
  4. Available native metrics include: sessions, conversations, resolution rate, escalation rate, customer satisfaction (CSAT), average handling time, and abandonment.

Evidence to capture: screenshot of the Analytics → Copilot Studio landing page showing your tenant name and at least one environment with non-zero session counts. Store the screenshot under maintainers-local/tenant-evidence/2.9/ (gitignored).

Step 2 — Enable per-agent analytics in Copilot Studio

  1. Open Microsoft Copilot Studio.
  2. Select an agent → Analytics in the left navigation.
  3. Review the four built-in panes:
    • Summary — sessions, engagement, satisfaction
    • Sessions — granular session data with topic resolution outcomes
    • Topics — per-topic firing rate, completion, and escalation
    • Generative AI — generative answer usage, source coverage, and (where available) quality signals
  4. For autonomous agents, also review the Autonomous pane (trigger success rate, action completion, escalation frequency).
  5. Confirm the Hybrid analytics view is enabled if the agent participates in multi-agent or Foundry-orchestrated flows.

FSI note: Native CSAT only fires when the agent prompts for it. Confirm a satisfaction survey topic is published and routed at end-of-conversation, otherwise CSAT remains empty and supervisory reviews lack required evidence.

Application Insights provides enterprise-grade telemetry (full session traces, KQL-queryable events, alert rules, retention configurable to satisfy SEC 17a-4) that the built-in dashboards do not.

  1. In the Azure Portal, create or identify an Application Insights resource:
    • Region: same geography as your tenant data residency
    • Workspace-based (links to a Log Analytics workspace — required for KQL and alerts)
    • Retention: set to at least 365 days for Zone 2; 2 years archived to immutable storage for Zone 3 to help meet SEC 17a-4(f)
  2. Copy the Connection string (preferred) or Instrumentation key.
  3. In Copilot Studio → agent → Settings → Advanced → Application Insights, paste the connection string and save.
  4. Confirm telemetry begins flowing (allow 5–15 minutes; Application Insights has ingestion latency).
  5. Repeat for every Zone 2 and Zone 3 agent. Maintain a mapping of AgentName → ApplicationInsightsResourceId in your governance inventory (linked to Control 3.1).

WORM retention

Application Insights and Log Analytics retention alone is not WORM. For SEC 17a-4(f) compliance, configure a continuous export (Diagnostic Settings → Archive to storage account with immutability policy in time-based retention mode, locked) or use Microsoft Purview Data Lifecycle Management retention locks on the storage container.

Step 4 — Configure Power Platform analytics export (Zone 2/3)

For dashboarding, longitudinal trend analysis, and cross-environment views.

  1. PPAC → Data integration → Self-service analytics.
  2. + New export:
    • Name: FSI-Agent-Analytics-Export
    • Destination: Azure Data Lake Storage Gen2 (must be in the same region as your tenant)
    • Tables: select Copilot Studio analytics tables
    • Frequency: daily incremental
  3. Save. The first export typically lands within 24 hours.
  4. In Power BI, create workspace Agent-Performance-Analytics and connect to the ADLS Gen2 export via a Dataflow Gen2 or direct Azure Data Lake Storage Gen2 connector.

Step 5 — Build the Power BI performance dashboard

Create one report per zone with these visuals (minimum):

Visual Source Purpose
KPI cards: sessions / 24h, error rate, p95 response time, CSAT Application Insights or ADLS export Operational glance
Latency percentiles (p50 / p95 / p99) over 30 days Application Insights requests / customMetrics Fed SR 26-2 (formerly SR 11-7) performance trend
Error rate by agent (stacked) Application Insights exceptions Hotspot identification
Topic resolution & escalation funnel ADLS export — topic table Completion quality
CSAT distribution + trend ADLS export — satisfaction table Customer-impact signal
Generative answer source coverage Copilot Studio Generative AI pane export Grounding evidence

Set workspace access to AI Governance Lead, Operations Team, Agent Owners, and (read-only) Internal Audit.

The pre-built Agent Usage & Performance Workbook contains a Power BI template you can import.

Step 6 — Configure threshold alerting

Built-in (Power Automate, all zones)

  1. Power Automate → + Create → Scheduled cloud flow.
  2. Name FSI-Agent-Performance-Alert, recurrence hourly (Zone 3) / daily (Zones 1–2).
  3. Action: query the Power BI dataset or ADLS export for threshold breaches.
  4. Conditions per zone:
Zone Error rate p95 response time CSAT floor
Zone 1 > 5% > 30 s n/a
Zone 2 > 2% > 15 s < 3.5 / 5
Zone 3 > 1% > 5 s < 4.0 / 5
  1. Send Teams + email to the AI Governance distribution list. For Zone 3, escalate to Model Risk Manager and on-call CISO if breach persists > 30 minutes.

Application Insights alert rules (Zone 2/3)

  1. Azure Portal → Application Insights → Alerts → Create → Alert rule.
  2. Signal: requests/failed count, requests/duration p95, custom metric hallucinationRate (if implemented).
  3. Action group: include Teams webhook, ITSM connector (ServiceNow/Jira), and on-call paging (PagerDuty/Splunk OnCall).
  4. Smart Detection (Azure Monitor) — enable for Zone 3 to catch anomalies the static thresholds miss.

Step 7 — Establish review cadence and document it

Review Frequency Attendees Required evidence
Operational Weekly (Zone 2/3); monthly (Zone 1) Ops, Agent Owners Alert log, top error topics, CSAT delta
Business Monthly AI Governance Lead, Stakeholders KPI trend, ROI workbook, optimization backlog
Model risk Quarterly (Zone 3) Model Risk Manager, AI Governance Lead Fed SR 26-2 (formerly SR 11-7) monitoring memo, hallucination/grounding metrics, drift analysis
Executive Quarterly Leadership, Compliance, Internal Audit Cross-zone scorecard, regulatory posture, incidents

Document the schedule in your supervisory procedures (FINRA Rule 3110) and store meeting minutes on WORM-capable SharePoint or Purview-retained storage.

Configuration by Governance Level

Setting Baseline (Zone 1) Recommended (Zone 2) Regulated (Zone 3)
Native analytics Enabled Enabled + per-agent review Enabled + Hybrid + Autonomous panes
Application Insights Optional Required Required + custom RAI events
Analytics export Optional ADLS export daily ADLS export + WORM archive
Dashboard Summary Per-agent detail Real-time + drill-down + MRM tab
Alerting Error rate only Error + latency + CSAT All metrics + Smart Detection + paging
Review cadence Monthly Weekly + monthly + quarterly Daily + weekly + monthly + quarterly MRM
Telemetry retention 30 days 365 days 2 years on WORM storage

Validation checklist

  • PPAC → Analytics → Copilot Studio renders data
  • Each Zone 2/3 agent shows a configured Application Insights resource ID
  • ADLS export shows recent files
  • Power BI workspace Agent-Performance-Analytics exists with at least one published report and a recent refresh
  • At least one alert rule fires successfully against a temporarily lowered threshold
  • Review cadence is documented in supervisory procedures with named attendees
  • Evidence artifacts staged under maintainers-local/tenant-evidence/2.9/ for the next examination

Back to Control 2.9 | PowerShell Setup | Verification & Testing | Troubleshooting

Updated: May 2026 | Version: v1.6.2 | UI Verification Status: Current