Control 3.2: Usage Analytics and Activity Monitoring
Control ID: 3.2
Pillar: Reporting
Regulatory Reference: FINRA 4511, FINRA 25-07, SEC 17a-3/4 (incl. 17a-4(f) electronic records), SOX 404, GLBA 501(b)
Last UI Verified: April 2026
Governance Levels: Baseline / Recommended / Regulated
Agent 365 Architecture Update
Agent 365 Observability provides rich usage analytics via Application Insights, offering consolidated activity monitoring across all agent types through a single telemetry pipeline. See Unified Agent Governance for observability and usage analytics details.
Objective
Establish comprehensive monitoring of AI agent usage, performance, and activity patterns through Power Platform Admin Center dashboards, alerts, and audit integration to support governance, compliance, and operational optimization.
Why This Matters for FSI
- FINRA 4511 / 25-07: Books-and-records and AI-specific recordkeeping require monitored evidence of agent activity, prompt/response capture, and supervisory review trails. Native PPAC retention (28 days) does not meet FINRA's six-year requirement on its own — pair with extended retention (Purview, Application Insights, or custom pipeline).
- SEC 17a-3 / 17a-4(f): Examination readiness depends on time-indexed, tamper-evident records of customer-data processing. SEC 17a-4(f) accepts electronic records when stored in non-rewriteable, non-erasable form (or with audit-trail equivalents under the 2022 amendments) for the prescribed retention period (generally 3–6 years).
- SOX 404: Monitoring of agents that touch financial-reporting data produces the operating-effectiveness evidence required for management's assertion and external-auditor walkthroughs.
- GLBA 501(b): Tracks access to customer NPI inside agent conversations and downstream connector calls; supports the "monitor" element of the Safeguards Rule risk-assessment lifecycle.
- OCC 2011-12 / Fed SR 11-7: Model-risk management expects ongoing monitoring of model performance, drift, and operational issues — usage analytics, success rate, and deny-event monitoring contribute to that evidence base.
Automation Available
See Compliance Dashboard in FSI-AgentGov-Solutions for aggregated compliance reporting across the framework control catalog with zone-based filtering.
Control Description
Usage analytics and activity monitoring form the backbone of effective AI governance. The Power Platform Admin Center provides a comprehensive Monitor section with:
| Capability | Description | FSI Relevance |
|---|---|---|
| Monitor Overview | Centralized view of platform health | Quick governance status |
| Alerts (Preview) | Pre-built and custom alert rules | Proactive issue detection |
| Logs | Activity and error logging | Audit trail and troubleshooting |
| Copilot Studio Dashboard | Agent-specific success metrics | Agent performance monitoring |
For environments with Managed Environments enabled, additional usage insights including weekly digests and adoption trends are available.
New Analytics Features (2025-2026)
Microsoft has expanded analytics capabilities significantly:
| Feature | Status | Release | Description |
|---|---|---|---|
| Agent Dashboard | GA | Ignite 2025 | Centralized agent adoption measurement across M365 |
| Action Usage Analytics | GA | November 2025 | Connector and API call tracking per agent |
| Copilot Benchmarks | GA | Ignite 2025 | Peer comparison metrics across industry verticals |
| Copilot Chat Insights | Expanded | February 2026 | 50-license minimum removed; available to all tenants |
| New Usage Page (PPAC) | Preview | January 2026 | Unified usage dashboard with drill-down capabilities |
Data Availability
- DAU/MAU metrics require user authentication; anonymous usage is not tracked
- Native retention for PPAC analytics is 28 days; export to Log Analytics for extended retention
- Settings changes may take up to 8 hours to reflect in dashboards
Custom Power BI Analytics Infrastructure
For organizations requiring analytics beyond native PPAC capabilities (extended retention, cross-environment correlation, custom metrics), implement a custom Power BI pipeline:
Infrastructure Components:
| Component | Purpose | FSI Consideration |
|---|---|---|
| Dataverse | Source for agent metadata, conversation logs, custom telemetry | Already included with Copilot Studio |
| Synapse Link for Dataverse | Real-time data export to Data Lake | Requires Azure subscription; Premium capacity recommended |
| Azure Data Lake Storage Gen2 | Long-term storage for compliance retention | 7-year retention for SEC 17a-4; immutable storage optional |
| Power BI Premium | Enterprise reporting with large dataset support | Recommended for datasets >1GB |
When to Implement Custom Pipeline:
| Requirement | Native PPAC | Custom Pipeline |
|---|---|---|
| 28-day retention | ✅ | ✅ |
| 1+ year retention | ❌ | ✅ |
| Cross-environment correlation | ❌ | ✅ |
| Custom KPIs/metrics | Limited | ✅ |
| Regulatory evidence export | Manual | Automated |
| Real-time dashboards | Limited | ✅ |
Licensing Requirements
Synapse Link requires Azure subscription. Power BI Premium or Premium Per User licensing recommended for large-scale deployments. Confirm costs before implementation.
Key Configuration Points
- Enable access to PPAC Monitor section for governance personnel
- Configure pre-built Microsoft alert rules (e.g., "High-use agents have success rate under 90%")
- Create custom alerts for organization-specific thresholds
- Set up weekly digest notifications for Managed Environments
- Configure audit log integration for compliance reporting
- Establish dashboard review cadence by governance tier
Zone-Specific Requirements
| Zone | Requirement | Rationale |
|---|---|---|
| Zone 1 (Personal) | Monthly dashboard review; >80% success-rate target; rely on native PPAC + Purview Audit (Standard) retention (≥180 days where licensed). | Baseline visibility for low-risk personal productivity use; FSI books-and-records reach for these agents is limited. |
| Zone 2 (Team) | Weekly dashboard review; >90% success rate; pre-built alerts enabled with named recipients; Purview Audit retention extended to ≥1 year; daily deny-event report (Policy Block, DLP Match, RAI ContentFiltered). | Shared agents create supervisory obligations under FINRA 3110/4511; weekly cadence and alerting support timely supervisory review. |
| Zone 3 (Enterprise) | Daily dashboard review; >95% success rate; real-time alerting with documented response SLAs; executive monthly reporting; conversation-level telemetry (Application Insights "Log sensitive activity properties" enabled with PII governance); 6–7 year retention via Purview long-term retention or custom Synapse Link → ADLS pipeline aligned with SEC 17a-4(f). | Customer-facing and regulated workloads require examination-ready evidence with the longest retention horizon; near-real-time detection supports incident-response SLAs and OCC 2011-12 ongoing monitoring expectations. |
Roles & Responsibilities
| Role | Responsibility |
|---|---|
| Power Platform Admin | Configure PPAC Monitor dashboards, alert rules, Managed Environment usage insights, and Copilot Studio analytics access. |
| Purview Audit Admin | Configure audit log retention policies (Standard / Premium), validate Copilot/PowerApps RecordType ingestion, and grant Purview Audit Reader access for evidence collection. |
| Purview Compliance Admin | Validate audit log search access, configure long-term retention policies aligned with FINRA/SEC requirements. |
| AI Governance Lead | Define monitoring requirements, dashboard cadence, and KPI thresholds by zone; review executive summaries. |
| Compliance Officer | Review compliance reports, validate evidence completeness, attest to monitoring posture for examinations. |
| Operations Team | Respond to alerts within zone-specific SLA; investigate performance and deny-event spikes; document incident handoffs. |
Related Controls
| Control | Relationship |
|---|---|
| 2.1 - Managed Environments | Required for usage insights |
| 1.7 - Audit Logging | Provides activity tracking foundation |
| 3.1 - Agent Inventory | Correlates metrics to agent inventory |
| 3.3 - Compliance Reporting | Compliance Dashboard integrates usage metrics (Compliance Dashboard) |
| 3.4 - Incident Reporting | Alert-driven incident response |
Implementation Playbooks
Step-by-Step Implementation
This control has detailed playbooks for implementation, automation, testing, and troubleshooting:
- Portal Walkthrough — Step-by-step portal configuration
- PowerShell Setup — Automation scripts
- Verification & Testing — Test cases and evidence collection
- Troubleshooting — Common issues and resolutions
Agent Usage & Performance Workbook
For organizations where ALM separation-of-duties policies restrict direct access to the Copilot Studio Analytics tab, the Agent Usage & Performance Workbook provides Azure Monitor-based usage analytics, adoption metrics, and business value estimation — without requiring Copilot Studio portal access. See the Deployment Guide for setup instructions.
Verification Criteria
Confirm control effectiveness by verifying:
- PPAC Monitor section is accessible to designated administrators (Power Platform Admin role).
- Pre-built alert rules are enabled and routed to named recipients with documented response SLAs.
- Custom alerts are created for Zone 3 success-rate thresholds (<95%), after-hours activity, and customer-data access anomalies.
- Weekly digest emails are received for Managed Environments and reach the compliance distribution.
- Purview Audit captures Copilot Studio / PowerApps RecordType events and retention is configured to meet zone requirements (Zone 2 ≥1 year; Zone 3 ≥6 years aligned with FINRA 4511 / SEC 17a-4).
- For Zone 2/3, daily deny-event reporting (Policy Block, DLP Match, RAI ContentFiltered, XPIA, Jailbreak) is operational and reviewed within SLA.
- Where Application Insights is used for conversation-level monitoring (Zone 2/3), both the agent-level "Log sensitive activity properties" and the environment-level "Allow conversation transcripts" settings are enabled, and PII governance controls are applied to the App Insights resource.
- Dashboard review cadence is documented per zone, and review attestations are retained as audit evidence.
Daily Operational Monitoring
While weekly and monthly reporting address strategic oversight, Zone 2/3 environments require daily operational monitoring to detect and respond to deny events, content filtering, and policy blocks in near-real-time.
Deny Event Categories
| Category | Source | Description | FSI Relevance |
|---|---|---|---|
| Policy Block | Purview Audit (CopilotInteraction) | DLP policy prevented resource access | NPI (Non-Public Information) protection evidence |
| XPIA Detection | Purview Audit (CopilotInteraction) | Cross-prompt injection attempt detected | Security incident tracking |
| Jailbreak Attempt | Purview Audit (CopilotInteraction) | User attempted to bypass agent guardrails | Adversarial input logging |
| RAI Content Filter | Application Insights (ContentFiltered) | Responsible AI filter blocked response | Model risk control evidence |
| DLP Match | Purview Audit (DlpRuleMatch) | Sensitivity-based blocking in Copilot location | Data governance enforcement |
Daily Monitoring Cadence
| Zone | Monitoring Requirement | Response SLA |
|---|---|---|
| Zone 1 | Weekly summary (optional) | Best effort |
| Zone 2 | Daily automated report | 4-hour response |
| Zone 3 | Daily automated report + real-time alerts | 15-minute response |
Implementation
For organizations requiring daily deny event correlation across Purview Audit, DLP, and Application Insights:
- Deny Event Correlation Report Playbook - Multi-source correlation with Power BI dashboard
Environment Provisioning Monitoring
For baseline configuration that enables usage insights from environment creation:
- Environment Lifecycle Management - Environments created with usage insights enabled automatically
Additional Resources
- Power Platform Admin Center Monitoring
- Set Up Alerts in PPAC
- Managed Environment Usage Insights
- Copilot Studio Analytics
- Unified Audit Log
Microsoft Audit Reporting Tools
For enhanced Copilot/AI reporting beyond native M365 Admin Center capabilities, see:
- Microsoft Audit Reporting Tools Playbook - AI-in-One Dashboard and PAX (Portable Audit eXporter) for enterprise-scale analytics
Agent 365 Observability SDK (Preview)
Preview Notice
Microsoft Agent 365 SDK and Agent Essentials are in limited preview (Frontier program). Verify feature availability and GA timelines before implementing production controls dependent on these capabilities. Expect changes before general availability.
Agent 365 SDK introduces OpenTelemetry-based observability for Blueprint-registered agents, providing FSI-compliant telemetry capture and export capabilities.
Observability SDK Architecture:
Agent 365 SDK Application
↓
Observability SDK
↓
OpenTelemetry Collector
↓ ↓
Azure Monitor Third-Party SIEM
↓
Application Insights → DSPM Activity Explorer
Key Capabilities for FSI Compliance:
| Capability | Description | Regulatory Alignment |
|---|---|---|
| Structured Telemetry | Captures agent interactions with consistent schema | FINRA 4511, FINRA 25-07 records requirements |
| Prompt/Response Logging | Full conversation capture for audit trail | FINRA 4511, FINRA 25-07 recordkeeping requirements |
| Correlation IDs | Links multi-turn conversations and agent handoffs | SEC 17a-3 transaction tracing |
| Export Formats | OTLP, JSON, compliance-ready formats | SOX 404 evidence requirements |
Developer Instrumentation Guidance:
For Zone 2-3 agents built with Agent 365 SDK, implement the following instrumentation:
# Example: Agent 365 SDK Observability Configuration
from agent365.observability import TelemetryClient
telemetry = TelemetryClient(
connection_string="<Application-Insights-Connection-String>",
enable_prompt_logging=True, # Required for Zone 3
enable_pii_scrubbing=True, # Recommended for all zones
retention_days=365 # Align with regulatory requirements
)
# Instrument agent interactions
@telemetry.trace_interaction
async def process_request(request):
# Agent logic here
pass
OpenTelemetry Integration Patterns:
| Pattern | Use Case | Implementation |
|---|---|---|
| Direct Export | Single Application Insights instance | Configure SDK connection string |
| Collector Sidecar | Multi-destination routing | Deploy OTEL Collector with exporters |
| Dapr Integration | Kubernetes deployments | Use Dapr observability building block |
Zone-Specific Observability Requirements:
| Zone | Telemetry Requirement | Retention |
|---|---|---|
| Zone 1 | Basic metrics (optional) | 90 days |
| Zone 2 | Full telemetry with prompt logging | 1 year |
| Zone 3 | Full telemetry + real-time alerting + compliance export | 7–10 years |
Integration with DSPM for AI:
Observability SDK telemetry flows into DSPM Activity Explorer when: 1. Application Insights is configured as the telemetry destination 2. DSPM Extended Insights is enabled (see Control 1.6) 3. Agent 365 workload is included in DSPM scope
This enables sensitive data detection in agent conversations and supports oversharing assessments.
Additional Resources:
- Microsoft Learn: Agent 365 SDK Overview (Preview) - Observability and telemetry capabilities
- Microsoft Learn: OpenTelemetry with Azure Monitor - OpenTelemetry integration patterns
Observability by Agent Type
(A) Copilot Studio Agents:
- Power Platform Admin Center analytics
- Managed Environment insights
- Microsoft Purview Audit logs
- Application Insights (requires explicit enablement — see below)
Copilot Studio Application Insights: Sensitive Properties Required
When configuring Application Insights for Copilot Studio agents, two settings control whether conversation content is included in telemetry:
| Setting | Location | Effect |
|---|---|---|
| Log activities | Copilot Studio > Agent > Settings > Advanced > Application Insights | Enables basic (sanitized) telemetry flow |
| Log sensitive activity properties | Copilot Studio > Agent > Settings > Advanced > Application Insights | Includes conversation text, user IDs, and node details in telemetry |
| Allow conversation transcripts | PPAC > Environment > Settings > Product > Features | Tenant-level prerequisite for transcript storage |
Without "Log sensitive activity properties" enabled, queries against the customEvents table (e.g., BotMessage, UserMessage events) will return records with empty text fields — creating a false sense of audit coverage.
For FSI organizations relying on Application Insights for conversation-level monitoring (Zone 2-3), both the agent-level and environment-level settings must be enabled. PII governance controls should be applied to the Application Insights resource.
(B) Agent 365 SDK Agents (Preview):
- OpenTelemetry SDK integration
- Application Insights workbooks
- Custom telemetry configuration
Updated: April 2026 | Version: v1.4.0 | UI Verification Status: Current