Control 3.2: Usage Analytics and Activity Monitoring
Control ID: 3.2 Pillar: Reporting Regulatory Reference: FINRA 4511, SEC 17a-3/4, SOX 404, GLBA 501(b) Last UI Verified: February 2026 Governance Levels: Baseline / Recommended / Regulated Last Verified: 2026-02-03
Agent 365 Architecture Update
Agent 365 Observability provides rich usage analytics via Application Insights, offering consolidated activity monitoring across all agent types through a single telemetry pipeline. See Unified Agent Governance for observability and usage analytics details.
Objective
Establish comprehensive monitoring of AI agent usage, performance, and activity patterns through Power Platform Admin Center dashboards, alerts, and audit integration to support governance, compliance, and operational optimization.
Why This Matters for FSI
- FINRA 4511: Monitor AI system activity for books and records compliance
- SEC 17a-3/4: Track usage of systems processing customer data for examination readiness
- SOX 404: Monitor systems involved in financial reporting for control evidence
- GLBA 501(b): Track access to customer information for safeguards compliance
Control Description
Usage analytics and activity monitoring form the backbone of effective AI governance. The Power Platform Admin Center provides a comprehensive Monitor section with:
| Capability | Description | FSI Relevance |
|---|---|---|
| Monitor Overview | Centralized view of platform health | Quick governance status |
| Alerts (Preview) | Pre-built and custom alert rules | Proactive issue detection |
| Logs | Activity and error logging | Audit trail and troubleshooting |
| Copilot Studio Dashboard | Agent-specific success metrics | Agent performance monitoring |
For environments with Managed Environments enabled, additional usage insights including weekly digests and adoption trends are available.
New Analytics Features (2025-2026)
Microsoft has expanded analytics capabilities significantly:
| Feature | Status | Release | Description |
|---|---|---|---|
| Agent Dashboard | GA | Ignite 2025 | Centralized agent adoption measurement across M365 |
| Action Usage Analytics | GA | November 2025 | Connector and API call tracking per agent |
| Copilot Benchmarks | GA | Ignite 2025 | Peer comparison metrics across industry verticals |
| Copilot Chat Insights | Expanded | February 2026 | 50-license minimum removed; available to all tenants |
| New Usage Page (PPAC) | Preview | January 2026 | Unified usage dashboard with drill-down capabilities |
Data Availability
- DAU/MAU metrics require user authentication; anonymous usage is not tracked
- Native retention for PPAC analytics is 28 days; export to Log Analytics for extended retention
- Settings changes may take up to 8 hours to reflect in dashboards
Custom Power BI Analytics Infrastructure
For organizations requiring analytics beyond native PPAC capabilities (extended retention, cross-environment correlation, custom metrics), implement a custom Power BI pipeline:
Dataverse (Agent Metadata)
↓
Synapse Link
↓
Azure Data Lake
↓
Power BI
Infrastructure Components:
| Component | Purpose | FSI Consideration |
|---|---|---|
| Dataverse | Source for agent metadata, conversation logs, custom telemetry | Already included with Copilot Studio |
| Synapse Link for Dataverse | Real-time data export to Data Lake | Requires Azure subscription; Premium capacity recommended |
| Azure Data Lake Storage Gen2 | Long-term storage for compliance retention | 7-year retention for SEC 17a-4; immutable storage optional |
| Power BI Premium | Enterprise reporting with large dataset support | Recommended for datasets >1GB |
When to Implement Custom Pipeline:
| Requirement | Native PPAC | Custom Pipeline |
|---|---|---|
| 28-day retention | ✅ | ✅ |
| 1+ year retention | ❌ | ✅ |
| Cross-environment correlation | ❌ | ✅ |
| Custom KPIs/metrics | Limited | ✅ |
| Regulatory evidence export | Manual | Automated |
| Real-time dashboards | Limited | ✅ |
Licensing Requirements
Synapse Link requires Azure subscription. Power BI Premium or Premium Per User licensing recommended for large-scale deployments. Confirm costs before implementation.
Key Configuration Points
- Enable access to PPAC Monitor section for governance personnel
- Configure pre-built Microsoft alert rules (e.g., "High-use agents have success rate under 90%")
- Create custom alerts for organization-specific thresholds
- Set up weekly digest notifications for Managed Environments
- Configure audit log integration for compliance reporting
- Establish dashboard review cadence by governance tier
Zone-Specific Requirements
| Zone | Requirement | Rationale |
|---|---|---|
| Zone 1 (Personal) | Monthly dashboard review; >80% success rate target | Baseline visibility, low risk |
| Zone 2 (Team) | Weekly dashboard review; >90% success rate; enable alerts | Shared agents require accountability |
| Zone 3 (Enterprise) | Daily review; >95% success rate; real-time alerting; executive reporting | Customer-facing, highest regulatory scrutiny |
Roles & Responsibilities
| Role | Responsibility |
|---|---|
| Power Platform Admin | Configure monitoring dashboards and alert rules |
| Compliance Officer | Review compliance reports, validate audit log access |
| AI Governance Lead | Define monitoring requirements, review executive summaries |
| Operations Team | Respond to alerts, investigate performance issues |
Related Controls
| Control | Relationship |
|---|---|
| 2.1 - Managed Environments | Required for usage insights |
| 1.7 - Audit Logging | Provides activity tracking foundation |
| 3.1 - Agent Inventory | Correlates metrics to agent inventory |
| 3.3 - Compliance Reporting | Compliance Dashboard integrates usage metrics (Compliance Dashboard) |
| 3.4 - Incident Reporting | Alert-driven incident response |
Implementation Playbooks
Step-by-Step Implementation
This control has detailed playbooks for implementation, automation, testing, and troubleshooting:
- Portal Walkthrough — Step-by-step portal configuration
- PowerShell Setup — Automation scripts
- Verification & Testing — Test cases and evidence collection
- Troubleshooting — Common issues and resolutions
Agent Usage & Performance Workbook
For organizations where ALM separation-of-duties policies restrict direct access to the Copilot Studio Analytics tab, the Agent Usage & Performance Workbook provides Azure Monitor-based usage analytics, adoption metrics, and business value estimation — without requiring Copilot Studio portal access. See the Deployment Guide for setup instructions.
Verification Criteria
Confirm control effectiveness by verifying:
- PPAC Monitor section is accessible to designated administrators
- Pre-built alert rules are enabled and configured with appropriate recipients
- Custom alerts are created for Zone 3 success rate thresholds (<95%)
- Weekly digest emails are being received for Managed Environments
- Audit logs capture agent activity with expected retention
- Dashboard review cadence is documented and followed
Daily Operational Monitoring
While weekly and monthly reporting address strategic oversight, Zone 2/3 environments require daily operational monitoring to detect and respond to deny events, content filtering, and policy blocks in near-real-time.
Deny Event Categories
| Category | Source | Description | FSI Relevance |
|---|---|---|---|
| Policy Block | Purview Audit (CopilotInteraction) | DLP policy prevented resource access | NPI (Non-Public Information) protection evidence |
| XPIA Detection | Purview Audit (CopilotInteraction) | Cross-prompt injection attempt detected | Security incident tracking |
| Jailbreak Attempt | Purview Audit (CopilotInteraction) | User attempted to bypass agent guardrails | Adversarial input logging |
| RAI Content Filter | Application Insights (ContentFiltered) | Responsible AI filter blocked response | Model risk control evidence |
| DLP Match | Purview Audit (DlpRuleMatch) | Sensitivity-based blocking in Copilot location | Data governance enforcement |
Daily Monitoring Cadence
| Zone | Monitoring Requirement | Response SLA |
|---|---|---|
| Zone 1 | Weekly summary (optional) | Best effort |
| Zone 2 | Daily automated report | 4-hour response |
| Zone 3 | Daily automated report + real-time alerts | 15-minute response |
Implementation
For organizations requiring daily deny event correlation across Purview Audit, DLP, and Application Insights:
- Deny Event Correlation Report Playbook - Multi-source correlation with Power BI dashboard
Environment Provisioning Monitoring
For baseline configuration that enables usage insights from environment creation:
- Environment Lifecycle Management - Environments created with usage insights enabled automatically
Additional Resources
- Power Platform Admin Center Monitoring
- Set Up Alerts in PPAC
- Managed Environment Usage Insights
- Copilot Studio Analytics
- Unified Audit Log
Microsoft Audit Reporting Tools
For enhanced Copilot/AI reporting beyond native M365 Admin Center capabilities, see:
- Microsoft Audit Reporting Tools Playbook - AI-in-One Dashboard and PAX (Portable Audit eXporter) for enterprise-scale analytics
Agent 365 Observability SDK (Preview)
Preview Notice
Microsoft Agent 365 SDK and Agent Essentials are in limited preview (Frontier program). Verify feature availability and GA timelines before implementing production controls dependent on these capabilities. Expect changes before general availability.
Agent 365 SDK introduces OpenTelemetry-based observability for Blueprint-registered agents, providing FSI-compliant telemetry capture and export capabilities.
Observability SDK Architecture:
Agent 365 SDK Application
↓
Observability SDK
↓
OpenTelemetry Collector
↓ ↓
Azure Monitor Third-Party SIEM
↓
Application Insights → DSPM Activity Explorer
Key Capabilities for FSI Compliance:
| Capability | Description | Regulatory Alignment |
|---|---|---|
| Structured Telemetry | Captures agent interactions with consistent schema | FINRA 4511 records requirements |
| Prompt/Response Logging | Full conversation capture for audit trail | FINRA 4511 recordkeeping requirements |
| Correlation IDs | Links multi-turn conversations and agent handoffs | SEC 17a-3 transaction tracing |
| Export Formats | OTLP, JSON, compliance-ready formats | SOX 404 evidence requirements |
Developer Instrumentation Guidance:
For Zone 2-3 agents built with Agent 365 SDK, implement the following instrumentation:
# Example: Agent 365 SDK Observability Configuration
from agent365.observability import TelemetryClient
telemetry = TelemetryClient(
connection_string="<Application-Insights-Connection-String>",
enable_prompt_logging=True, # Required for Zone 3
enable_pii_scrubbing=True, # Recommended for all zones
retention_days=365 # Align with regulatory requirements
)
# Instrument agent interactions
@telemetry.trace_interaction
async def process_request(request):
# Agent logic here
pass
OpenTelemetry Integration Patterns:
| Pattern | Use Case | Implementation |
|---|---|---|
| Direct Export | Single Application Insights instance | Configure SDK connection string |
| Collector Sidecar | Multi-destination routing | Deploy OTEL Collector with exporters |
| Dapr Integration | Kubernetes deployments | Use Dapr observability building block |
Zone-Specific Observability Requirements:
| Zone | Telemetry Requirement | Retention |
|---|---|---|
| Zone 1 | Basic metrics (optional) | 90 days |
| Zone 2 | Full telemetry with prompt logging | 1 year |
| Zone 3 | Full telemetry + real-time alerting + compliance export | 7–10 years |
Integration with DSPM for AI:
Observability SDK telemetry flows into DSPM Activity Explorer when: 1. Application Insights is configured as the telemetry destination 2. DSPM Extended Insights is enabled (see Control 1.6) 3. Agent 365 workload is included in DSPM scope
This enables sensitive data detection in agent conversations and supports oversharing assessments.
Additional Resources:
- Microsoft Learn: Agent 365 SDK Overview (Preview) - Observability and telemetry capabilities
- Microsoft Learn: OpenTelemetry with Azure Monitor - OpenTelemetry integration patterns
Observability by Agent Type
(A) Copilot Studio Agents:
- Power Platform Admin Center analytics
- Managed Environment insights
- Microsoft Purview Audit logs
- Application Insights (requires explicit enablement — see below)
Copilot Studio Application Insights: Sensitive Properties Required
When configuring Application Insights for Copilot Studio agents, two settings control whether conversation content is included in telemetry:
| Setting | Location | Effect |
|---|---|---|
| Log activities | Copilot Studio > Agent > Settings > Advanced > Application Insights | Enables basic (sanitized) telemetry flow |
| Log sensitive activity properties | Copilot Studio > Agent > Settings > Advanced > Application Insights | Includes conversation text, user IDs, and node details in telemetry |
| Allow conversation transcripts | PPAC > Environment > Settings > Product > Features | Tenant-level prerequisite for transcript storage |
Without "Log sensitive activity properties" enabled, queries against the customEvents table (e.g., BotMessage, UserMessage events) will return records with empty text fields — creating a false sense of audit coverage.
For FSI organizations relying on Application Insights for conversation-level monitoring (Zone 2-3), both the agent-level and environment-level settings must be enabled. PII governance controls should be applied to the Application Insights resource.
(B) Agent 365 SDK Agents (Preview):
- OpenTelemetry SDK integration
- Application Insights workbooks
- Custom telemetry configuration
Updated: February 2026 | Version: v1.2 | UI Verification Status: Current