Skip to content

Control 3.2: Usage Analytics and Activity Monitoring

Overview

Control ID: 3.2 Control Name: Usage Analytics and Activity Monitoring Regulatory Reference: FINRA 4511, SEC 17a-3/4, SOX 404, GLBA 501(b) Setup Time: 1-2 hours

Purpose

Usage analytics and activity monitoring form the backbone of effective AI governance, enabling organizations to maintain visibility into agent behavior, performance, and compliance posture. For financial services institutions, comprehensive monitoring is essential to demonstrate regulatory compliance, detect anomalies that may indicate operational or security issues, and optimize agent performance for customer-facing interactions. This control establishes the monitoring framework necessary to support audit readiness and continuous improvement.

Description

Monitoring agent usage and activity is essential for governance, performance optimization, and regulatory compliance. The Power Platform Admin Center provides a comprehensive Monitor section with alerts, logs, and product-specific dashboards including Copilot Studio agent health metrics.

See Power Platform Admin Center monitoring for detailed capabilities.

Key Capabilities

Capability Description FSI Relevance
Monitor Overview Centralized view of platform health Quick governance status
Alerts (Preview) Pre-built and custom alert rules Proactive issue detection
Logs Activity and error logging Audit trail and troubleshooting
Copilot Studio Dashboard Agent-specific success metrics Agent performance monitoring

Prerequisites

Primary Owner Admin Role: Power Platform Admin Supporting Roles: None

Licenses Required

License Requirement Level Purpose
Power Platform Premium Required Access to PPAC monitoring features
Microsoft 365 E5 Required Unified audit log and advanced analytics
Microsoft 365 E5 Compliance Recommended Enhanced compliance reporting
Copilot Studio Required Agent-specific analytics access

Permissions Required

Role Scope Purpose
Power Platform Administrator Tenant Full access to monitoring dashboards
Copilot Studio Admin Environment Agent-specific metrics and configuration
Compliance Administrator Tenant Audit log access and compliance reporting
Global Reader Tenant Read-only access to monitoring data
Environment Admin Environment Environment-level monitoring

Dependencies

Dependency Control Reference Status Check
Managed Environments Control 2.1 Required for usage insights
Audit Logging Control 1.7 Required for activity tracking
Agent Inventory Control 3.1 Recommended for correlation
Environment Strategy Control 2.2 Required for zone-based monitoring

Pre-Setup Checklist

  • [ ] Power Platform Admin Center access confirmed
  • [ ] Required licenses assigned to monitoring personnel
  • [ ] Managed Environments enabled for target environments
  • [ ] Audit logging configured and validated
  • [ ] Alert notification recipients identified
  • [ ] Monitoring review schedule established
  • [ ] Zone classification for agents documented
  • [ ] Retention policies aligned with regulatory requirements

Governance Levels

Level 1 - Baseline

Requirement Configuration
Monitor access PPAC Monitor section accessible
Basic metrics Agent session success rate tracked
Review frequency Monthly review of metrics

Minimum requirements:

  • Access PPAC Monitor section regularly
  • Review Copilot Studio dashboard monthly
  • Document any performance issues
Requirement Configuration
Pre-built alerts Enable Microsoft pre-built alert rules
Custom alerts Create organization-specific alerts
Dashboard review Weekly monitoring dashboard review
Trend analysis Track metrics over time

FSI recommendations:

  • Enable "High-use agents have success rate under 90%" alert
  • Create custom alerts for Tier 2+ (team/enterprise-managed) agents
  • Weekly dashboard reviews for compliance
  • Document and investigate all alerts

Level 4 - Regulated/High-Risk

Requirement Configuration
Real-time monitoring Daily dashboard reviews
Comprehensive alerting All relevant pre-built + custom alerts
Executive reporting Monthly metrics reports to governance committee
SLA tracking Monitor against defined SLAs
Incident integration Alerts integrated with incident management

FSI requirements:

  • Daily review of Tier 3 (enterprise managed) agent metrics
  • Immediate investigation of success rate drops
  • Monthly executive summary with trend analysis
  • Integration with GRC tools if available
  • Document all alert responses

Usage Insights (Managed Environments)

For environments with Managed Environment enabled, additional usage insights are available:

Insight Description Recipients
Weekly digest Email summary of top resources Admins
Additional recipients Compliance team inclusion Configurable
Usage trends Adoption and activity patterns Dashboard

Configuring Usage Insights

  1. Navigate to environment in PPAC
  2. Edit Managed Environments settings
  3. Check "Include insights for this environment in the weekly email digest"
  4. Add compliance team to additional recipients

See Usage insights for details.

Setup & Configuration

PPAC Monitor Section

Accessing Monitor

  1. Open Power Platform Admin Center
  2. Navigate to Monitor in left navigation
  3. Select the appropriate sub-section:
  4. Overview - Platform health summary
  5. Alerts (Preview) - Alert rules and notifications
  6. Logs - Activity and error logs
  7. Copilot Studio - Agent-specific metrics

Monitor Overview

The Overview page provides:

Component Description Governance Use
Summary cards Key metrics at a glance Quick health check
Custom alerts Organization-specific alerts Tailored monitoring
Product links Direct access to product dashboards Drill-down analysis

Alerts (Preview)

The Alerts section enables proactive monitoring through pre-built and custom rules.

Pre-Built Microsoft Rules

Microsoft provides pre-configured alert rules for common scenarios:

Alert Rule Description FSI Relevance
High-use agents have a success rate under 90% Monitors heavily used agents for degraded performance Critical for customer-facing agents
Environment capacity alerts Warns when environments approach limits Capacity planning
Security-related alerts Detects potential security concerns Compliance monitoring

Creating Custom Alerts

  1. Navigate to Monitor → Alerts
  2. Click + New alert rule
  3. Configure alert conditions and thresholds
  4. Set notification recipients
  5. Save and enable the rule

FSI Recommendation: Create alerts for:

  • Agent success rate drops below 95% (Tier 3 / enterprise-managed)
  • Unusual usage patterns
  • High error rates in production environments

See Set up alerts for configuration details.

Logs

The Logs section provides detailed activity information for troubleshooting and audit purposes.

Log Type Description Retention
Activity logs User and admin actions Per retention policy
Error logs System and application errors Per retention policy
Audit logs Security-relevant events Per compliance requirements

FSI Use Cases:

  • Investigate agent failures
  • Track configuration changes
  • Support regulatory examinations
  • Identify error patterns

Copilot Studio Dashboard

The Copilot Studio-specific monitoring page provides agent health metrics:

Metric Description Target
Agent session success rate Percentage of successful sessions >95% for Tier 3 (enterprise managed)
Recent sessions List of recent agent interactions Real-time monitoring
Environment Environment hosting the agent Location tracking
Managed Whether environment is managed Governance status

Key Performance Indicators

KPI Tier 1 Target Tier 2 Target Tier 3 Target
Success rate >80% >90% >95%
Response time <10s <5s <3s
Availability 95% 99% 99.9%

PowerShell Configuration

Get Usage Metrics from PPAC

# Connect to Power Platform Admin Center
Connect-PowerPlatformAdmin

# Get all environments with usage data
$environments = Get-AdminPowerAppEnvironment

# Retrieve usage metrics for each environment
foreach ($env in $environments) {
    Write-Host "Environment: $($env.DisplayName)" -ForegroundColor Cyan

    # Get environment-level analytics
    $analytics = Get-AdminPowerAppEnvironmentAnalytics -EnvironmentName $env.EnvironmentName

    Write-Host "  Active Users: $($analytics.ActiveUsers)"
    Write-Host "  Total Sessions: $($analytics.TotalSessions)"
    Write-Host "  Last Updated: $($analytics.LastUpdated)"
}

Export Copilot Studio Analytics

# Get Copilot Studio agent analytics
function Export-CopilotStudioAnalytics {
    param(
        [string]$EnvironmentId,
        [string]$OutputPath = ".\CopilotAnalytics",
        [int]$DaysBack = 30
    )

    # Create output directory
    if (-not (Test-Path $OutputPath)) {
        New-Item -ItemType Directory -Path $OutputPath -Force | Out-Null
    }

    # Calculate date range
    $endDate = Get-Date
    $startDate = $endDate.AddDays(-$DaysBack)

    # Get agent list
    $agents = Get-AdminPowerAppEnvironmentAgents -EnvironmentName $EnvironmentId

    $analyticsData = @()

    foreach ($agent in $agents) {
        $agentMetrics = @{
            AgentId = $agent.Id
            AgentName = $agent.DisplayName
            Environment = $EnvironmentId
            SuccessRate = $agent.SuccessRate
            TotalSessions = $agent.TotalSessions
            AverageResponseTime = $agent.AvgResponseTime
            LastActiveDate = $agent.LastActiveDate
            Zone = $agent.CustomProperties.Zone
        }
        $analyticsData += [PSCustomObject]$agentMetrics
    }

    # Export to CSV
    $fileName = "CopilotAnalytics_$(Get-Date -Format 'yyyyMMdd_HHmmss').csv"
    $analyticsData | Export-Csv -Path (Join-Path $OutputPath $fileName) -NoTypeInformation

    Write-Host "Analytics exported to: $OutputPath\$fileName" -ForegroundColor Green
    return $analyticsData
}

# Usage
# Export-CopilotStudioAnalytics -EnvironmentId "env-id-here" -DaysBack 30

Query Audit Logs for Agent Activity

# Query unified audit log for Copilot Studio activities
function Get-AgentAuditLogs {
    param(
        [string]$AgentName = "*",
        [int]$DaysBack = 7,
        [string]$OutputPath = ".\AuditLogs"
    )

    # Connect to Exchange Online for audit log access
    Connect-ExchangeOnline

    $endDate = Get-Date
    $startDate = $endDate.AddDays(-$DaysBack)

    # Search for Copilot Studio activities
    $auditLogs = Search-UnifiedAuditLog `
        -StartDate $startDate `
        -EndDate $endDate `
        -RecordType CopilotStudio `
        -ResultSize 5000

    # Parse and filter results
    $parsedLogs = $auditLogs | ForEach-Object {
        $auditData = $_.AuditData | ConvertFrom-Json
        [PSCustomObject]@{
            Timestamp = $_.CreationDate
            User = $_.UserIds
            Operation = $_.Operations
            AgentName = $auditData.AgentName
            Environment = $auditData.EnvironmentId
            SessionId = $auditData.SessionId
            Success = $auditData.ResultStatus -eq "Success"
            Details = $auditData.ExtendedProperties
        }
    }

    # Filter by agent name if specified
    if ($AgentName -ne "*") {
        $parsedLogs = $parsedLogs | Where-Object { $_.AgentName -like "*$AgentName*" }
    }

    # Export results
    if (-not (Test-Path $OutputPath)) {
        New-Item -ItemType Directory -Path $OutputPath -Force | Out-Null
    }

    $fileName = "AgentAuditLogs_$(Get-Date -Format 'yyyyMMdd_HHmmss').csv"
    $parsedLogs | Export-Csv -Path (Join-Path $OutputPath $fileName) -NoTypeInformation

    Write-Host "Found $($parsedLogs.Count) audit log entries" -ForegroundColor Cyan
    return $parsedLogs
}

# Usage
# Get-AgentAuditLogs -AgentName "CustomerService" -DaysBack 30

Create Usage Report Scripts

# Generate comprehensive usage report for FSI governance
function New-FSIUsageReport {
    param(
        [string]$ReportPath = ".\Reports",
        [string]$ReportName = "FSI_Agent_Usage_Report",
        [int]$DaysBack = 30,
        [switch]$IncludeAuditData
    )

    Write-Host "Generating FSI Usage Report..." -ForegroundColor Cyan

    # Create report directory
    if (-not (Test-Path $ReportPath)) {
        New-Item -ItemType Directory -Path $ReportPath -Force | Out-Null
    }

    # Initialize report data
    $reportData = @{
        GeneratedDate = Get-Date
        ReportPeriod = "$DaysBack days"
        Environments = @()
        Summary = @{}
    }

    # Get all managed environments
    $environments = Get-AdminPowerAppEnvironment | Where-Object { $_.IsManagedEnvironment -eq $true }

    foreach ($env in $environments) {
        $envData = @{
            Name = $env.DisplayName
            Type = $env.EnvironmentType
            Zone = $env.CustomProperties.Zone
            Agents = @()
            Metrics = @{}
        }

        # Get agents in environment
        $agents = Get-AdminPowerAppEnvironmentAgents -EnvironmentName $env.EnvironmentName

        foreach ($agent in $agents) {
            $agentData = @{
                Name = $agent.DisplayName
                SuccessRate = $agent.SuccessRate
                Sessions = $agent.TotalSessions
                AvgResponseTime = $agent.AvgResponseTime
                Status = if ($agent.SuccessRate -ge 95) { "Healthy" }
                         elseif ($agent.SuccessRate -ge 90) { "Warning" }
                         else { "Critical" }
            }
            $envData.Agents += $agentData
        }

        # Calculate environment metrics
        $envData.Metrics = @{
            TotalAgents = $agents.Count
            HealthyAgents = ($envData.Agents | Where-Object { $_.Status -eq "Healthy" }).Count
            AverageSuccessRate = ($envData.Agents | Measure-Object -Property SuccessRate -Average).Average
        }

        $reportData.Environments += $envData
    }

    # Calculate summary metrics
    $allAgents = $reportData.Environments.Agents
    $reportData.Summary = @{
        TotalEnvironments = $reportData.Environments.Count
        TotalAgents = ($allAgents | Measure-Object).Count
        OverallSuccessRate = ($allAgents | Measure-Object -Property SuccessRate -Average).Average
        CriticalAgents = ($allAgents | Where-Object { $_.Status -eq "Critical" }).Count
        WarningAgents = ($allAgents | Where-Object { $_.Status -eq "Warning" }).Count
    }

    # Export report
    $timestamp = Get-Date -Format 'yyyyMMdd_HHmmss'
    $jsonPath = Join-Path $ReportPath "${ReportName}_${timestamp}.json"
    $reportData | ConvertTo-Json -Depth 10 | Out-File $jsonPath

    # Generate HTML summary
    $htmlPath = Join-Path $ReportPath "${ReportName}_${timestamp}.html"
    Generate-HTMLReport -Data $reportData -OutputPath $htmlPath

    Write-Host "Report generated: $jsonPath" -ForegroundColor Green
    return $reportData
}

# Usage
# New-FSIUsageReport -DaysBack 30 -IncludeAuditData

Financial Sector Considerations

Regulatory Context

Primary Regulations: FINRA 4511, SEC 17a-3/4, SOX 404, GLBA 501(b)

Regulation Monitoring Requirement
FINRA 4511 Monitor AI system activity for books and records
SEC 17a-3/4 Track usage of systems processing customer data
SOX 404 Monitor systems in financial reporting
GLBA 501(b) Track access to customer information

Examination Considerations

Regulators may request:

  • Agent usage statistics and trends
  • Performance metrics and SLA compliance
  • Alert history and response documentation
  • Evidence of regular monitoring reviews

Regulatory Mapping

Regulation Section Monitoring Requirement Implementation
FINRA 4511 (a)(1) Books and records for AI systems must be maintained and monitored Enable comprehensive audit logging; configure alerts for anomalies
FINRA 4511 (c) Preserve records in accessible format Export usage data to immutable storage; maintain 6-year retention
SEC 17a-3 (a)(17) Records of communications related to business Track all agent interactions involving customer communications
SEC 17a-4 (b)(4) Electronic storage with audit trail Configure audit logging with tamper-evident storage
GLBA 501(b) Safeguards Monitor access to customer NPI Create alerts for unusual data access patterns
SOX 404 Control testing Evidence of monitoring activities Generate weekly monitoring reports; document alert responses

Governance Tier Monitoring Requirements

Use governance tiers (Tier 1-3) for the control posture; do not map tiers directly to Dev/Test/Prod environment tiers.

Requirement Tier 1 (Personal Productivity) Tier 2 (Team Collaboration) Tier 3 (Enterprise Managed)
Dashboard Review Monthly Weekly Daily
Success Rate Threshold >80% >90% >95%
Alert Response Time Next business day 4 hours 1 hour
Performance SLA None 99% 99.9%
Audit Log Retention 90 days 1 year 7 years
Executive Reporting Quarterly summary Monthly summary Weekly summary
Incident Escalation Standard Expedited Immediate
Real-time Monitoring Not required Business hours 24/7

FSI Example: Usage Monitoring Configuration

# FSI Agent Usage Monitoring Configuration
monitoring:
  global:
    enabled: true
    retention_days: 2555  # 7 years for regulatory compliance
    export_format: "JSON"
    immutable_storage: true

  alerts:
    pre_built:
      - name: "High-use agents success rate under 90%"
        enabled: true
        severity: "High"
      - name: "Environment capacity warning"
        enabled: true
        severity: "Medium"

    custom:
      - name: "Tier 3 Success Rate Alert"
        condition: "agent.successRate < 95 AND agent.governanceTier == 'Tier3'"
        severity: "Critical"
        notification:
          - email: "ops-team@bank.com"
          - teams_channel: "Agent-Ops-Alerts"
          - sms: "+1-555-OPS-LINE"
        response_sla: "1 hour"

      - name: "Customer Data Access Anomaly"
        condition: "agent.dataAccessCount > (agent.avgDataAccess * 3)"
        severity: "Critical"
        notification:
          - email: "security@bank.com"
          - siem_integration: true
        escalation: "immediate"

      - name: "After-Hours Activity"
        condition: "agent.activityTime NOT IN businessHours AND agent.governanceTier == 'Tier3'"
        severity: "Medium"
        notification:
          - email: "compliance@bank.com"

  dashboards:
    zone_3_production:
      refresh_interval: "5 minutes"
      metrics:
        - success_rate
        - response_time
        - error_count
        - session_volume
        - data_access_events
      retention: "365 days"

  reporting:
    weekly:
      recipients:
        - "platform-admin@bank.com"
        - "compliance@bank.com"
      include:
        - success_rate_trends
        - alert_summary
        - capacity_metrics

    monthly:
      recipients:
        - "cto@bank.com"
        - "cro@bank.com"
        - "governance-committee@bank.com"
      include:
        - executive_summary
        - regulatory_compliance_status
        - trend_analysis
        - recommendations

  integrations:
    siem:
      enabled: true
      destination: "splunk.bank.com:9997"
      format: "CEF"
      events:
        - security_alerts
        - access_anomalies
        - configuration_changes

    grc:
      enabled: true
      platform: "ServiceNow GRC"
      sync_frequency: "daily"
      controls:
        - "3.2-usage-analytics"

Zone-Specific Configuration

Zone 1 (Personal Productivity):

  • Apply a baseline minimum of Usage Analytics and Activity Monitoring controls that impacts tenant-wide safety (where applicable), and document any exceptions for personal agents.
  • Avoid expanding scope beyond the user’s own data unless explicitly justified.
  • Rationale: reduces risk from personal use while keeping friction low; legal/compliance can tighten later.

Zone 2 (Team Collaboration):

  • Apply the control for shared agents and shared data sources; require an identified owner and an approval trail.
  • Validate configuration in a pilot environment before broader rollout; retain evidence (screenshots/exports/logs).
  • Rationale: shared agents increase blast radius; controls must be consistently applied and provable.

Zone 3 (Enterprise Managed):

  • Require the strictest configuration for Usage Analytics and Activity Monitoring controls and enforce it via policy where possible (not manual-only).
  • Treat changes as controlled (change ticket + documented testing); retain evidence (screenshots/exports/logs).
  • Rationale: enterprise agents handle the most sensitive content and are the highest audit/regulatory risk.

Verification & Testing

Step Action Expected Result
1 Navigate to PPAC → Monitor Monitor section displayed
2 Access Alerts (Preview) Alert rules visible
3 Review pre-built rules Microsoft rules listed
4 Access Copilot Studio dashboard Agent metrics displayed
5 Check success rate metric Current success rate shown
6 Verify alert notifications Test alert sends notification

Integration with Other Monitoring

System Integration Purpose
Microsoft 365 Audit Unified audit log Comprehensive activity history
Azure Monitor Advanced analytics Custom dashboards and alerts
SIEM systems Log forwarding Security monitoring
GRC tools Compliance reporting Regulatory evidence

Troubleshooting & Validation

Common Issues and Solutions

Issue Symptoms Root Cause Solution
Missing Usage Data Dashboard shows no metrics or incomplete data Managed Environment not enabled or data latency Verify Managed Environment is enabled; wait 24-48 hours for initial data population
Alerts Not Triggering Conditions met but no notifications received Notification recipients not configured or email filtering Verify alert rule is enabled; check notification settings; add sender to safe list
Incorrect Success Rates Reported rates don't match observed behavior Metric calculation includes all session types Review session definitions; filter by interaction type for accurate customer-facing metrics
Audit Log Gaps Missing entries for specific time periods Unified audit log latency or search scope Audit logs may take up to 24 hours to appear; expand search date range; verify RecordType filter
Performance Dashboard Timeout Dashboard fails to load or times out Large data volume or environment count Apply environment filters; reduce date range; schedule reports instead of real-time queries

Diagnostic Steps

  1. Verify Managed Environment Status 

    Get-AdminPowerAppEnvironment -EnvironmentName "env-id" | Select-Object DisplayName, IsManagedEnvironment

  2. Check Alert Rule Configuration

  3. Navigate to PPAC → Monitor → Alerts
  4. Verify rule is in "Enabled" state

  5. Test notification by manually triggering condition

  6. Validate Audit Log Access 

    Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-1) -EndDate (Get-Date) -RecordType CopilotStudio -ResultSize 10

  7. Review Data Freshness

  8. Check "Last Updated" timestamp on dashboard
  9. Data typically refreshes every 24 hours

Additional Resources

Topic URL Description
Power Platform Admin Center Monitoring https://learn.microsoft.com/en-us/power-platform/admin/admin-activity-logging Overview of PPAC monitoring capabilities
Set Up Alerts in PPAC https://learn.microsoft.com/en-us/power-platform/admin/monitoring/monitor-copilot-studio Configure pre-built and custom alert rules
Managed Environment Usage Insights https://learn.microsoft.com/en-us/power-platform/admin/managed-environment-usage-insights Usage analytics for managed environments
Copilot Studio Analytics https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-overview Agent-specific analytics and reporting
Unified Audit Log https://learn.microsoft.com/en-us/purview/audit-log-search Search and export audit log data
Power Platform CoE Starter Kit https://learn.microsoft.com/en-us/power-platform/guidance/coe/setup Additional monitoring and governance tools
Control Relationship
Control 3.1: Agent Inventory Track agent locations
Control 3.4: Incident Reporting Respond to alerts
Control 2.1: Managed Environments Usage insights
Control 1.7: Audit Logging Activity logging

Support & Questions

For implementation support or questions about this control, contact:

  • AI Governance Lead (governance direction)
  • Compliance Officer (regulatory requirements)
  • Power Platform Admin (technical setup)

Updated: Dec 2025
Version: v1.0 Beta (Dec 2025)
UI Verification Status: ❌ Needs verification