Skip to content

Control 4.5: Copilot Usage Analytics and Adoption Reporting — Verification & Testing

Test cases and evidence collection procedures for Copilot usage analytics and adoption reporting.

Test Cases

Test 1: Usage Dashboard Data Accuracy

  • Objective: Verify that Copilot usage dashboard data accurately reflects actual usage
  • Steps:
  • Have 5 test users perform known Copilot interactions across different applications.
  • Wait 48 hours for reporting data to update.
  • Check the M365 Admin Center Copilot usage report for the test users.
  • Verify the reported activity matches the known test interactions.
  • Expected Result: Dashboard accurately reflects the test users' Copilot activity within 48 hours.
  • Evidence: Comparison of known test activity with dashboard-reported data.

Test 2: Report Privacy Controls

  • Objective: Confirm that usage report privacy settings are correctly configured
  • Steps:
  • Log in as a Reports Reader role user and access the Copilot usage report.
  • Verify that user-identifiable information is shown or hidden per the configured privacy settings.
  • Log in as a Purview Compliance Admin and verify they can access identifiable data.
  • Confirm the audit log records who accessed user-identifiable usage reports.
  • Expected Result: Privacy settings correctly control visibility of identifiable data per role.
  • Evidence: Screenshots showing different data visibility for different roles.

Test 3: Adoption KPI Tracking

  • Objective: Validate that adoption KPIs are accurately calculated and tracked over time
  • Steps:
  • Run the adoption report scripts for 7-day, 30-day, and 90-day periods.
  • Verify the active user count is consistent across data sources.
  • Confirm the adoption rate calculation is correct (active / licensed * 100).
  • Compare with the M365 Admin Center dashboard for consistency.
  • Expected Result: KPI calculations are accurate and consistent across reporting methods.
  • Evidence: KPI comparison report showing calculated values from multiple sources.

Test 4: Department-Level Reporting Accuracy

  • Objective: Confirm department-level adoption reports correctly aggregate user data
  • Steps:
  • Run the department-level adoption report script.
  • Manually verify 3 departments by cross-referencing user lists and activity.
  • Confirm department assignments match Entra ID department attributes.
  • Verify totals sum correctly across all departments.
  • Expected Result: Department adoption data is accurately aggregated from individual user data.
  • Evidence: Department report with manual verification notes.

Evidence Collection

Evidence Item Source Format Retention
Usage dashboard screenshot M365 Admin Center Screenshot Monthly archive
Adoption report PowerShell/Graph CSV Monthly archive
Privacy settings config Admin Center Screenshot With control documentation
KPI tracking log Custom report Spreadsheet 1 year

Compliance Mapping

Regulation Requirement How This Control Helps
FFIEC Management Booklet IT investment monitoring Supports compliance with technology utilization tracking
12 CFR part 30, appendix D (OCC Heightened Standards) Technology governance Helps meet expectations for AI technology adoption oversight
Sarbanes-Oxley §404 Internal control over IT assets Supports IT asset management and utilization controls

Next Steps