Skip to content

Control 4.5: Copilot Usage Analytics and Adoption Reporting — Verification & Testing

Conservative tests that validate whether usage-detail reporting is accessible and produces records in a live tenant.

Non-Claims (Scope Guardrails)

Passing these tests does not prove:

  • dashboard interpretation quality
  • adoption quality or business impact
  • regulatory sufficiency on its own
  • complete user coverage across licensed and unlicensed populations

Test Cases

Test 1: Permission and Role Gate

  • Objective: Confirm the caller is authorized to query Copilot usage-detail APIs.
  • Steps:
  • Connect with Connect-MgGraph -Scopes "Reports.Read.All".
  • Run Get-MgContext | Select-Object Scopes.
  • Validate the identity is assigned a supported delegated admin role (if using delegated auth), or app-only consent exists for Reports.Read.All.
  • Expected Result: Reports.Read.All is present and authorization requirements are met.
  • Fail Closed: Missing scope, missing role, or consent errors.

Test 2: Beta Endpoint/Cmdlet Reachability

  • Objective: Confirm the documented beta API path is reachable from the tenant.
  • Steps:
  • Run Get-MgBetaReportMicrosoft365CopilotUsageUserDetail -Period D7 -Format application/json -OutFile .\copilot-usage-d7.json.
  • Parse the file with Get-Content -Raw .\copilot-usage-d7.json | ConvertFrom-Json.
  • Expected Result: Command succeeds and returns structured JSON payload.
  • Fail Closed: HTTP/auth errors, malformed output, or missing file.

Test 3: Record Presence Validation

  • Objective: Validate that the export contains usage-detail records.
  • Steps:
  • Count records in .value.
  • Capture reportRefreshDate, userPrincipalName, and at least one activity-date field from a sample row.
  • Expected Result: Record count is >= 1.
  • Fail Closed: Zero records returned.

Test 4: Response and Export Behavior Validation

  • Objective: Confirm expected Microsoft Graph response/export behavior for the selected mode.
  • Steps:
  • JSON path: verify 200 OK semantics and JSON payload in output file.
  • Optional CSV path: call legacy beta endpoint with $format=text/csv and capture redirect response details.
  • Expected Result: JSON mode returns data; CSV mode shows documented redirect/download behavior.
  • Fail Closed: Response mode differs from docs without documented explanation.

Test 5: Throttling and Retry Controls

  • Objective: Validate handling of Graph throttling events.
  • Steps:
  • On 429 Too Many Requests, inspect Retry-After.
  • Re-run after the required wait time.
  • Log retry count and final outcome.
  • Expected Result: Process follows Retry-After guidance and logs retries.
  • Fail Closed: Repeated throttling with no successful retrieval in the approved execution window.

Test 6: Caveat and Coverage Documentation

  • Objective: Confirm that known API limitations are documented with evidence.
  • Steps:
  • Record that the control uses a beta API surface.
  • Record that availability is global-service-only per docs.
  • Record that unlicensed Copilot Chat usage is excluded from this API.
  • Expected Result: Verification packet includes explicit caveat statements.
  • Fail Closed: Caveats omitted from evidence package.

Evidence Collection

Evidence Item Source Format Retention
Beta cmdlet execution transcript PowerShell session Text log Quarterly archive
Usage-detail export artifact Microsoft Graph beta API JSON/CSV Quarterly archive
Authorization proof Entra app consent + role assignment Screenshot/export With control record
Throttling/retry log Execution transcript Text log With control record
Caveat statement Control runbook record Markdown/PDF With committee packet

Operational Decision Rule

  • Pass: authorization valid, export succeeds, and at least one record is present.
  • Fail: missing permissions, unreachable endpoint, unresolved throttling, or empty result set.
  • Manual follow-up required: pass/fail result must be reviewed with reporting cadence evidence before final control attestation.

Next Steps