Testing Scenarios — Audit Logging Compliance Automation (ALCA)¶

15 test scenarios for validating the ALCA solution, plus a troubleshooting guide for 10 common issues.

Detection Scenarios¶

Scenario 1: All Environments Compliant¶

Setup: All Power Platform environments have Purview unified audit enabled. All Dataverse environments have org-level audit enabled.

Expected results: - All environments show Compliant status - Summary: N/N compliant, 0 non-compliant, 0 errors - Dataverse records all have fsi_compliancestatus = 100000000 (Compliant)

Verification: 1. Run Test-AuditLoggingCompliance.ps1 with required parameters 2. Verify console output shows all environments as Compliant 3. Check Dataverse table for matching records

Scenario 2: Mixed Compliance (Compliant + Non-Compliant)¶

Setup: At least one environment has Purview audit enabled and one does not. At least one Dataverse environment has org-level audit disabled.

Expected results: - Mix of Compliant and Non-Compliant statuses - Summary reflects accurate counts - Non-compliant records have fsi_compliancestatus = 100000001

Verification: 1. Run detection runbook 2. Verify per-environment status matches actual configuration 3. Confirm Dataverse records have correct status values

Scenario 3: Purview Unified Audit Disabled¶

Setup: Disable Purview unified audit log ingestion for a test environment via Set-AdminAuditLogConfig.

Expected results: - Affected environment shows Non-Compliant - fsi_auditenabled = false in Dataverse record - Other environments unaffected

Verification: 1. Disable audit: Set-AdminAuditLogConfig -UnifiedAuditLogIngestionEnabled $false 2. Run detection runbook 3. Verify environment shows Non-Compliant with PurviewAuditEnabled = False

Scenario 4: Audit Event Validation¶

Setup: Environment with audit enabled but no recent audit events in last 7 days.

Expected results: - Environment still shows Compliant if audit configuration is correct (event presence is informational, not a compliance gate) - LastAuditEvent field shows "None in last 7 days"

Verification: 1. Run detection on an environment with audit enabled 2. Verify LastAuditEvent column in CSV output 3. Confirm compliance status based on configuration, not event presence

Remediation Scenarios¶

Scenario 5: WhatIf Mode¶

Setup: Non-compliant environments exist. Run with -WhatIf switch.

Expected results: - Console shows [WHATIF] Would enable... for each action - No actual changes made to any environment - No Dataverse compliance records updated - Summary shows all actions as simulated

Verification: 1. Run Enable-AuditLogging.ps1 -WhatIf 2. Verify [WHATIF] prefix on all action lines 3. Confirm environment audit settings unchanged after run 4. Verify Dataverse records unchanged

Scenario 6: Dataverse Org-Level Audit Enablement¶

Setup: A Dataverse environment with isauditenabled = false.

Expected results: - Runbook enables org-level audit via PATCH to /api/data/v9.2/organizations - Entity-level audit enabled on 6 entities - Validation confirms changes after 5-second wait - Dataverse record updated to Compliant

Verification: 1. Confirm isauditenabled = false before remediation 2. Run remediation runbook (without WhatIf) 3. Verify isauditenabled = true via Dataverse Web API 4. Check entity audit settings for bot, botcomponent, connectionreference, environmentvariablevalue, workflow, systemuser

Scenario 7: Tenant-Wide Purview Unified Audit Enablement¶

Setup: Purview unified audit disabled at tenant level. Run with -EnableTenantUnifiedAudit.

Expected results: - Console shows tenant-wide change warning - Set-AdminAuditLogConfig -UnifiedAuditLogIngestionEnabled $true is called - All environments benefit from tenant-level change

Verification: 1. Verify current state: (Get-AdminAuditLogConfig).UnifiedAuditLogIngestionEnabled 2. Run remediation with -EnableTenantUnifiedAudit 3. Verify post-state: (Get-AdminAuditLogConfig).UnifiedAuditLogIngestionEnabled -eq $true

Scenario 8: Validation Failure After Remediation¶

Setup: Environment where audit enablement is applied but validation re-read still shows disabled (e.g., propagation delay exceeds 5 seconds).

Expected results: - Remediation actions are attempted - Validation re-read fails - Status set to Remediation Pending (not Compliant) - Environment flagged for follow-up in summary

Verification: 1. This scenario is typically transient — run detection again after 60 seconds 2. If persistent, check Dataverse for fsi_compliancestatus = 100000002

Infrastructure Scenarios¶

Scenario 9: Retry and Throttling (429)¶

Setup: High-volume tenant with many environments that may trigger API throttling.

Expected results: - Invoke-WithRetry handles 429 responses with exponential backoff - Verbose output shows retry attempts and delays - Scanning completes successfully (may take longer)

Verification: 1. Run detection with -Verbose to see retry logs 2. Verify all environments are processed despite throttling 3. Check that no environments are skipped

Scenario 10: Multi-Recipient Email Notification¶

Setup: Configure -NotificationToAddresses with multiple comma-separated recipients.

Expected results: - HTML email sent to all recipients - CSV attachment included - Email subject includes compliance counts - saveToSentItems = false (shared mailbox)

Verification: 1. Run detection with -SendEmail -NotificationToAddresses "user1@example.com,user2@example.com" 2. Verify all recipients received the email 3. Verify CSV attachment is present and contains all environment data

Scenario 11: Dataverse Upsert — Create New Record¶

Setup: First-time scan of an environment not yet in the compliance table.

Expected results: - Write-DataverseComplianceRecord queries for existing record (GET returns empty) - Creates new record via POST - Record has correct environment ID, name, audit status, and compliance status

Verification: 1. Verify environment has no existing record in fsi_auditenvironmentcompliances 2. Run detection 3. Verify new record created with correct field values

Scenario 12: Dataverse Upsert — Update Existing Record¶

Setup: Re-scan an environment that already has a compliance record.

Expected results: - Write-DataverseComplianceRecord finds existing record (GET returns match) - Updates record via PATCH (not POST) - fsi_lastchecked timestamp updated - Status reflects current compliance state

Verification: 1. Note the existing record's fsi_lastchecked value 2. Run detection 3. Verify fsi_lastchecked updated, record ID unchanged

Error Handling Scenarios¶

Scenario 13: Per-Environment Error (Continue Scanning)¶

Setup: An environment that causes an error (e.g., Dataverse API returns 403 for one environment).

Expected results: - Error caught per-environment, does not stop scanning - Error environment gets ComplianceStatus = Error with error message - Remaining environments processed normally - Summary includes error count

Verification: 1. Verify environment with restricted access exists 2. Run detection 3. Confirm error environment shows Error status with descriptive message 4. Confirm all other environments still processed

Scenario 14: Fatal Authentication Failure¶

Setup: Revoke Managed Identity permissions (e.g., remove Power Platform Administrator role).

Expected results: - Authentication fails immediately at Step 1 - Runbook exits with error (does not proceed to scanning) - Error message indicates authentication failure - No Dataverse records written

Verification: 1. Temporarily remove MI role assignment 2. Run detection 3. Verify [FATAL] message in output 4. Restore MI role assignment after test

Scenario 15: Scheduled Execution¶

Setup: Link the detection runbook to a schedule (see Scheduling Guide).

Expected results: - Runbook executes automatically at scheduled time - Job status shows Completed - Dataverse records updated - Email sent (if configured)

Verification: 1. Configure schedule per scheduling guide 2. Wait for scheduled execution 3. Check Automation Account → Jobs for completed job 4. Verify Dataverse compliance records updated 5. Verify email received (if SendEmail configured)

Troubleshooting Guide¶

Issue 1: Power Platform Authentication Failure¶

Symptoms: - [FATAL] Power Platform authentication failed - Error: Add-PowerAppsAccount : The token could not be acquired

Possible causes: - Managed Identity not assigned Power Platform Administrator role - MI not enabled on the Automation Account - Incorrect tenant domain

Resolution: 1. Verify MI is enabled: Automation Account → Identity → Status = On 2. Verify role: Entra ID → Roles and administrators → Power Platform Administrator → Check MI is listed 3. Test token acquisition in Test Pane: Get-ManagedIdentityToken -Resource "https://api.bap.microsoft.com/"

Issue 2: Exchange Online Authentication Failure¶

Symptoms: - [FATAL] Exchange Online authentication failed - Error: Connect-ExchangeOnline : Failed to acquire token

Possible causes: - Managed Identity not assigned Exchange Online Admin role - ExchangeOnlineManagement module not imported or wrong version - Incorrect -Organization parameter

Resolution: 1. Verify role: Entra ID → Roles and administrators → Exchange Online Admin → Check MI is listed 2. Verify module: Automation Account → Modules → ExchangeOnlineManagement Status = Available on Runtime 7.2 3. Verify TenantDomain parameter matches yourdomain.onmicrosoft.com format

Issue 3: Dataverse 401 Access Denied¶

Symptoms: - Dataverse API calls fail with 401 Unauthorized - Error in Invoke-DataverseRequest: The user is not a member of the organization

Possible causes: - MI not added as Application User in the target Dataverse environment - Application User does not have System Administrator role - Token scoped to wrong Dataverse URL

Resolution: 1. Navigate to Power Platform Admin Center → target environment → Settings → Application users 2. Verify MI is listed as an Application User 3. Verify the assigned security role is System Administrator 4. Ensure DataverseEnvironmentUrl parameter matches the governance environment URL

Issue 4: Email Not Sent¶

Symptoms: - [ERROR] Failed to send email - Error: MailboxNotEnabledForRESTAPI or AccessDenied

Possible causes: - MI does not have Mail.Send Graph application permission - Admin consent not granted for Mail.Send - Shared mailbox does not have SendAs permission for MI - Incorrect NotificationFromAddress

Resolution: 1. Verify Graph permission: Use Get-MgServicePrincipalAppRoleAssignment to check Mail.Send 2. Grant admin consent: Enterprise Applications → MI → Permissions → Grant admin consent 3. Verify SendAs: Get-RecipientPermission -Identity "shared-mailbox@domain.com" | Where Trustee -match "MI-name" 4. Ensure NotificationFromAddress matches the shared mailbox address exactly

Issue 5: Dataverse Table Not Updated¶

Symptoms: - Detection runs without errors but no records appear in fsi_auditenvironmentcompliances - Or records appear but fields are not updated

Possible causes: - Table does not exist (schema not deployed) - Alternate key not configured on fsi_environmentid - Dataverse token scoped to wrong environment

Resolution: 1. Verify table exists: Navigate to Dataverse → Tables → Search for Audit Environment Compliance 2. Run the schema creation script: python create_audit_compliance_schema.py 3. Verify DataverseEnvironmentUrl points to the governance environment (not a target environment)

Issue 6: 429 Throttling (Too Many Requests)¶

Symptoms: - Verbose logs show multiple retry attempts - Runbook takes significantly longer than expected - Error: 429 Too Many Requests in verbose output

Possible causes: - Large number of environments (50+) causing API rate limits - Concurrent automation scripts hitting same APIs - Power Platform API throttling limits reached

Resolution: 1. This is handled automatically by Invoke-WithRetry with exponential backoff 2. If persistent, increase MaxRetries parameter in function calls 3. Consider staggering scheduled runs to avoid peak API usage times 4. For 100+ environments, consider batching (future enhancement)

Issue 7: Validation Failure After Remediation¶

Symptoms: - Remediation runs but validation step shows audit still disabled - Status set to Remediation Pending instead of Compliant

Possible causes: - Propagation delay exceeds the 5-second wait - Cached API response returning stale data - Insufficient permissions to modify entity definitions

Resolution: 1. Wait 60 seconds and re-run detection to check actual state 2. If still failing, increase the propagation wait (modify Start-Sleep in runbook) 3. Verify MI has System Administrator role in the target environment 4. Check entity-level audit settings manually via Dataverse Web API

Issue 8: Search-UnifiedAuditLog Cmdlet Not Found¶

Symptoms: - Error: The term 'Search-UnifiedAuditLog' is not recognized - Or: Search-UnifiedAuditLog requires Exchange Online connection

Possible causes: - ExchangeOnlineManagement module not imported - Exchange Online not connected before searching - Module version incompatible with PowerShell 7.2

Resolution: 1. Verify module: Automation Account → Modules → ExchangeOnlineManagement Runtime 7.2 → Status: Available 2. Ensure Connect-ExchangeOnline succeeds before Search-UnifiedAuditLog 3. Update module to latest version if Status shows issues

Issue 9: CSV Export Path Failure¶

Symptoms: - Error writing CSV to $env:TEMP - Export-Csv : Access to the path is denied

Possible causes: - $env:TEMP not defined in Azure Automation context - Disk space issue in sandbox

Resolution: 1. Azure Automation runbooks have access to $env:TEMP by default 2. If failing, use a hardcoded fallback: $csvPath = "C:\Temp\AuditCompliance.csv" 3. Verify the CSV is generated correctly in the Test Pane before scheduling

Issue 10: WhatIf Not Working (Changes Still Applied)¶

Symptoms: - Running with -WhatIf still makes actual changes - Environments are modified despite WhatIf flag

Possible causes: - Incorrect WhatIf parameter binding - SupportsShouldProcess not properly configured in CmdletBinding

Resolution: 1. Verify the script is using $PSCmdlet.ShouldProcess() gates around all modification calls 2. Ensure -WhatIf is passed at the command line (not just referenced as a variable) 3. Test with: .\Enable-AuditLogging.ps1 -DataverseEnvironmentUrl $url -TenantDomain $domain -WhatIf 4. Verify [WHATIF] prefix appears in output for each would-be action

Updated: February 2026 | Version: v1.0.3