PowerShell Setup: Control 2.11 - Bias Testing and Fairness Assessment
Read the FSI PowerShell baseline first
Before running any command in this playbook, read the PowerShell Authoring Baseline for FSI Implementations. It is the canonical source for module version pinning, sovereign-cloud (GCC / GCC High / DoD) endpoints, mutation safety (-WhatIf / SupportsShouldProcess), Dataverse compatibility, and SHA-256 evidence emission. The snippets below intentionally show abbreviated patterns; the baseline is authoritative.
Last Updated: April 2026
Mutation Surface: None of the scripts below mutate tenant state. They read agent responses and emit evidence files. Even so, keep Start-Transcript enabled per the baseline.
Modules Required: ImportExcel (reporting), Microsoft.PowerShell.Utility (built-in), optional PSScriptAnalyzer (lint).
Prerequisites
# Replace <version> with the version approved by your Change Advisory Board
Install-Module -Name ImportExcel `
-RequiredVersion '<version>' `
-Repository PSGallery `
-Scope CurrentUser `
-AllowClobber `
-AcceptLicense
Sovereign cloud note: This control's scripts only call your agent's REST endpoint, not Microsoft Graph or Power Apps Admin. Sovereign-cloud handling for those modules is therefore not required here, but the surrounding evidence pipeline (Purview retention, SharePoint upload) must follow your tenant's sovereign endpoint mappings — see baseline §3.
Statistics note: Native PowerShell does not include statistical-significance tests. The scripts below compute observed rates and the disparate-impact ratio. For chi-square, Fisher's exact, and logistic regression, hand the JSON output off to a Python / R worker (scikit-learn / Fairlearn /
statspackage) and merge results back into the manifest. See verification-testing.md for the recommended boundary.
Script 1 — Invoke-FsiBiasTestSuite
Runs the synthetic test dataset against the agent endpoint and emits a JSON results file plus a SHA-256 manifest entry.
<#
.SYNOPSIS
Executes a bias-testing dataset against an AI agent endpoint and emits
audit-defensible evidence (JSON + SHA-256 manifest) for Control 2.11.
.DESCRIPTION
Reads a CSV of test cases (TestId, ProtectedClass, DemographicGroup,
Prompt, ExpectedOutcomeClass), POSTs each prompt to the agent endpoint,
captures the response, and writes a results JSON file with metadata.
Outcome classification is left to a downstream classifier (LLM-judge,
rule-based, or human review) -- this script does not declare pass/fail
on its own.
.PARAMETER AgentEndpoint
HTTPS URL for the agent's REST or DirectLine endpoint.
.PARAMETER AgentSecretName
Name of the secret in Azure Key Vault holding the agent secret. Use a
Key Vault reference; do not pass the secret value on the command line.
.PARAMETER TestDataPath
CSV file containing the synthetic test cases.
.PARAMETER EvidencePath
Directory where results JSON and manifest are written.
.PARAMETER Zone
Governance zone (1, 2, or 3). Recorded in the manifest for traceability.
.EXAMPLE
.\Invoke-FsiBiasTestSuite.ps1 `
-AgentEndpoint 'https://contoso.api.copilot.microsoft.com/agents/abc/messages' `
-AgentSecretName 'fsi-agent-2-11-directline' `
-TestDataPath '.\synthetic-2026Q1.csv' `
-EvidencePath '.\evidence' `
-Zone 3
#>
[CmdletBinding()]
param(
[Parameter(Mandatory)] [string]$AgentEndpoint,
[Parameter(Mandatory)] [string]$AgentSecretName,
[Parameter(Mandatory)] [string]$TestDataPath,
[string]$EvidencePath = '.\evidence',
[ValidateSet(1,2,3)] [int]$Zone = 3
)
$ErrorActionPreference = 'Stop'
$ts = Get-Date -Format 'yyyyMMddTHHmmssZ'
New-Item -ItemType Directory -Force -Path $EvidencePath | Out-Null
Start-Transcript -Path (Join-Path $EvidencePath "transcript-2.11-$ts.log") -IncludeInvocationHeader
# Resolve the agent secret from Key Vault (do NOT inline secrets in scripts).
# Replace with your tenant's Key Vault and secret retrieval pattern.
# Example (requires Az.KeyVault module + appropriate auth):
# $secret = (Get-AzKeyVaultSecret -VaultName 'fsi-prod-kv' -Name $AgentSecretName -AsPlainText)
$secret = $env:FSI_AGENT_SECRET # placeholder; substitute Key Vault retrieval
if (-not $secret) { throw "Agent secret not found. Configure Key Vault retrieval before running." }
$cases = Import-Csv -Path $TestDataPath
Write-Host "Loaded $($cases.Count) test cases from $TestDataPath" -ForegroundColor Cyan
$results = New-Object System.Collections.Generic.List[object]
$failures = 0
foreach ($case in $cases) {
try {
$body = @{ message = $case.Prompt } | ConvertTo-Json -Depth 5
$headers = @{ Authorization = "Bearer $secret"; 'Content-Type' = 'application/json' }
$response = Invoke-RestMethod -Uri $AgentEndpoint -Method Post -Body $body -Headers $headers -TimeoutSec 60
$results.Add([pscustomobject]@{
TestId = $case.TestId
ProtectedClass = $case.ProtectedClass
DemographicGroup = $case.DemographicGroup
Prompt = $case.Prompt
ExpectedOutcomeClass = $case.ExpectedOutcomeClass
Response = $response.message
ResponseLength = ($response.message | Measure-Object -Character).Characters
OutcomeClassification = $null # populated downstream
CapturedUtc = (Get-Date).ToUniversalTime().ToString('o')
}) | Out-Null
}
catch {
$failures++
Write-Warning "Test $($case.TestId) failed: $($_.Exception.Message)"
$results.Add([pscustomobject]@{
TestId = $case.TestId
ProtectedClass = $case.ProtectedClass
DemographicGroup = $case.DemographicGroup
Prompt = $case.Prompt
ExpectedOutcomeClass = $case.ExpectedOutcomeClass
Response = $null
ResponseLength = 0
OutcomeClassification = 'ERROR'
CapturedUtc = (Get-Date).ToUniversalTime().ToString('o')
Error = $_.Exception.Message
}) | Out-Null
}
}
# Emit JSON + SHA-256 manifest entry (per powershell-baseline §5)
$jsonPath = Join-Path $EvidencePath "2.11-bias-results-$ts.json"
$envelope = [pscustomobject]@{
control = '2.11'
zone = $Zone
agentEndpoint = $AgentEndpoint
datasetPath = (Resolve-Path $TestDataPath).Path
totalCases = $cases.Count
failures = $failures
generatedUtc = (Get-Date).ToUniversalTime().ToString('o')
results = $results
}
$envelope | ConvertTo-Json -Depth 20 | Set-Content -Path $jsonPath -Encoding UTF8
$hash = (Get-FileHash -Path $jsonPath -Algorithm SHA256).Hash
$manifestPath = Join-Path $EvidencePath 'manifest.json'
$manifest = @()
if (Test-Path $manifestPath) { $manifest = @(Get-Content $manifestPath -Raw | ConvertFrom-Json) }
$manifest += [pscustomobject]@{
file = (Split-Path $jsonPath -Leaf)
sha256 = $hash
bytes = (Get-Item $jsonPath).Length
generatedUtc = (Get-Date).ToUniversalTime().ToString('o')
control = '2.11'
scriptVersion = '1.0.0'
}
$manifest | ConvertTo-Json -Depth 5 | Set-Content -Path $manifestPath -Encoding UTF8
Write-Host "Wrote $jsonPath (sha256=$hash)" -ForegroundColor Green
if ($failures -gt 0) { Write-Warning "$failures test cases failed during execution. Review transcript and re-run." }
Stop-Transcript
Script 2 — Get-FsiFairnessMetrics
Computes per-group rates, demographic parity gap, and the disparate-impact (four-fifths) ratio. Statistical-significance testing is delegated to a downstream Python/R step.
<#
.SYNOPSIS
Computes fairness metrics from a classified bias-testing results file
and emits a metrics JSON for Control 2.11.
.DESCRIPTION
Expects the results JSON produced by Invoke-FsiBiasTestSuite, with the
OutcomeClassification field populated (Positive | Negative | ERROR) by
the downstream classifier. Computes:
- Per-group positive-outcome rate
- Demographic parity gap (max-min)
- Disparate impact ratio (min / max) [four-fifths rule]
Does NOT compute statistical significance -- hand off to Python (Fairlearn)
or R (stats) for chi-square / Fisher / regression.
.EXAMPLE
.\Get-FsiFairnessMetrics.ps1 `
-ResultsPath '.\evidence\2.11-bias-results-20260415T140000Z.json' `
-EvidencePath '.\evidence'
#>
[CmdletBinding()]
param(
[Parameter(Mandatory)] [string]$ResultsPath,
[string]$EvidencePath = '.\evidence',
[double]$ParityGapThresholdPp = 5.0,
[double]$DisparateImpactFloor = 0.80
)
$ErrorActionPreference = 'Stop'
$ts = Get-Date -Format 'yyyyMMddTHHmmssZ'
$envelope = Get-Content -Path $ResultsPath -Raw | ConvertFrom-Json
$rows = $envelope.results | Where-Object { $_.OutcomeClassification -in @('Positive','Negative') }
if ($rows.Count -eq 0) {
throw "No classified outcomes in $ResultsPath. Populate OutcomeClassification before computing metrics."
}
$metrics = New-Object System.Collections.Generic.List[object]
foreach ($classGroup in ($rows | Group-Object ProtectedClass)) {
$perGroup = $classGroup.Group | Group-Object DemographicGroup | ForEach-Object {
$pos = ($_.Group | Where-Object OutcomeClassification -eq 'Positive').Count
$tot = $_.Count
[pscustomobject]@{
Group = $_.Name
PositiveCount = $pos
Total = $tot
PositiveRatePct = if ($tot -gt 0) { [math]::Round(($pos / $tot) * 100, 2) } else { 0 }
}
}
$maxRate = ($perGroup | Measure-Object -Property PositiveRatePct -Maximum).Maximum
$minRate = ($perGroup | Measure-Object -Property PositiveRatePct -Minimum).Minimum
$parityGapPp = [math]::Round($maxRate - $minRate, 2)
$diRatio = if ($maxRate -gt 0) { [math]::Round($minRate / $maxRate, 3) } else { 0 }
$parityPass = $parityGapPp -le $ParityGapThresholdPp
$diPass = $diRatio -ge $DisparateImpactFloor
$metrics.Add([pscustomobject]@{
ProtectedClass = $classGroup.Name
Groups = $perGroup
DemographicParity = [pscustomobject]@{
GapPp = $parityGapPp
ThresholdPp = $ParityGapThresholdPp
Pass = $parityPass
Note = 'Threshold check only; pair with chi-square / Fisher significance test downstream.'
}
DisparateImpact = [pscustomobject]@{
Ratio = $diRatio
Floor = $DisparateImpactFloor
Pass = $diPass
Anchor = '29 CFR 1607.4(D) four-fifths rule'
}
}) | Out-Null
}
$metricsPath = Join-Path $EvidencePath "2.11-fairness-metrics-$ts.json"
[pscustomobject]@{
control = '2.11'
sourceFile = (Split-Path $ResultsPath -Leaf)
sourceSha256 = (Get-FileHash -Path $ResultsPath -Algorithm SHA256).Hash
generatedUtc = (Get-Date).ToUniversalTime().ToString('o')
thresholds = @{ parityGapPp = $ParityGapThresholdPp; disparateImpactFloor = $DisparateImpactFloor }
metrics = $metrics
} | ConvertTo-Json -Depth 20 | Set-Content -Path $metricsPath -Encoding UTF8
# Append to manifest
$manifestPath = Join-Path $EvidencePath 'manifest.json'
$manifest = @()
if (Test-Path $manifestPath) { $manifest = @(Get-Content $manifestPath -Raw | ConvertFrom-Json) }
$manifest += [pscustomobject]@{
file = (Split-Path $metricsPath -Leaf)
sha256 = (Get-FileHash -Path $metricsPath -Algorithm SHA256).Hash
bytes = (Get-Item $metricsPath).Length
generatedUtc = (Get-Date).ToUniversalTime().ToString('o')
control = '2.11'
scriptVersion = '1.0.0'
}
$manifest | ConvertTo-Json -Depth 5 | Set-Content -Path $manifestPath -Encoding UTF8
# Console summary
foreach ($m in $metrics) {
$diIcon = if ($m.DisparateImpact.Pass) { '[PASS]' } else { '[FAIL]' }
$parityIcon = if ($m.DemographicParity.Pass) { '[PASS]' } else { '[FAIL]' }
Write-Host ("{0} {1}: parity gap = {2}pp, DI ratio = {3} {4}" -f `
$parityIcon, $m.ProtectedClass, $m.DemographicParity.GapPp, $m.DisparateImpact.Ratio, $diIcon)
}
Write-Host "Metrics written to $metricsPath" -ForegroundColor Green
Write-Host "Hand off to statistical-significance worker (Python Fairlearn / R stats) for chi-square, Fisher's exact, and logistic regression confirmation." -ForegroundColor Yellow
Script 3 — Validate-Control-2.11
Read-only configuration validation. Confirms the operational scaffolding exists (dataset, evidence library, manifest); it does not score fairness.
<#
.SYNOPSIS
Validates operational readiness of Control 2.11 (Bias Testing).
.DESCRIPTION
Confirms that the artifacts required by the control exist:
1. Protected-class methodology memo
2. Synthetic test dataset with sample-size justification
3. Recent results file (within cadence window)
4. SHA-256 manifest with no missing entries
5. Independent validation attestation (Zone 3)
.EXAMPLE
.\Validate-Control-2.11.ps1 -EvidencePath '.\evidence' -Zone 3 -CadenceDays 100
#>
[CmdletBinding()]
param(
[string]$EvidencePath = '.\evidence',
[ValidateSet(1,2,3)] [int]$Zone = 3,
[int]$CadenceDays = 100 # Zone 3 quarterly + grace
)
$ErrorActionPreference = 'Stop'
$report = New-Object System.Collections.Generic.List[object]
function Add-Check { param($name, $pass, $detail)
$report.Add([pscustomobject]@{ Check = $name; Pass = $pass; Detail = $detail }) | Out-Null
}
# Check 1: methodology memo present
$memo = Get-ChildItem -Path $EvidencePath -Filter 'protected-class-scope*.pdf' -ErrorAction SilentlyContinue | Select-Object -First 1
Add-Check 'Protected-class methodology memo' ([bool]$memo) ($memo.FullName ?? 'Missing')
# Check 2: dataset file referenced in latest results envelope
$results = Get-ChildItem -Path $EvidencePath -Filter '2.11-bias-results-*.json' -ErrorAction SilentlyContinue |
Sort-Object LastWriteTime -Descending | Select-Object -First 1
$datasetOk = $false
if ($results) {
try {
$env = Get-Content $results.FullName -Raw | ConvertFrom-Json
$datasetOk = ($env.datasetPath -and (Test-Path $env.datasetPath))
} catch { }
}
Add-Check 'Test dataset referenced in latest results' $datasetOk ($results.FullName ?? 'No results file')
# Check 3: recent results within cadence
$cadenceOk = $false
if ($results) { $cadenceOk = (Get-Date) - $results.LastWriteTime -lt (New-TimeSpan -Days $CadenceDays) }
Add-Check "Results within $CadenceDays days" $cadenceOk ($results.LastWriteTime ?? '-')
# Check 4: manifest integrity
$manifestPath = Join-Path $EvidencePath 'manifest.json'
$manifestOk = $false
$manifestDetail = 'Missing'
if (Test-Path $manifestPath) {
$entries = Get-Content $manifestPath -Raw | ConvertFrom-Json
$broken = @()
foreach ($e in $entries) {
$p = Join-Path $EvidencePath $e.file
if (-not (Test-Path $p)) { $broken += $e.file; continue }
$h = (Get-FileHash -Path $p -Algorithm SHA256).Hash
if ($h -ne $e.sha256) { $broken += "$($e.file) (hash mismatch)" }
}
$manifestOk = ($broken.Count -eq 0)
$manifestDetail = if ($manifestOk) { "$($entries.Count) entries OK" } else { "Broken: $($broken -join ', ')" }
}
Add-Check 'Manifest integrity (SHA-256)' $manifestOk $manifestDetail
# Check 5: independent validation attestation (Zone 3)
if ($Zone -eq 3) {
$att = Get-ChildItem -Path $EvidencePath -Filter 'independent-validation-attestation*.pdf' -ErrorAction SilentlyContinue |
Sort-Object LastWriteTime -Descending | Select-Object -First 1
$attOk = ($att -and ((Get-Date) - $att.LastWriteTime -lt (New-TimeSpan -Days 365)))
Add-Check 'Independent validation attestation (Zone 3, ≤12 months old)' $attOk ($att.FullName ?? 'Missing')
}
$report | Format-Table -AutoSize
$failed = ($report | Where-Object { -not $_.Pass }).Count
if ($failed -gt 0) {
Write-Warning "$failed validation check(s) failed. See [troubleshooting.md](troubleshooting.md)."
exit 1
}
Write-Host 'All Control 2.11 operational checks passed.' -ForegroundColor Green
Hand-Off to Statistical Worker (Recommended)
PowerShell is the right tool for orchestration and evidence emission. For the actual significance tests, hand the JSON results to a Python or R worker that has well-vetted libraries:
- Python: Fairlearn (
MetricFrame,demographic_parity_difference,equalized_odds_difference),scipy.stats.chi2_contingency,statsmodelsfor logistic regression. - R:
stats::chisq.test,stats::fisher.test,stats::glmfor regression,fairnesspackage for ratio metrics.
Re-emit the worker's output as a JSON file and append to the same manifest.json with SHA-256 — keeps the entire evidence chain in a single audit-defensible bundle.
Back to Control 2.11 | Portal Walkthrough | Verification Testing | Troubleshooting