Eval Gate¶

CI evaluation gate for Copilot Studio agent promotions — a Power CAT Copilot Studio Kit wrapper that enforces PASS/SOFT-FAIL/HARD-FAIL quality and safety gates (OWASP LLM Top 10 + FSI regulatory overlay, accuracy, fallback, format compliance) across Dev→Test and Test→Prod, with required-reviewer human sign-off before production.

Mapped Controls¶

2.3, 2.5, 2.12, 2.20

Prerequisites¶

Role	Requirement
`power-platform-admin`	Install the Power CAT Copilot Studio Kit (managed solution) into each Dataverse environment (Dev/Test/Prod) and configure its Direct Line connection to the target agent.
`security-admin`	Register a Microsoft Entra ID application with a certificate credential and grant it Dataverse user_impersonation plus read/write on the Kit's mspcat_* tables in each environment; publish the certificate thumbprint and Direct Line secret as GitHub Actions environment secrets.
`github-admin`	Configure the production GitHub Actions environment with required reviewer judep_microsoft (gating Test→Prod regardless of eval score, per framework Decision #8) and copy the workflow files from eval-gate/workflows/ into .github/workflows/.

Verification¶

Run eval-gate/Invoke-EvalGate.ps1 -Environment dev -TestSetId safety-baseline-v1 -WhatIf for a pre-flight check, then a live -PromotionContext DevToTest run against a Kit-provisioned Dev environment; confirm the gate resolves the safety baseline, applies eval-thresholds.json, and emits a result JSON with exit code 0 (PASS), 1 (SOFT-FAIL), or 2 (HARD-FAIL).

Documentation¶

Document
Deployment
Prerequisites
Threat Model

View source on GitHub { .md-button }