Assessment Engine Coverage Matrix
This page is generated by scripts/generate_coverage_matrix.py from assessment/manifest/controls.json and the EVALUATORS registry in assessment/engine/score.py. Do not edit by hand.
It is the honest answer to what does the assessment engine actually automate today? and is intended to prevent confusion between manual by design and evaluator not yet implemented.
Evaluator states
| State | Icon | Meaning |
|---|---|---|
auto_evaluable |
✅ | A bespoke evaluator is registered for the check's pass_condition and the engine can score it from collected telemetry. |
unimplemented_evaluator |
⚠️ | The manifest declares a pass_condition, but no evaluator function is registered yet. The generic fallback returns unknown. |
manual_only |
📝 | The control is manual by design — either automation: manual in the manifest or all collection methods are non-automatable. Reviewer must answer the manual question. |
Summary
By control
| State | Count | Share |
|---|---|---|
| ✅ Auto | 7 | 9.0% |
| 📝 Manual | 38 | 48.7% |
| ⚠️ Unimplemented | 33 | 42.3% |
| Total | 78 | 100% |
By check
| State | Count | Share |
|---|---|---|
| ✅ Auto | 9 | 9.6% |
| 📝 Manual | 20 | 21.3% |
| ⚠️ Unimplemented | 65 | 69.1% |
| Total | 94 | 100% |
Registered evaluators
assessment/engine/score.py registers 11 bespoke evaluator functions:
audit_log_enabledca_policy_requires_mfaca_policy_targets_copilot_studiocopilot_retention_policy_existsfsi_publisher_group_existsgrounding_sources_approvedno_everyone_assignmentno_external_sharing_on_groundingprod_env_has_security_groupprod_env_is_managedshare_everyone_disabled
Drift warning — the following evaluators are registered but no manifest check uses them as a pass_condition. This usually means the manifest condition string drifted from the evaluator key:
ca_policy_targets_copilot_studioprod_env_has_security_group
Per-pillar matrix
Pillar 1 – Security
| Control | Title | State | Auto | Unimpl | Manual | Collection | Caveats |
|---|---|---|---|---|---|---|---|
| 1.1 | Control 1.1: Restrict Agent Publishing by Authorization | ✅ Auto | 3 | 0 | 0 | Graph_API, PPAC_PowerShell | |
| 1.10 | Control 1.10: Communication Compliance Monitoring | ⚠️ Unimplemented | 0 | 1 | 0 | Purview_PowerShell | pass_condition: comm_compliance_policy_exists declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 1.11 | Control 1.11: Conditional Access and Phishing-Resistant MFA | ✅ Auto | 1 | 2 | 0 | Graph_API | pass_condition: signin_frequency_set declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: phishing_resistant_mfa declared in manifest but no bespoke evaluator registered in s… |
| 1.12 | Control 1.12: Insider Risk Detection and Response | ⚠️ Unimplemented | 0 | 1 | 0 | Purview_PowerShell | pass_condition: insider_risk_policy_exists declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 1.13 | Control 1.13: Sensitive Information Types (SITs) and Pattern Recognition | ⚠️ Unimplemented | 0 | 2 | 0 | PPAC_PowerShell, Purview_PowerShell | pass_condition: sit_count_adequate declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: dlp_references_sits declared in manifest but no bespoke evaluator registered in score.… |
| 1.14 | Control 1.14: Data Minimization and Agent Scope Control | 📝 Manual | 0 | 0 | 0 | — | |
| 1.15 | Control 1.15: Encryption: Data in Transit and at Rest | ⚠️ Unimplemented | 0 | 2 | 0 | Graph_API | pass_condition: tls_12_enforced declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: at_rest_encryption_verified declared in manifest but no bespoke evaluator registered in s… |
| 1.16 | Control 1.16: Information Rights Management (IRM) for Documents | 📝 Manual | 0 | 0 | 0 | — | |
| 1.17 | Control 1.17: Endpoint Data Loss Prevention (Endpoint DLP) | ⚠️ Unimplemented | 0 | 1 | 0 | Purview_PowerShell | pass_condition: endpoint_dlp_policy_exists declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 1.18 | Control 1.18: Application-Level Authorization and Role-Based Access Control (RBAC) | ⚠️ Unimplemented | 0 | 2 | 0 | Graph_API | pass_condition: rbac_least_privilege declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: no_excessive_admin declared in manifest but no bespoke evaluator registered in score… |
| 1.19 | Control 1.19: eDiscovery for Agent Interactions | ⚠️ Unimplemented | 0 | 2 | 0 | Purview_PowerShell | pass_condition: ediscovery_agent_scope declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: ediscovery_copilot_content declared in manifest but no bespoke evaluator registere… |
| 1.2 | Control 1.2: Agent Registry and Integrated Apps Management | ⚠️ Unimplemented | 0 | 3 | 0 | Graph_API | pass_condition: agent_inventory_exists declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: auth_mode_configured declared in manifest but no bespoke evaluator registered in s… |
| 1.20 | Control 1.20: Network Isolation and Private Connectivity | 📝 Manual | 0 | 0 | 2 | Azure_API | Manual review required. |
| 1.21 | Control 1.21: Adversarial Input Logging | ⚠️ Unimplemented | 0 | 1 | 0 | Purview_PowerShell | pass_condition: prompt_response_logging declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 1.22 | Control 1.22: Information Barriers for AI Agents | ⚠️ Unimplemented | 0 | 2 | 0 | Purview_PowerShell | pass_condition: ib_policy_active declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: ib_segments_configured declared in manifest but no bespoke evaluator registered in score… |
| 1.23 | Control 1.23: Step-Up Authentication for AI Agent Operations | ⚠️ Unimplemented | 0 | 1 | 0 | Graph_API | pass_condition: stepup_auth_deployed declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 1.24 | Control 1.24: Defender AI Security Posture Management (AI-SPM) | 📝 Manual | 0 | 0 | 0 | Azure_API | |
| 1.25 | Control 1.25: MIME Type Restrictions for File Uploads | ⚠️ Unimplemented | 0 | 2 | 0 | PPAC_PowerShell | pass_condition: mime_zone_compliant declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: blocked_extensions_enforced declared in manifest but no bespoke evaluator registered … |
| 1.26 | Control 1.26: Agent File Upload and File Analysis Restrictions | ⚠️ Unimplemented | 0 | 2 | 0 | Graph_API, PPAC_REST | pass_condition: file_upload_zone_appropriate declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: label_inheritance_configured declared in manifest but no bespoke evaluator r… |
| 1.27 | Control 1.27: AI Agent Content Moderation Enforcement | ⚠️ Unimplemented | 0 | 3 | 0 | Graph_API, PPAC_REST | pass_condition: moderation_level_set declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: moderation_zone_compliant declared in manifest but no bespoke evaluator registered i… |
| 1.28 | Control 1.28: Policy-Based Agent Publishing Restrictions | ⚠️ Unimplemented | 0 | 2 | 0 | PPAC_PowerShell | pass_condition: dlp_publishing_restrictions declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: security_scan_enabled declared in manifest but no bespoke evaluator registere… |
| 1.29 | Control 1.29: Global Secure Access: Network Controls for Copilot Studio Agents | ⚠️ Unimplemented | 0 | 2 | 0 | Graph_API | pass_condition: gsa_profile_linked declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: wcf_policy_configured declared in manifest but no bespoke evaluator registered in scor… |
| 1.3 | Control 1.3: SharePoint Content Governance and Permissions | 📝 Manual | 0 | 0 | 2 | SharePoint_Graph | Manual review required. |
| 1.4 | Control 1.4: Advanced Connector Policies (ACP) | ⚠️ Unimplemented | 0 | 3 | 0 | PPAC_PowerShell | pass_condition: dlp_policy_exists declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: acp_allowlist_configured declared in manifest but no bespoke evaluator registered in sc… |
| 1.5 | Control 1.5: Data Loss Prevention (DLP) and Sensitivity Labels | ⚠️ Unimplemented | 0 | 2 | 0 | PPAC_PowerShell, Purview_PowerShell | pass_condition: dlp_scope_covers_agents declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: sensitivity_labels_enabled declared in manifest but no bespoke evaluator register… |
| 1.6 | Control 1.6: Microsoft Purview DSPM for AI | ⚠️ Unimplemented | 0 | 1 | 0 | Purview_PowerShell | pass_condition: dspm_policy_exists declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 1.7 | Control 1.7: Comprehensive Audit Logging and Compliance | ✅ Auto | 1 | 1 | 0 | Purview_PowerShell | pass_condition: audit_plan_tier_adequate declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 1.8 | Control 1.8: Runtime Protection and External Threat Detection | 📝 Manual | 0 | 0 | 1 | Sentinel_KQL | Manual review required. |
| 1.9 | Control 1.9: Data Retention and Deletion Policies | ✅ Auto | 1 | 1 | 0 | Purview_PowerShell | pass_condition: retention_duration_adequate declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
Pillar 2 – Management
| Control | Title | State | Auto | Unimpl | Manual | Collection | Caveats |
|---|---|---|---|---|---|---|---|
| 2.1 | Control 2.1: Managed Environments | ✅ Auto | 1 | 1 | 0 | PPAC_PowerShell | pass_condition: managed_policies_enforced declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 2.10 | Control 2.10: Patch Management and System Updates | ⚠️ Unimplemented | 0 | 1 | 0 | PPAC_PowerShell | pass_condition: version_info_available declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 2.11 | Control 2.11: Bias Testing and Fairness Assessment | 📝 Manual | 0 | 0 | 0 | — | |
| 2.12 | Control 2.12: Supervision and Oversight (FINRA Rule 3110) | 📝 Manual | 0 | 0 | 0 | — | |
| 2.13 | Control 2.13: Documentation and Record Keeping | 📝 Manual | 0 | 0 | 0 | — | |
| 2.14 | Control 2.14: Training and Awareness Program | 📝 Manual | 0 | 0 | 0 | — | |
| 2.15 | Control 2.15: Environment Routing and Auto-Provisioning | ⚠️ Unimplemented | 0 | 2 | 0 | PPAC_PowerShell | pass_condition: routing_rules_configured declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: routing_zone_enforced declared in manifest but no bespoke evaluator registered i… |
| 2.16 | Control 2.16: RAG Source Integrity Validation | 📝 Manual | 0 | 0 | 1 | SharePoint_Graph | Manual review required. |
| 2.17 | Control 2.17: Multi-Agent Orchestration Limits | ⚠️ Unimplemented | 0 | 1 | 0 | PPAC_PowerShell | pass_condition: orchestration_limits_set declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 2.18 | Control 2.18: Automated Conflict of Interest Testing | 📝 Manual | 0 | 0 | 0 | — | |
| 2.19 | Control 2.19: Customer AI Disclosure and Transparency | 📝 Manual | 0 | 0 | 0 | — | |
| 2.2 | Control 2.2: Environment Groups and Tier Classification | ⚠️ Unimplemented | 0 | 2 | 0 | PPAC_PowerShell | pass_condition: env_groups_exist declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: tier_classification_present declared in manifest but no bespoke evaluator registered in … |
| 2.20 | Control 2.20: Adversarial Testing and Red Team Framework | 📝 Manual | 0 | 0 | 0 | — | |
| 2.21 | Control 2.21: AI Marketing Claims and Substantiation | 📝 Manual | 0 | 0 | 0 | — | |
| 2.22 | Control 2.22: Inactivity Timeout Enforcement | ⚠️ Unimplemented | 0 | 2 | 0 | PPAC_REST | pass_condition: timeout_configured declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: timeout_value_adequate declared in manifest but no bespoke evaluator registered in sco… |
| 2.23 | Control 2.23: User Consent and AI Disclosure Enforcement | 📝 Manual | 0 | 0 | 0 | — | |
| 2.24 | Control 2.24: Agent Feature Enablement and Restriction Governance | ⚠️ Unimplemented | 0 | 2 | 0 | PPAC_PowerShell | pass_condition: genai_flags_reviewed declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: external_plugins_controlled declared in manifest but no bespoke evaluator registered… |
| 2.25 | Control 2.25: Microsoft Agent 365 — Admin Center Governance Console | ⚠️ Unimplemented | 0 | 2 | 0 | Graph_API | pass_condition: agent365_console_accessible declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: agent365_inventory_visible declared in manifest but no bespoke evaluator regi… |
| 2.26 | Control 2.26: Entra Agent ID — Identity Governance for Agents | ⚠️ Unimplemented | 0 | 2 | 0 | Graph_API | pass_condition: agent_ids_registered declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: access_packages_configured declared in manifest but no bespoke evaluator registered … |
| 2.3 | Control 2.3: Change Management and Release Planning | ⚠️ Unimplemented | 0 | 1 | 0 | PPAC_PowerShell | pass_condition: approval_workflow_exists declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 2.4 | Control 2.4: Business Continuity and Disaster Recovery | 📝 Manual | 0 | 0 | 1 | SharePoint_Graph | Manual review required. |
| 2.5 | Control 2.5: Testing, Validation, and Quality Assurance | 📝 Manual | 0 | 0 | 0 | — | |
| 2.6 | Control 2.6: Model Risk Management (OCC Bulletin 2026-13 / Fed SR 26-2) | 📝 Manual | 0 | 0 | 0 | — | |
| 2.7 | Control 2.7: Vendor and Third-Party Risk Management | 📝 Manual | 0 | 0 | 0 | — | |
| 2.8 | Control 2.8: Access Control and Segregation of Duties | ⚠️ Unimplemented | 0 | 2 | 0 | Graph_API | pass_condition: sod_matrix_match declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: no_dev_prod_overlap declared in manifest but no bespoke evaluator registered in score.py… |
| 2.9 | Control 2.9: Agent Performance Monitoring and Optimization | ⚠️ Unimplemented | 0 | 1 | 0 | PPAC_PowerShell | pass_condition: usage_metrics_available declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
Pillar 3 – Reporting
| Control | Title | State | Auto | Unimpl | Manual | Collection | Caveats |
|---|---|---|---|---|---|---|---|
| 3.1 | Control 3.1: Agent Inventory and Metadata Management | ⚠️ Unimplemented | 0 | 2 | 0 | Graph_API | pass_condition: agent_inventory_exportable declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: all_agents_classified declared in manifest but no bespoke evaluator registered… |
| 3.10 | Control 3.10: Hallucination Feedback Loop | 📝 Manual | 0 | 0 | 0 | — | |
| 3.11 | Control 3.11: Centralized Agent Inventory Enforcement | 📝 Manual | 0 | 0 | 0 | — | |
| 3.12 | Control 3.12: Agent Governance Exception and Override Management | 📝 Manual | 0 | 0 | 0 | — | |
| 3.13 | Control 3.13: Agent 365 Admin Center Analytics and Reporting | 📝 Manual | 0 | 0 | 0 | — | |
| 3.14 | Control 3.14: Agent 365 Observability SDK and Custom Agent Telemetry | 📝 Manual | 0 | 0 | 0 | — | |
| 3.2 | Control 3.2: Usage Analytics and Activity Monitoring | 📝 Manual | 0 | 0 | 0 | — | |
| 3.3 | Control 3.3: Compliance and Regulatory Reporting | 📝 Manual | 0 | 0 | 1 | SharePoint_Graph | Manual review required. |
| 3.4 | Control 3.4: Incident Reporting and Root Cause Analysis | 📝 Manual | 0 | 0 | 1 | SharePoint_Graph | Manual review required. |
| 3.5 | Control 3.5: Cost Allocation and Budget Tracking | ⚠️ Unimplemented | 0 | 2 | 0 | PPAC_PowerShell | pass_condition: usage_dashboard_enabled declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: data_export_configured declared in manifest but no bespoke evaluator registered i… |
| 3.6 | Control 3.6: Orphaned Agent Detection and Remediation | 📝 Manual | 0 | 0 | 0 | — | |
| 3.7 | Control 3.7: PPAC Security Posture Assessment | ⚠️ Unimplemented | 0 | 2 | 0 | PPAC_PowerShell | pass_condition: security_posture_retrievable declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. pass_condition: posture_score_adequate declared in manifest but no bespoke evaluator registe… |
| 3.8 | Control 3.8: Copilot Hub and Governance Dashboard | 📝 Manual | 0 | 0 | 0 | — | |
| 3.9 | Control 3.9: Microsoft Sentinel Integration | 📝 Manual | 0 | 0 | 2 | Sentinel_KQL | Manual review required. |
Pillar 4 – SharePoint
| Control | Title | State | Auto | Unimpl | Manual | Collection | Caveats |
|---|---|---|---|---|---|---|---|
| 4.1 | Control 4.1: SharePoint Information Access Governance (IAG) / Restricted Content Discovery | 📝 Manual | 0 | 0 | 2 | SharePoint_Graph | Manual review required. |
| 4.2 | Control 4.2: Site Access Reviews and Certification | 📝 Manual | 0 | 0 | 1 | SharePoint_Graph | Manual review required. |
| 4.3 | Control 4.3: Site and Document Retention Management | 📝 Manual | 0 | 0 | 2 | SharePoint_Graph | Manual review required. |
| 4.4 | Control 4.4: Guest and External User Access Controls | ✅ Auto | 1 | 0 | 1 | SharePoint_Graph | Manual review required. |
| 4.5 | Control 4.5: SharePoint Security and Compliance Monitoring | 📝 Manual | 0 | 0 | 1 | SharePoint_Graph | Manual review required. |
| 4.6 | Control 4.6: Grounding Scope Governance | ✅ Auto | 1 | 1 | 0 | PPAC_PowerShell, SharePoint_Graph | pass_condition: grounding_scope_restricted declared in manifest but no bespoke evaluator registered in score.py. Result will be unknown. |
| 4.7 | Control 4.7: Microsoft 365 Copilot Data Governance | 📝 Manual | 0 | 0 | 0 | — | |
| 4.8 | Control 4.8: Item-Level Permission Scanning for Agent Knowledge Sources | 📝 Manual | 0 | 0 | 2 | SharePoint_Graph | Manual review required. |
| 4.9 | Control 4.9: Embedded File Content Governance | 📝 Manual | 0 | 0 | 0 | — |
How to add a new evaluator
- Add a
_eval_<name>(collected, source_key)function toassessment/engine/score.pyreturning(passed: bool | None, evidence: str). - Register it in the
EVALUATORSdict using the same string as the manifest'spass_condition. - Add a fixture and a unit test in
assessment/tests/. - Re-run
python scripts/generate_coverage_matrix.pyand commit the regenerateddocs/reference/assessment-coverage.md.
FSI Agent Governance Framework v1.6.2 - May 2026