Unified Monitoring System Architecture

This document describes the comprehensive architecture of the FSI-AgentGov monitoring system for detecting external changes that may require framework updates.

---

Executive Summary

Purpose: Proactively detect changes in external sources (Microsoft Learn documentation, regulatory notices) that may impact the FSI Agent Governance Framework, enabling timely documentation updates.

Implementation Status: Active and operational as of February 2026.

Coverage: - Microsoft Learn: ~207 URLs monitored daily - Regulatory Sources: Federal Register (SEC, CFTC, OCC, Federal Reserve) + FINRA notices monitored weekly

Key Principle: ONE unified monitoring system with multiple source adapters, not separate independent monitors.

---

System Overview

┌──────────────────────────────────────────────────────────────────────────┐
│                     UNIFIED MONITORING ARCHITECTURE                      │
│                                                                          │
│  ┌────────────────────────────────────────────────────────────────┐     │
│  │                    SHARED INFRASTRUCTURE                        │     │
│  │  ┌─────────────────────────────────────────────────────────┐   │     │
│  │  │  monitoring_shared.py (Unified Framework)                │   │     │
│  │  │  - State management                                      │   │     │
│  │  │  - Report generation                                     │   │     │
│  │  │  - Control mapping                                       │   │     │
│  │  │  - Change classification                                 │   │     │
│  │  └─────────────────────────────────────────────────────────┘   │     │
│  │                                                                 │     │
│  │  ┌────────────────┐         ┌──────────────────────────────┐   │     │
│  │  │ Shared State   │         │ Shared Report Directory      │   │     │
│  │  │ data/monitor-  │         │ reports/monitoring/          │   │     │
│  │  │ state.json     │         │ - learn-changes-*.md         │   │     │
│  │  │                │         │ - regulatory-changes-*.md    │   │     │
│  │  └────────────────┘         └──────────────────────────────┘   │     │
│  └─────────────────────────────────────────────────────────────┘   │     │
│                                                                      │     │
│  ┌────────────────────────────────────────────────────────────────┐     │
│  │                     SOURCE ADAPTERS                            │     │
│  ├────────────────────────────────────────────────────────────────┤     │
│  │  ┌─────────────────────┐     ┌─────────────────────────────┐  │     │
│  │  │ learn_monitor.py    │     │ regulatory_monitor.py       │  │     │
│  │  │ - Fetch Learn URLs  │     │ - Fetch Federal Register    │  │     │
│  │  │ - Hash content      │     │ - Fetch FINRA notices       │  │     │
│  │  │ - Classify changes  │     │ - Keyword matching          │  │     │
│  │  │ - Map to controls   │     │ - Map to controls           │  │     │
│  │  │                     │     │                             │  │     │
│  │  │ Schedule: Daily     │     │ Schedule: Weekly            │  │     │
│  │  │ 6:00 AM UTC         │     │ Wednesday 6:00 AM UTC       │  │     │
│  │  └─────────────────────┘     └─────────────────────────────┘  │     │
│  └────────────────────────────────────────────────────────────────┘     │
│                                                                          │
│  ┌────────────────────────────────────────────────────────────────┐     │
│  │                  AI-ASSISTED REVIEW                             │     │
│  │  ┌──────────────────────────────────────────────────────────┐  │     │
│  │  │ .claude/skills/review-learn-changes.md                   │  │     │
│  │  │ - Learn reports: Auto-draft edits                        │  │     │
│  │  │ - Regulatory reports: Triage only                        │  │     │
│  │  │                                                           │  │     │
│  │  │ Invocation: Manual (user runs /review-learn-changes)     │  │     │
│  │  └──────────────────────────────────────────────────────────┘  │     │
│  └────────────────────────────────────────────────────────────────┘     │
│                                                                          │
└──────────────────────────────────────────────────────────────────────────┘

---

Unified Framework

monitoring_shared.py

Location: scripts/monitoring_shared.py Purpose: Core framework providing shared utilities for all monitoring sources

Key Functions:

Function	Purpose
load_state()	Load monitoring state from unified state file
save_state()	Save monitoring state to unified state file
classify_change()	Classify changes (CRITICAL/HIGH/MEDIUM/NOISE)
map_to_controls()	Map changes to affected framework controls
generate_report()	Generate markdown change reports
normalize_content()	Clean and normalize content for comparison
compute_hash()	Generate SHA-256 content hashes

Design Principle: Source adapters (Learn Monitor, Regulatory Monitor) are thin wrappers that fetch content and delegate to monitoring_shared.py for all common operations.

---

State File

data/monitor-state.json

Purpose: Unified state file tracking all monitored sources

Structure:

{
  "last_updated": "2026-02-04T06:00:00Z",
  "sources": {
    "learn": {
      "urls": {
        "https://learn.microsoft.com/...": {
          "hash": "sha256:abc123...",
          "last_checked": "2026-02-04T06:00:00Z",
          "title": "Managed Environments Overview",
          "normalized_content": "..."
        }
      }
    },
    "regulatory": {
      "federal_register": {
        "last_checked": "2026-02-04T06:00:00Z",
        "items": {
          "2026-02228": {
            "title": "SEC Rule Filing...",
            "hash": "sha256:def456...",
            "agency": "SEC",
            "document_type": "Rule"
          }
        }
      },
      "finra": {
        "last_checked": "2026-02-04T06:00:00Z",
        "items": {
          "trade-reporting-notice-20260114": {
            "title": "Trade Reporting Notice...",
            "hash": "sha256:ghi789...",
            "notice_type": "Trade Reporting"
          }
        }
      }
    }
  }
}

Benefits of Unified State: - Single source of truth for all monitoring - Cross-source correlation possible - Simplified backup and restore - Reduced state file management complexity

---

Report Directory

reports/monitoring/

Purpose: Unified directory for all monitoring reports

Report Types:

Report Pattern	Source	Frequency	Purpose
learn-changes-YYYY-MM-DD.md	Learn Monitor	Daily (if changes)	Microsoft Learn documentation updates
regulatory-changes-YYYY-MM-DD.md	Regulatory Monitor	Weekly (if changes)	Federal Register + FINRA regulatory notices

Report Format (Learn):

# Learn Monitor Report - YYYY-MM-DD

**Run Date:** YYYY-MM-DD
**URLs Monitored:** ~207
**Meaningful Changes:** N

## Summary

[Executive summary table]

## HIGH Priority Changes

### URL: [Microsoft Learn URL]

**Affected Controls:** 1.1, 1.5, 2.3
**Affected Playbooks:** 1.1/portal-walkthrough.md

**What Changed:**
[Diff showing before/after]

Report Format (Regulatory):

# Regulatory Monitor Report - YYYY-MM-DD

**Run Date:** YYYY-MM-DD
**New Items:** N
**Sources:** Federal Register (SEC, CFTC, OCC, Federal Reserve) + FINRA Notices

## Summary

[Executive summary table by category]

## HIGH Priority Items

### [Regulatory Item Title]

**Source:** SEC
**Potentially Affected Controls:** [keyword-matched controls]

[Summary of item]

Why Unified Directory: - Single location for all change reports - Simplified AI-assisted review workflow - Easier backup and archival - Clear organizational structure

---

Source Adapters

Learn Monitor

Script: scripts/learn_monitor.py Workflow: .github/workflows/learn-monitor.yml Schedule: Daily at 6:00 AM UTC

Data Source: docs/reference/microsoft-learn-urls.md (~207 URLs)

Process: 1. Load URLs from watchlist 2. Fetch each URL via HTTP 3. Extract meaningful content (remove headers, navigation, metadata) 4. Normalize content (whitespace, encoding) 5. Compute SHA-256 hash 6. Compare to previous hash in state file 7. If changed, classify change (CRITICAL/HIGH/MEDIUM/NOISE) 8. Map to affected controls via URL-to-control mapping 9. Generate report if meaningful changes detected 10. Update state file

Exit Codes: - 0 - No meaningful changes - 1 - Meaningful changes detected (triggers PR creation) - 2 - Error during execution

PR Creation Logic: - Every Sunday (baseline run), OR - When exit code = 1 (changes detected)

Regulatory Monitor

Script: scripts/regulatory_monitor.py Workflow: .github/workflows/regulatory-monitor.yml Schedule: Weekly on Wednesday at 6:00 AM UTC

Data Sources: - Federal Register API (SEC, CFTC, OCC, Federal Reserve) - FINRA regulatory notices (web scraping)

Process: 1. Query Federal Register API for recent documents from target agencies 2. Scrape FINRA regulatory notices 3. For each new item: - Extract title, abstract, document type - Compute content hash - Check if already seen in state file - If new, classify relevance (CRITICAL/HIGH/MEDIUM/NOISE) via keyword matching - Suggest potentially affected controls based on keywords 4. Generate report if new items detected 5. Update state file

Exit Codes: - 0 - No new items - 1 - New items detected (triggers PR creation) - 2 - Error during execution

Keywords for Control Mapping:

Keyword Category	Keywords	Maps To
AI/ML	artificial intelligence, machine learning, automated decision	1.8, 2.6, 2.15
Recordkeeping	records, retention, books and records, archive	3.1, 3.2, 3.3
Supervision	supervision, review, oversight, monitoring	2.6, 2.15, 3.5
Data Protection	privacy, data security, PII, confidential	1.1, 1.2, 1.3
Content Moderation	harmful content, content review, moderation	1.27, 1.8

---

AI-Assisted Review

Skill: .claude/skills/review-learn-changes.md Invocation: Manual via /review-learn-changes

Learn Report Workflow

When a Learn change report is created:

1. User invokes /review-learn-changes in Claude Code
2. Skill reads reports/monitoring/learn-changes-*.md
3. For each HIGH priority change:
4. Read affected control/playbook
5. Analyze the diff from Microsoft Learn
6. Draft specific edit
7. Present summary to user:
8. Proposed updates (auto-draftable)
9. Flagged for human review (policy/regulatory language)
10. Skipped (formatting/noise)
11. User confirms: "Apply the N updates"
12. Skill applies edits using Edit tool
13. Runs mkdocs build --strict validation
14. User reviews, commits, and pushes

Auto-Draft Eligible Changes: - UI navigation step updates - Date/deadline changes - Feature GA/deprecation notices - URL redirects - New documentation cross-references

Flagged for Human Review: - Policy language changes - New compliance requirements - Security guidance changes - Licensing changes

Regulatory Report Workflow

When a regulatory change report is created:

1. User invokes /review-learn-changes in Claude Code
2. Skill reads reports/monitoring/regulatory-changes-*.md
3. For each CRITICAL/HIGH item:
4. Read suggested affected controls
5. Assess relevance to AI agent governance
6. Validate keyword-based suggestions
7. Present triage summary:
8. Requires human review (with relevance assessment)
9. Out of scope (dismissible)
10. User conducts detailed analysis
11. User determines if framework updates needed

IMPORTANT: Regulatory changes are NEVER auto-edited per CONTRIBUTING.md safety rules. The skill provides triage assistance only.

---

Change Classification

4-Tier System:

Classification	Criteria	Action
CRITICAL	Affects portal-walkthrough.md playbooks, breaking changes, security vulnerabilities	Immediate update required
HIGH	UI steps, policy language, deprecations, new capabilities	Review and update
MEDIUM	General content updates, minor documentation changes	Review optional
NOISE	Metadata, dates, formatting, out-of-scope regulatory items	Ignore

Learn Classification Logic:

• Contains UI keywords (click, navigate, portal, admin center) + affects playbook → CRITICAL
• Contains policy keywords (required, must, prohibited, compliance) → HIGH
• Contains deprecation keywords (deprecated, retired, no longer) → HIGH
• Contains release keywords (preview, GA, generally available) → HIGH
• Substantive content change → MEDIUM
• Metadata/formatting only → NOISE

Regulatory Classification Logic:

• Contains AI/supervision/recordkeeping keywords → HIGH (if SEC/FINRA/OCC/Fed)
• Contains data protection keywords → HIGH (if GLBA-relevant agency)
• Trading rules, fee schedules, non-AI topics → NOISE
• Keyword matching to framework controls → Suggests affected controls

---

Maintenance Procedures

Weekly Maintenance (~30 minutes)

Monday Morning:

1. Check Learn Monitor PR (created Sunday baseline or when changes detected)

   gh pr list --label "learn-watch"
   

2. Review change report in reports/monitoring/learn-changes-*.md

3. Check summary for HIGH/CRITICAL changes

4. Note number of affected controls

5. Run AI-assisted review (if HIGH priority changes exist)

   # In Claude Code
   /review-learn-changes
   

6. Apply updates and validate

   # After skill applies edits
   python3 -m mkdocs build --strict
   

7. Commit and merge PR

   git commit -m "docs: Apply Learn Monitor updates from [date]"
   git push
   gh pr merge --squash
   

Thursday Morning:

1. Check Regulatory Monitor PR (created Wednesday if changes detected)

   gh pr list --label "monitoring"
   

2. Review change report in reports/monitoring/regulatory-changes-*.md

3. Check summary for HIGH items

4. Note suggested affected controls

5. Run AI-assisted triage

   # In Claude Code
   /review-learn-changes
   

6. Conduct detailed analysis for items flagged as "Requires Human Review"

7. Read full regulatory text
8. Assess applicability to AI agent governance

9. Determine if framework updates needed

10. Update framework if needed (separate work session)

11. Create issue tracking regulatory change
12. Plan documentation updates

13. Implement and test

14. Merge PR (updates state file even if no framework changes)

    gh pr merge --squash
    

Adding New URLs to Learn Monitor

Process:

1. Add URL to watchlist

   # Edit docs/reference/microsoft-learn-urls.md
   # Add new URL under appropriate section
   

2. Map URL to controls

   # Edit scripts/learn_monitor.py
   # Update URL_TO_CONTROLS mapping (around line 70-150)
   

3. Run test

   python3 scripts/learn_monitor.py --url "https://learn.microsoft.com/..." --dry-run
   

4. Commit changes

   git add docs/reference/microsoft-learn-urls.md scripts/learn_monitor.py
   git commit -m "feat: Add [topic] to Learn Monitor watchlist"
   

Next scheduled run will pick up the new URL automatically.

Adding New Regulatory Sources

Federal Register Agencies:

Edit scripts/regulatory_monitor.py around line 30:

FEDERAL_REGISTER_AGENCIES = [
    "securities-and-exchange-commission",
    "commodity-futures-trading-commission",
    "comptroller-of-the-currency",
    "federal-reserve-system",
    # Add new agency slug here
]

FINRA Notice Types:

Edit scripts/regulatory_monitor.py around line 250:

FINRA_CATEGORIES = [
    "regulatory-notices",
    "disciplinary-actions",
    # Add new category here
]

Keyword Mapping:

Edit scripts/monitoring_shared.py around line 400:

REGULATORY_KEYWORDS = {
    "ai_governance": {
        "keywords": ["artificial intelligence", "machine learning", ...],
        "controls": ["1.8", "2.6", "2.15"]
    },
    # Add new keyword category here
}

---

Troubleshooting

Issue	Symptoms	Resolution
Workflow not running	No PRs created for days	Check GitHub Actions settings, verify GITHUB_TOKEN permissions
State file corrupted	JSON parsing errors	Restore from backup or delete and re-run monitor (establishes new baseline)
False positives	Too many NOISE changes classified as HIGH	Tune classification keywords in monitoring_shared.py
Missing changes	Known changes not detected	Verify URL in watchlist, check URL_TO_CONTROLS mapping
API rate limits	Federal Register API 429 errors	Workflow runs weekly to avoid limits; if persistent, add retry logic
FINRA scraping failure	Regulatory monitor fails	FINRA HTML structure changed; update scraping logic in regulatory_monitor.py
mkdocs build fails after updates	Broken links or syntax errors	Review AI-drafted edits manually, fix syntax, re-run build

---

Monitoring Approach Evaluation

Requirement (MON-03): Assess monitoring approach with documented rationale.

Chosen Approach: Unified Monitoring with Source Adapters

Rationale:

Why unified framework: - Reduced code duplication: Common operations (state management, report generation, control mapping) implemented once - Consistent classification: All sources use the same 4-tier classification system - Simplified maintenance: Bug fixes and enhancements apply to all sources - Cross-source correlation: Future capability to detect related changes across sources - Single state file: Easier backup, restore, and troubleshooting

Why source adapters: - Separation of concerns: Each adapter focuses on its data source specifics - Independent schedules: Learn runs daily, Regulatory runs weekly (different change frequencies) - Extensibility: Adding new sources (e.g., Azure Updates, Microsoft Tech Community) is straightforward - Different workflows: Learn changes can be auto-drafted, regulatory changes require human review

Why manual AI-assisted review: - Cost control: User decides when to invoke AI assistance (not every run) - Human oversight: Critical for regulatory content per CONTRIBUTING.md - Flexibility: User can review subset of changes, skip low-priority items - Safety: Prevents automated commits of incorrect edits

Alternative Approaches Considered

Alternative 1: Separate Independent Monitors

Description: Maintain completely separate scripts, state files, and report directories for Learn vs. Regulatory.

Pros: - Complete independence (failure in one doesn't affect the other) - Simpler initial implementation

Cons: - Code duplication: 60-70% of code is identical (state management, classification, reporting) - Inconsistent classification: Hard to maintain consistent standards across monitors - Multiple state files: data/learn-state.json + data/regulatory-state.json - Multiple report dirs: reports/learn-changes/ + reports/regulatory-changes/ - Maintenance burden: Bug fixes need to be applied twice

Decision: Rejected. The duplication cost outweighs the independence benefit.

Alternative 2: Fully Automated with GitHub Actions

Description: GitHub Actions workflow automatically invokes Claude API to draft and commit edits when changes detected.

Pros: - Fully automated, no human invocation needed - Faster response to changes

Cons: - API costs: ~$0.14 per Learn Monitor run × 365 days = ~$51/year (plus regulatory runs) - Safety risk: Automated commits of incorrect edits - Claude API key in GitHub Secrets: Security consideration - No human review gate: Violates CONTRIBUTING.md requirement for regulatory content

Decision: Rejected for now. Manual invocation provides better cost control and safety. Could revisit as "Phase 4" enhancement with proper guardrails.

Alternative 3: Polling vs. Push-Based

Description: Instead of polling external sources on a schedule, use webhooks or RSS feeds to get push notifications.

Pros: - More real-time detection - Lower polling overhead

Cons: - Microsoft Learn has no webhooks: Would need RSS feed or undocumented API - Federal Register has RSS: But still requires polling to fetch full content - FINRA has no RSS: Web scraping still required - Complexity: Managing webhook endpoints, authentication, security - Reliability: Push-based systems fail silently; polling failures are obvious

Decision: Rejected. Polling is reliable, simple, and sufficient for the framework's needs (daily/weekly cadence acceptable).

---

State AI Law Monitoring

Status: Manual monitoring (not automated)

Coverage: - Colorado AI Act (SB 24-205) - Effective June 30, 2026 (extended from February 1, 2026 via SB 25B-004) - California AI Transparency Act (AB 2013) - Signed September 2024 - New York City Local Law 144 - Effective July 2023 - Illinois Biometric Information Privacy Act (BIPA) - Relevant to AI - Texas Data Privacy and Security Act (HB 4) - Effective July 2024

Process: - Monthly review of state legislature tracking services - Quarterly review of framework for state law alignment - Ad-hoc updates when new laws enacted

Why not automated: - No unified API for state legislature tracking - High variance in relevance (most state bills not applicable) - Human judgment required to assess applicability to M365 AI agents - Low change frequency (quarterly review sufficient)

Future consideration: Add RSS feeds for key state legislature sites if change frequency increases.

---

Related Documentation

• Learn Monitor Guide: learn-monitor-guide.md - Learn Monitor user documentation
• AI-Assisted Review: learn-monitor-ai-enhancement.md - AI-assisted review implementation
• Claude Code Skill: .claude/skills/review-learn-changes.md - User-invocable skill
• Microsoft Learn URLs: microsoft-learn-urls.md - Monitored URL watchlist
• Contributing Guide: CONTRIBUTING.md - Language guidelines and safety rules

---

FSI Agent Governance Framework v1.2.51 - February 2026