Skip to content

Control 3.14: Agent 365 Observability SDK and Custom Agent Telemetry

Control ID: 3.14 Pillar: Reporting Regulatory Reference: FINRA Rule 4511 (General Requirements for Books and Records), SEC Rule 17a-3 (Records to Be Made by Certain Exchange Members, Brokers, and Dealers), SEC Rule 17a-4 (Records to Be Preserved by Certain Exchange Members, Brokers, and Dealers), OCC Bulletin 2011-12 (Sound Practices for Model Risk Management), SOX Section 302 (Corporate Responsibility for Financial Reports), SOX Section 404 (Management Assessment of Internal Controls) Last UI Verified: April 2026 Governance Levels: Baseline / Recommended / Regulated

Agent 365 GA (May 1, 2026) — SDK Integration Maturing

The Agent 365 Observability SDK and its integration with the M365 Admin Center metrics dashboard will benefit from the Agent 365 GA release on May 1, 2026. At GA, tenants with Agent 365 or Microsoft 365 E7 licensing gain access to the full observability surface without separate Frontier enrollment. The Entra agentSignIn log type and Is Agent = Yes sign-in filter remain in Preview. MicrosoftServicePrincipalSignInLogs as an Entra diagnostic setting is in Public Preview and requires explicit opt-in. Core SDK packages (Python, JavaScript, .NET) are available for implementation; SDK feature scope may expand at or after GA. Validate features against current Microsoft Learn documentation before production deployment. Last UI verification: March 2026 (Frontier preview); re-verify after May 1, 2026.

Objective

Require the implementation of the Microsoft Agent 365 Observability SDK for all custom and non-Microsoft-built AI agents deployed in the tenant, so that agent telemetry (sessions, tool calls, exceptions, and performance metrics) flows into Microsoft Purview, Microsoft Defender, and the M365 Admin Center Analytics dashboard in support of FINRA Rule 4511 audit trail requirements and SEC Rule 17a-3/17a-4 recordkeeping obligations.

Why This Matters for FSI

Custom AI agents — including agents built on Azure AI Foundry, third-party frameworks, or proprietary firm development platforms — produce no automated telemetry visible to M365 Admin Center unless the Agent 365 Observability SDK is implemented. This creates a supervisory blind spot that regulators may interpret as a failure of the firm's written supervisory procedures under FINRA Rule 3110.

The regulatory stakes are significant:

  • FINRA Rule 4511: Requires broker-dealers to make and keep books and records in the manner prescribed by SEC Rules 17a-3 and 17a-4. Agent session logs, tool call records, and exception events may constitute business records when those agents are performing regulated activities. Without the Observability SDK, these records may not exist in a retrievable, examiner-accessible format. Firms should consult counsel on which specific agent activities trigger recordkeeping obligations.
  • FINRA Regulatory Notice 25-07 (AI Tools Guidance): Reminds firms that supervisory and recordkeeping obligations apply when AI tools are used in regulated activities. The Observability SDK helps establish the audit trail required to evidence supervision under existing FINRA Rules 3110 and 4511. Note: 25-07 is interpretive guidance on AI use, not a separate retention-duration rule — retention durations derive from FINRA 4511 / SEC 17a-4.
  • SEC Rule 17a-4: Requires records to be preserved in a non-rewriteable, non-erasable format for defined retention periods. The agentSignIn log and Purview audit records generated by the Observability SDK provide the structured, tamper-evident records that help meet this requirement.
  • OCC Bulletin 2011-12 and Federal Reserve SR 11-7: Establish model risk management expectations for the use of quantitative models and automated decision systems. AI agents used in financial analysis, credit assessment, or customer-facing functions may be subject to model risk governance, which requires observable, auditable performance records. The Observability SDK's exception rate and session data helps support model performance monitoring obligations; firms should map specific agents to their model inventory and risk tier.
  • SOX 302/404: Management certifications that internal controls over financial reporting are effective implicitly cover automated processes that contribute to financial reporting workflows. An agent without telemetry cannot be tested for control effectiveness.
  • Cross-Tenant and Forensic Investigation: The new Entra agentSignIn log attributes — including AppOwnerTenantId, ResourceOwnerTenantId, SessionID, SourceAppClientID, and ASN (Autonomous System Number) — enable precise attribution of agent actions across multi-tenant architectures and support forensic investigation of anomalous agent behavior, consistent with the SEC's cybersecurity disclosure rules and FINRA's guidance on cybersecurity practices.

Control Description

Capability Description
Agent 365 Observability SDK Microsoft SDK built on the OpenTelemetry (OTel) industry standard. Required for custom (non-Microsoft-built) agents to appear in M365 Admin Center analytics. Available for Python, JavaScript, and .NET. Enables IT admins to monitor custom agent activity in the same centralized dashboard used for Microsoft-built agents.
Session Telemetry Captures the full agent invocation lifetime — start time, end time, session duration, and outcome — for every agent session. Provides the authoritative record of when an agent was active and what it accomplished.
Tool Call Telemetry Records every tool execution within an agent session, including tool name, input parameters (where permitted by data handling policy), execution time, and result status. Provides granular traceability for multi-step agent workflows.
Exception Telemetry Captures all agent errors and failures with structured error codes, stack traces (where applicable), and contextual session data. Feeds exception rate metrics in the Admin Center dashboard and triggers Defender alerts.
Performance Metrics Session duration, response times, and throughput metrics. Supports capacity planning and model performance monitoring under OCC 2011-12.
Purview Audit Log Integration Observability telemetry flows automatically into Microsoft Purview Unified Audit Log, providing compliance-grade, tamper-evident records accessible to eDiscovery, compliance investigations, and regulatory examinations.
Defender Integration Exception and anomaly events feed into Microsoft Defender for threat detection and incident response workflows. Enables correlation of agent behavior anomalies with security events.
Entra agentSignIn Logs (Preview) New resource type in Entra sign-in logs specifically for agent sign-in activity. Distinguishes agent authentication from human user authentication. Filterable via "Is Agent = Yes" in Entra admin center.
MicrosoftServicePrincipalSignInLogs (Public Preview) Optional Entra diagnostic log stream capturing first-party service-to-service token requests between Microsoft-owned services. Opt-in via Entra diagnostic settings. Provides visibility into Microsoft platform-level agent authentication events.
New Entra Sign-In Attributes AppOwnerTenantId: tenant owning the application. ResourceOwnerTenantId: tenant owning the accessed resource. SessionID: correlates events across a single agent session. SourceAppClientID: identifies the originating agent application. UserAgent: client application string. ASN (Autonomous System Number): network-level attribution for agent traffic.
Auto-Registration (Microsoft Products) Agents built on Copilot Studio and Azure AI Foundry are automatically integrated — no SDK implementation required. The Observability SDK is required only for non-Microsoft-built agents.
Admin Center Integration SDK-instrumented agents contribute session, exception rate, and runtime data to the hero metrics on the Agent 365 Overview page (Control 3.13). Without SDK implementation, custom agents are invisible to these metrics.

Key Configuration Points

  • SDK Availability: Python (microsoft-agents-a365), JavaScript (npm package), .NET (NuGet packages). See PowerShell Setup playbook for full package inventory.
  • Environment Variable: Set ENABLE_A365_OBSERVABILITY_EXPORTER=true in the agent's runtime environment to activate log export to the Agent 365 service.
  • Token Resolver: The SDK requires a token resolver function that provides valid Microsoft identity credentials for the agent service principal. The agent must be registered in Microsoft Entra ID.
  • Entra Diagnostic Settings — MicrosoftServicePrincipalSignInLogs: Navigate to Entra Admin Center > Monitoring > Diagnostic settings > Add diagnostic setting > Select MicrosoftServicePrincipalSignInLogs > Route to Log Analytics workspace, Storage Account (for WORM retention), or Event Hub.
  • agentSignIn Log Access: Entra Admin Center > Monitoring > Sign-in logs > Select "Service principal sign-ins" tab > Filter "Is Agent = Yes".
  • Purview Audit Log Verification: Microsoft Purview compliance portal > Audit > Search > Filter by AgentSession or AgentToolCall record types to confirm telemetry ingestion.
  • Retention Configuration: Configure Purview audit log retention policies for agent telemetry records to help meet the FINRA 4511 / SEC 17a-4 minimum retention (6 years for most broker-dealer books and records). Default Purview retention (1 year on E3, longer on E5) may be insufficient and should be explicitly extended.

Custom Agent Metric Gap — Pre-SDK State

Any custom agent deployed prior to SDK implementation will have NO historical telemetry in Purview or Defender for the period before SDK activation. Document the deployment date of all custom agents and the SDK implementation date to establish the audit trail completeness boundary. Disclose this gap to Internal Audit and include it in the SOX IT general controls narrative if the agent performs functions relevant to financial reporting.

Zone 3 — Mandatory SDK Implementation

For Zone 3 (Regulated) tenants, SDK implementation is mandatory for ALL custom agents before those agents may be deployed into production environments. Deploying a custom agent in a Zone 3 environment without Observability SDK instrumentation violates this control and must be treated as a control deficiency requiring remediation within 30 days of identification.

OpenTelemetry Foundation

The Agent 365 Observability SDK is built on the OpenTelemetry (OTel) CNCF standard. Organizations with existing OTel infrastructure (e.g., Azure Monitor, Jaeger, or third-party APM platforms) can extend their existing telemetry pipelines to include Agent 365 data. Consult with your observability platform vendor before implementation to avoid duplicating telemetry streams.

Zone-Specific Requirements

Zone Requirement Rationale
Zone 1 — Baseline SDK implementation not required for standard Copilot Agent Builder agents (auto-instrumented by Microsoft). If any custom or third-party agents are deployed, document their presence and create a remediation plan for SDK implementation within 90 days. Baseline tenants are expected to use standard Microsoft-built agents with automatic instrumentation. Documentation of custom agent gaps prevents undetected supervisory blind spots.
Zone 1 — Baseline Review Entra sign-in logs monthly and apply "Is Agent = Yes" filter to understand the volume of agent authentication events. Establishes baseline familiarity with agent authentication telemetry before more stringent monitoring is required.
Zone 2 — Recommended SDK implementation is required for any custom or third-party agent deployed in the tenant. SDK must be implemented and verified before the agent is promoted to production. Zone 2 firms have sufficient AI deployment complexity that unmonitored custom agents create material supervisory gaps. Pre-production verification gates ensure no custom agent enters production without telemetry.
Zone 2 — Recommended Verify that each SDK-instrumented agent appears in M365 Admin Center Agent Registry and contributes to hero metrics within 48 hours of deployment. Confirms SDK integration is functional; provides the compliance record that the agent is within the supervisory perimeter.
Zone 2 — Recommended Review Entra agentSignIn logs weekly. Document review in supervisory log. Weekly cadence supports active supervisory program; documentation satisfies FINRA 3110 written evidence requirements.
Zone 3 — Regulated SDK implementation mandatory for ALL custom agents prior to production deployment. No exceptions without CCO and CISO documented waiver. Absolute requirement for regulated environments; no custom agent may operate outside the supervisory telemetry perimeter.
Zone 3 — Regulated Full telemetry must flow to both Microsoft Purview and Microsoft Defender. Verify integration for each agent using Purview audit search and Defender alert review within 24 hours of deployment. Dual-destination telemetry ensures records are preserved in a compliance-grade system (Purview) AND that security anomalies are detectable in real time (Defender).
Zone 3 — Regulated Daily review of exception events in Purview audit log or Defender incidents queue. Exception events must be triaged within 24 hours and root cause documented within 5 business days. Daily review satisfies continuous supervision expectation under FINRA 3110(b) for high-risk automated systems. Documented root cause supports OCC 2011-12 model performance monitoring obligation.
Zone 3 — Regulated agentSignIn logs should be exported and retained for a minimum of 6 years to help meet FINRA Rule 4511. Storage should be WORM-compliant to help meet SEC Rule 17a-4(f). Configure Entra diagnostic settings to route MicrosoftServicePrincipalSignInLogs to WORM-compliant Azure Blob Storage with a locked immutability policy. 6 years is the standard FINRA 4511 retention floor for most broker-dealer records. WORM (non-rewriteable, non-erasable) storage is required by SEC 17a-4(f) for broker-dealer electronic records; firms must validate the specific configuration with counsel and qualified third parties as required by 17a-4.
Zone 3 — Regulated Configure Entra diagnostic settings to enable MicrosoftServicePrincipalSignInLogs stream. Route to Log Analytics workspace AND WORM storage. Provides both real-time queryable telemetry (Log Analytics) and long-term immutable records (WORM storage) from a single diagnostic settings configuration.

Roles & Responsibilities

Role Responsibility
Application Development / AI Engineering Implement Agent 365 Observability SDK in all custom agent code. Configure service_name, service_namespace, and token_resolver per SDK specification. Set ENABLE_A365_OBSERVABILITY_EXPORTER=true in production runtime. Verify telemetry flow to Admin Center before production release.
IT Operations / Platform Engineering Configure Entra diagnostic settings for MicrosoftServicePrincipalSignInLogs. Provision WORM-compliant Azure Blob Storage for agentSignIn log retention. Configure Log Analytics workspace routing for real-time query capability. Maintain SDK package versions and apply security updates.
Chief Information Security Officer (CISO) Own the security telemetry integration between Observability SDK and Microsoft Defender. Establish incident response procedures for agent-related Defender alerts. Issue documented waivers (with CCO co-approval) for any Zone 3 exceptions to mandatory SDK requirement.
Compliance Officer (CCO) Own the regulatory recordkeeping program for agentSignIn logs and Purview audit records. Confirm FINRA 4511 / SEC 17a-4 retention periods are configured correctly and that supervisory procedures reflect FINRA Notice 25-07 expectations for AI tool use. Co-approve any Zone 3 waivers. Include agent telemetry coverage in annual compliance program review.
IT Risk / Governance Lead Maintain a registry of custom agents and their SDK implementation status. Conduct monthly verification that all custom agents are contributing telemetry to Admin Center metrics. Report SDK implementation gaps to CCO and CISO.
Internal Audit Annually test that SDK-instrumented agents are producing telemetry records in Purview. Verify agentSignIn log retention settings against FINRA 4511 / SEC 17a-4 requirements. Include Observability SDK implementation in IT general controls testing scope for SOX 404 purposes.
Business Line Agent Owners Ensure that business-line-sponsored custom agents are developed with SDK instrumentation as a non-negotiable development requirement. Participate in pre-production verification testing confirming telemetry visibility.
Control Relationship
1.7 — Comprehensive Audit Logging Audit logging foundation. Control 1.7 establishes the firm's overall M365 audit logging policy. The agentSignIn log type and Observability SDK telemetry extend the 1.7 audit trail specifically to agent authentication and activity events.
3.9 — Microsoft Sentinel Integration Sentinel ingestion of observability telemetry. Firms with Sentinel deployments should route Observability SDK exception events and agentSignIn logs to Sentinel for advanced threat detection and SIEM correlation. 3.9 defines the data connector and analytic rule configuration.
3.11 — Agent 365 Admin Center Analytics Dashboard consumer. The hero metrics visible in the 3.11 Admin Center Overview page are populated by the telemetry produced by SDK-instrumented agents. Without 3.14 implementation, custom agent data is absent from 3.13 metrics, creating a false picture of the supervisory dashboard.
3.2 — Usage Analytics Operational analytics complement. 3.2 covers per-agent usage analytics accessible through Copilot usage reports; 3.14 provides the underlying telemetry infrastructure that makes custom agent data available to both 3.2 and 3.13 reporting.
3.1 — Agent Inventory Inventory confirmation. The Agent Registry (Control 3.1) lists agents by registration; SDK telemetry confirms which registered agents are actively generating sessions. Discrepancies between registered agents (3.1) and agents generating telemetry (3.14) indicate inactive or mis-configured agents requiring investigation.

Implementation Playbooks

The following playbooks provide step-by-step implementation guidance for Control 3.14:

  1. Portal Walkthrough — Configuring Entra Diagnostic Settings and Verifying Telemetry in Admin Center
  2. PowerShell and SDK Setup — Installing the Observability SDK and Configuring Log Retention
  3. Verification Testing — Confirming Telemetry Flow to Purview, Defender, and Admin Center
  4. Troubleshooting — Resolving SDK Integration and Telemetry Ingestion Issues

Verification Criteria

  1. All custom agents deployed in the tenant have Agent 365 Observability SDK implemented (Python, JavaScript, or .NET as appropriate for the agent's runtime).
  2. Each SDK-instrumented custom agent is visible in the M365 Admin Center Agent Registry (Agents > All Agents) and contributes session data to the hero metrics dashboard.
  3. Observability telemetry (agent sessions, tool calls, exceptions) is confirmed to be flowing to Microsoft Purview Unified Audit Log via audit search returning AgentSession or equivalent record types.
  4. Observability telemetry is confirmed to be flowing to Microsoft Defender, with at least one test exception event generating a reviewable alert or incident entry.
  5. Entra sign-in logs are reviewed for agent activity using the "Is Agent = Yes" filter on the weekly cadence required by the firm's Zone designation.
  6. MicrosoftServicePrincipalSignInLogs Entra diagnostic setting is configured and routing to Log Analytics workspace (Zone 3 requirement).
  7. agentSignIn log retention policy is configured for a minimum of 6 years to help meet FINRA Rule 4511; storage destination is WORM-compliant to help meet SEC Rule 17a-4(f) (Zone 3 requirement). Firms should retain qualified third-party attestations as required by 17a-4(f)(3).
  8. Exception events are reviewed and root cause documented within the SLA defined in Zone-Specific Requirements.
  9. A registry of custom agents and their SDK implementation status is maintained and current, with no undocumented gaps.
  10. Internal Audit has completed at least one annual verification of Purview telemetry records for each SDK-instrumented agent.

Additional Resources

Documentation Currency

Agent 365 Observability SDK package names, configuration APIs, and Entra log schema attributes are subject to change during the Preview period. Always validate SDK installation instructions against current Microsoft Learn documentation and the official SDK repository before implementing in production. This control was last UI-verified in March 2026.

Updated: April 2026 | Version: v1.4.0 | UI Verification Status: Current