M365 tenant audit: what to check before an agent reads mail

It was a Tuesday in March. A client's IT lead had granted Mail.Read at the tenant scope to a vendor's new email triage agent. The kind of thing you tick during a twenty-minute onboarding call. The agent went live at 14:00. By 14:45 it had pulled 31,000 messages into its vector store, including the HR director's archive of resignation letters, contracts, and salary reviews going back to 2019.

Nobody had touched that HR mailbox in two years. The reason it still held six years of attachments was a Purview retention label called HR-Permanent, auto-applied to anything matching a payslip filename pattern. The label did exactly what it was configured to do. The problem was that the agent's service principal could now read everything the label had quietly preserved.

We rolled the access back inside an hour. The vendor agreed to purge their store and we got written confirmation. No regulator was involved. But the next morning we wrote down the checklist below and ran it on every other M365 tenant we had under management. Twelve tenants since. Four of them had something they did not know about.

The two things this checklist defends against

One: a service principal that can read more than the human who deployed it thinks it can.

Two: data that the business assumes is gone, but Purview has been faithfully retaining for years, ready to be vacuumed up by anything with Mail.Read.

Both are misconfigurations in isolation. The actual incident only happens when they overlap. So the audit looks at both sides at once.

Inventory the service principals before you grant anything new

The first command you run on a tenant you are about to add an agent to is not the grant. It is the inventory. Connect to Graph with read-only scopes and list every application that already holds mail permissions.

Connect-MgGraph -Scopes "Application.Read.All","Directory.Read.All"

Get-MgServicePrincipal -All -Filter "servicePrincipalType eq 'Application'" |
  ForEach-Object {
    $sp = $_
    Get-MgServicePrincipalAppRoleAssignment -ServicePrincipalId $sp.Id |
      ForEach-Object {
        [PSCustomObject]@{
          App        = $sp.DisplayName
          AppId      = $sp.AppId
          Permission = $_.AppRoleId
          Resource   = $_.ResourceDisplayName
        }
      }
  } | Where-Object { $_.Resource -eq "Microsoft Graph" } |
  Export-Csv -Path .\sp-inventory.csv -NoTypeInformation

You are looking for two things in that CSV. Any app you do not recognise. And any app that holds Mail.Read, Mail.ReadWrite, Mail.ReadBasic.All, or full_access_as_app without a corresponding RBAC restriction. The last detail is the killer: a Graph application permission grant is unconditionally tenant-wide unless you have scoped it.

Scope mailbox access before, not after

The Microsoft control that scopes Graph mail permissions to specific mailboxes is RBAC for Applications (the modern replacement for the older ApplicationAccessPolicy). Without it, Mail.Read means every mailbox on the tenant. With it, you can grant the agent's service principal read access to one mail-enabled security group only.

The configuration sits in Exchange Online, not Entra. That is part of why it gets missed. The IT lead who provisioned the app in the Entra admin centre will not see it. Run this before the consent prompt is even shown to your CTO.

Connect-ExchangeOnline

# Create a mail-enabled security group with the mailboxes the agent
# is allowed to read.
New-DistributionGroup `
  -Name "agent-triage-allowed" `
  -Type "Security" `
  -PrimarySmtpAddress "agent-triage-allowed@contoso.com"

Add-DistributionGroupMember `
  -Identity "agent-triage-allowed" `
  -Member "support@contoso.com"

# Register the agent's service principal and bind it to a scope
# that resolves only to members of that group.
New-ServicePrincipal -AppId "00000000-0000-0000-0000-000000000000" `
                     -ServiceId ""

New-ManagementScope -Name "agent-triage-scope" `
  -RecipientRestrictionFilter "MemberOfGroup -eq 'CN=agent-triage-allowed,OU=...'"

New-ManagementRoleAssignment `
  -App "" `
  -Role "Application Mail.Read" `
  -CustomResourceScope "agent-triage-scope"

If the agent vendor refuses to operate inside a scoped role, that is a signal. Walk away or negotiate. The cost of "just give it tenant-wide and we will sort it later" is the incident I described above.

Walk the retention labels before you trust the deletion log

Most M365 admins know retention labels exist. Fewer know which labels are auto-applied, which trigger immutability, and how far back the auto-apply has reached. This is where the HR incident actually started.

Run this in Purview to dump every retention label, its auto-apply rule, and its retention duration.

Connect-IPPSSession

Get-RetentionCompliancePolicy |
  ForEach-Object {
    $policy = $_
    Get-RetentionComplianceRule -Policy $policy.Name | ForEach-Object {
      [PSCustomObject]@{
        Policy       = $policy.Name
        Workload     = ($policy.Workload -join ",")
        Rule         = $_.Name
        Duration     = $_.RetentionDuration
        Action       = $_.RetentionComplianceAction
        ContentQuery = $_.ContentMatchQuery
        Enabled      = $policy.Enabled
      }
    }
  } | Export-Csv -Path .\retention-map.csv -NoTypeInformation

Two columns matter most. Duration set to Unlimited on a workload that includes Exchange. And ContentQuery using sensitive-type matching like SensitiveType:"Netherlands Citizen Service (BSN) Number" or filename patterns. Any agent with mail read will see everything those rules have preserved, whether the user thinks they deleted it or not.

The fix is rarely to remove the retention. The fix is to scope the agent so it cannot read the mailboxes those rules target. Marketing and support mailboxes are usually fine. HR, finance, and the CEO's office are not.

Turn on the audit log before the agent's first week

Mailbox auditing is on by default in M365 for owner, admin, and delegate actions. But application-level reads through Graph land in a different stream. You need both unified audit log and mailbox audit for application impersonation, and you need them on before the agent connects.

Set-OrganizationConfig -AuditDisabled $false

Get-Mailbox -ResultSize Unlimited |
  Set-Mailbox -AuditEnabled $true `
              -AuditLogAgeLimit 180 `
              -AuditOwner @{Add="MailItemsAccessed"}

# Then, 48h after the agent goes live, look at what it actually touched.
Search-UnifiedAuditLog `
  -StartDate (Get-Date).AddDays(-2) `
  -EndDate   (Get-Date) `
  -RecordType ExchangeItem `
  -ResultSize 5000 |
  Where-Object { $_.UserIds -like "*agent-triage*" } |
  Select-Object CreationDate, UserIds, Operations, AuditData |
  Export-Csv -Path .\agent-first-week.csv -NoTypeInformation

The MailItemsAccessed event is the one that tells you what the agent actually opened versus what it had permission to open. If those two numbers diverge by an order of magnitude, you have learned something important about the agent's behaviour before anyone else does.

Cover the service principal in Conditional Access

Conditional Access for workload identities is still under-used. Most tenants apply CA only to users. A service principal connecting from a random hosting region at 03:00 with no IP allowlist is the exact thing CA was built to block, and yet it usually does not. The licensing requires Entra Workload Identities Premium, which is a separate SKU you have to remember to add to the order.

Conditional Access policy: "Restrict agent service principals"

Assignments
  Workload identities: include
    - agent-triage (object id ...)
    - agent-knowledge-base (object id ...)

Conditions
  Locations: include any location, exclude trusted IPs
  Service principal risk: high

Grant
  Block access

You will not catch every misuse this way. You will catch the obvious ones, and "obvious" is most of what goes wrong.

The five-minute version

If you only have time for one pass before the consent prompt, run these five queries.

1. List every service principal with a Mail.* Graph permission. Anything you do not recognise gets revoked.
2. Confirm RBAC for Applications is scoped to a security group for every app on that list. No scope means tenant-wide.
3. Pull retention labels with unlimited duration or sensitive-type queries. Note which mailboxes they touch.
4. Verify mailbox audit is on with MailItemsAccessed for owners and admins. Without it you have no forensics.
5. Confirm a Conditional Access policy targets the agent's service principal. If you do not hold the Workload Identities Premium SKU, write down that gap.

Five queries. Maybe twenty minutes if you have not done it before, five the second time. None of them are heroic. The HR incident I opened with would have been caught on number two.

The silent-failure shape of agent incidents

The reason I keep coming back to this checklist is that misconfigured agent permissions are silent by construction. The agent works. It returns answers. It triages mail. Nothing breaks visibly. The only signal is in the audit log, and only if you turned it on, and only if you are looking.

When the failure mode of an AI system is silent over-reach rather than visible error, your usual ops instincts do not fire. The agent gives you what you asked for plus more, and the "more" never shows up in any dashboard. No alert is triggered when a service principal reads 30,000 messages it did not need to. It just becomes a row in an audit log nobody is querying.

The checklist above is how we make it show up.

What we automated out of this

When we built the email triage agent for a Dutch logistics client last quarter, the thing we ran into was exactly this gap between "the app is provisioned" and "the app is scoped." We ended up wrapping the five queries above into a pre-flight script that runs before any new app is consented to in their tenant, and a second pass that re-runs weekly with the deltas posted to a Teams channel. If you want a version of that for your own stack, our AI agents practice is built on top of audits like this one.

One thing to do today: open the Entra admin centre, filter Enterprise Applications by "API permissions contains Mail," and read the list. If you find a name you do not recognise, you have just started the audit.