Autonomous coding agents: anatomy of a CI fleet incident

The first SELinux flag landed at 04:12 on a Tuesday. By the time the on-call engineer in Groningen opened the audit log on her phone, an autonomous coding agent had rewritten 184 dnf-managed package manifests across nine Fedora build runners. The runners were the entire CI fleet of a 29-person DevOps consultancy. The agent had been told to harmonise the build environment. It had been working since 23:40.

This is the story of how that happened, what stopped it, and the four gates the shop runs now. We were not the consultancy in question. We were on a call with them the following Thursday to compare notes, because we had built a closely related agent for another client and almost shipped the same hole.

The setup

The consultancy runs CI on bare-metal Fedora 41 because the workloads include kernel-module builds for an industrial client. Their fleet is nine runners on a 10GbE switch, each provisioned by a single Ansible play and kept in sync by a weekly dnf upgrade window. A senior engineer had been quietly frustrated for months that the runners drifted. Someone would install gcc-toolset-13 on runner-04 to debug a customer issue, forget to remove it, and the build for tenant X would start preferring it. Manifests in git, hand-edited.

So she did what a competent engineer in 2026 does. She wrote a small CLI wrapper around an autonomous coding agent, gave it shell access to the runners over a bastion, pointed it at /etc/dnf/modules.d and the Ansible role, and told it to audit each runner, reconcile against the canonical manifest, and open a PR per runner with the diff. Read-only, in theory. The agent had been doing exactly this on staging for three weeks. Nothing had blown up.

The change that mattered was small. She had switched the agent from a self-built runner to a vendor SDK the week before, and the vendor's default tool config let the agent call dnf directly instead of just dnf --assumeno. The diff was one line in a YAML config:

tools:
  shell:
    allow_mutating: true   # was: false
    audit_log: /var/log/agent/shell.log
    timeout_seconds: 600

Nobody flagged it in review because the YAML had thirty other lines and the diff message read stack-aligned defaults. The mutating flag is a hint at how the vendor thought about it. The consultancy thought about it as a read-only audit tool. Same agent, two mental models, one config line of distance between them.

The first cascade

At 23:40 the agent started its nightly pass on runner-01. The canonical manifest for runner-01 specifies kernel-devel at the matched kernel ABI, plus a pinned llvm-19. The runner had drifted. It had llvm-20, installed by hand three weeks earlier during a debug session. The agent generated a remediation plan, called dnf downgrade llvm, and that pulled in a transaction that broke clang-tools-extra. The agent saw the broken transaction in its next tool call, decided the manifest itself was the inconsistency, and rewrote the manifest to match what was now installed. Then it committed and pushed.

That loop then ran on runner-02. Runner-02's manifest had been the canonical one. The agent rewrote it to match runner-02's drift. Then runner-03. Then runner-04. The agent was working through the fleet in alphanumeric order. Each rewrite was a single small commit, signed with the bot key, pushed to the working branch on the internal Gitea instance.

By the time the second cascade started, with the agent now reconciling runner-01 against the freshly-changed canonical (which by then was a snapshot of runner-09's drift), 184 manifest files had been touched and roughly 60 of them had triggered dnf transactions that installed or downgraded packages the runner did not need. The agent's confidence in each step was rational given its inputs. It was the inputs that were wrong.

The two things the model was doing well, in fact, were the things that made the failure mode so hard to catch at audit time. It was producing small, well-scoped commits. It was writing crisp, narrowly-scoped commit messages. Every diff in isolation looked like a competent engineer narrowing down drift, one runner at a time. Only the sequence read as wrong, and nothing in the loop was watching the sequence.

How SELinux caught it

SELinux did not catch the manifest rewrites. Those were file writes in a context the agent's user had been granted. What SELinux caught was the second cascade attempting to load a kernel module from a path that no longer matched its expected type. The type=AVC entry in /var/log/audit/audit.log was the first signal anything was wrong. Lightly redacted:

type=AVC msg=audit(1749614132.118:8421): avc:  denied  { module_load }
  for  pid=29411 comm="modprobe"
  path="/usr/lib/modules/6.11.7-300.fc41.x86_64/extra/abn_ind.ko"
  scontext=system_u:system_r:init_t:s0
  tcontext=system_u:object_r:default_t:s0
  tclass=system permissive=0

The denial fired their existing audit2why pager rule. The on-call saw kernel module load denied at 04:12, opened her laptop, and within four minutes had traced it back to the bot's commit log. She killed the agent's session, revoked the bot's SSH key on the bastion, and froze the Gitea branch.

The rollback

Recovery took 41 minutes per runner, in parallel. The shop had a working snapshot strategy already: Btrfs subvolume snapshots of /etc and /var/lib/dnf taken hourly via snapper. The on-call did three things, in order, on each runner.

First, list the snapshots and identify the pair taken at 23:00, before the agent had touched anything:

snapper -c etc list | awk '$2 ~ /^2[0-9]+/ {print}'
snapper -c dnf list | awk '$2 ~ /^2[0-9]+/ {print}'

Second, roll back the dnf state and the etc state to that pair, in a single transaction per runner, so the package database matched the manifest:

snapper -c etc undochange 412..0
snapper -c dnf undochange 198..0
dnf check

Third, revert the Gitea branch to the commit before the bot's first push, and re-run the canonical Ansible play against each runner to verify state convergence. Two runners failed verify (one had a stuck dnf lock from a half-finished transaction). Both were drained and rebuilt from the PXE image. Total wall-clock from the SELinux denial to a green CI: 1 hour 14 minutes. No client builds were lost because the affected tenant's nightly window starts at 06:00.

The forensic pass after recovery took another full day. Every package the agent had touched on every runner had to be checked against the canonical for tenant X's kernel module, because a wrong glibc minor version against that module's ABI would silently miscompile rather than fail at link time. Nothing was wrong, in the end. But the team budgeted the day, and they will budget the same day on the next agent-related incident, because they no longer trust their feel for what the blast radius might be.

Loop architecture, not the model

It is tempting to read this as the agent was too aggressive or the vendor defaults were wrong. Both are true in a narrow sense. Neither is the root cause. The root cause was that the wrapper allowed the agent to write to its own source of truth.

The canonical manifest was meant to be the ground state the agent reconciled toward. The wrapper gave the agent write access to the manifest because earlier versions had to update it during legitimate operator-approved changes. Once the agent could rewrite the manifest, a transient inconsistency (the broken llvm transaction on runner-01) gave the model a tool-shaped excuse to fix the wrong end of the equation. From there the cascade was a straight line.

This is the failure mode OWASP labels LLM08: Excessive Agency: an agent given a tool whose scope exceeds the user's actual intent, in a loop with no break condition. There is also a useful structural counter-argument coming from frameworks like Burr, which treat agent state as an explicit, inspectable graph rather than an emergent property of whatever the model decided to call next. The Groningen incident is a fair advert for that approach. The shop's old setup had the spec, the reconciler, and the writer collapsed into one process with one set of credentials, and the model walked across the seam without noticing it existed.

The four gates the shop runs now

None of these are clever. All of them are the kind of guardrail you write down once, then forget you have, until the audit log saves you.

Gate 1: read-only by default at the tool layer. The agent's shell wrapper now denies any command in a hard-coded mutating list (dnf install, dnf remove, dnf downgrade, rpm -e, git push, git commit, systemctl) unless a human-approved token is present in the call. The token is generated per ticket, valid for ten minutes, and burns on use. The token logic lives outside the agent's reach, in a sidecar service the agent talks to over a unix socket.

Gate 2: the manifest is immutable to the agent. The canonical manifest now lives in a separate repo with branch protection and CODEOWNERS, and the agent has a deploy key that only grants read. Drift PRs from the agent go against a sibling proposed-changes repo that a human merges into canonical. The agent literally cannot edit the file it is reconciling against.

Gate 3: SELinux is enforcing on every runner. This is obvious. It was already true. The point is that it was the only gate that fired in time. If you have been reading the recent Red Hat SELinux guidance as a nice-to-have, read it again as your last line of defence against your own automation. SELinux does not understand intent. That is its job. Yours is to put it in the loop's blast radius before the agent runs.

Gate 4: heartbeat-bounded sessions. The agent's session now expires every 30 minutes and must be re-authorised by a human. The first cascade ran for 4 hours 32 minutes without a person in the loop. A 30-minute heartbeat would have caught it before the second cascade started. The systemd fragment that enforces it:

[Service]
Type=oneshot
ExecStart=/usr/local/bin/agent-session-check
RuntimeMaxSec=1800
PrivateTmp=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/agent/state
NoNewPrivileges=yes
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

What we run into on every agent build now

When we built an autonomous patch-review agent for a Rotterdam logistics platform earlier this year, the thing we ran into was the same shape. Any tool the agent uses to observe state can become a tool to mutate state if the wiring is sloppy. We ended up solving it by splitting observation and mutation into two separate processes with different credentials, where only the mutation process requires a per-action human approval. That is the pattern we now propose for every AI agents engagement that touches production infrastructure.

If you are running an agent against your CI today, open one terminal, run ausearch -m AVC -ts today, and read the last twenty entries. That is your five-minute audit.