Most multi-agent demos I've seen would fail a Friday afternoon audit. They work because the happy path is short, the model is large, and nobody asked who approved what. That's fine for a hackathon. It's not fine for a law firm running discovery, an accountant closing a quarter, or a public body answering a parliamentary question. The hard part of multi-agent systems isn't getting agents to talk to each other — that's a weekend's work. The hard part is making the system behave the same way on Tuesday as it did on Monday, leaving a trail you can defend, and stopping it cold when something drifts. This article is about how I think about governance for those systems, built for Irish and EU regulated environments where "the model decided" is not an answer.
Why multi-agent systems break the old governance model
Single-model AI governance is, broadly, a solved problem on paper. You log the prompt, log the response, version the model, get sign-off, and run a periodic review. The auditor reads a row, sees an input and an output, and ticks a box. It's not perfect but it maps cleanly onto existing controls.
Multi-agent systems break that mapping in three ways. First, the input to any given agent is the output of another agent, which means the "prompt" you log is itself synthetic and may not exist in any human-readable form. Second, the chain is non-deterministic: the same user request can take five different paths through five different agents depending on intermediate scores, retrieval hits, or tool outputs. Third, responsibility diffuses — if a planner agent dispatches a worker that calls a tool that mutates a record, who authorised the mutation? The user? The planner? The worker? The toolmaker? Without a deliberate answer, the honest answer is "nobody", and that's the answer the regulator will write down.
The EU AI Act, GDPR Article 22, and the sectoral codes the Central Bank and the Data Protection Commission operate under all assume a controllable, explainable decision process. Multi-agent systems are explainable only if you design them to be. They are not explainable by default.
The planner-worker-auditor pattern
The pattern I've settled on after building the Intelligence Brain is planner-worker-auditor. It's not novel — variants exist in distributed systems literature going back decades — but applied to LLM agents it forces the right shape of accountability.
The planner receives the user request and produces a plan: an ordered list of steps, each step naming a worker, the inputs it should receive, and the expected output schema. The planner does not execute anything. It writes a plan and stops. That plan is a first-class object — stored, versioned, signed, reviewable. If a human needs to approve the plan before execution, that's a config flag, not a rewrite.
The workers are narrow. Each one does one job: extract entities from a document, compute a tax position, draft a clause, query a database. Workers do not call other workers. They take typed input, produce typed output, and log both. If a worker needs more information, it returns to the planner, which decides whether to amend the plan. This is the discipline that stops emergent loops where agents call each other in a circle until the budget runs out.
The auditor runs alongside, not after. It reads the plan, reads each worker's input and output, and checks invariants — did the plan touch a data class the user isn't entitled to see, did a worker produce output that contradicts an earlier step, did a tool call mutate something that wasn't in the plan. The auditor can halt execution. In regulated deployments, halting is the default; resumption requires a human.
This sounds heavy. It is heavy. It is also the only shape I've found that survives a real audit without retrofitting.
State, memory, and the provenance problem
Agents need memory to be useful. Memory is also the largest single source of governance pain. Once an agent remembers something across sessions, every output is potentially conditioned on data the user can't see and the auditor can't reconstruct.
I keep three tiers of memory and treat each one differently. Session memory lives only inside a single plan execution and is discarded at the end. It needs no governance beyond the session log. Tenant memory is shared across sessions for one organisation — the firm's own documents, prior decisions, cached extractions. It is governed by the tenant's access controls and is provenance-tagged: every fact in tenant memory carries a pointer to the source document, the worker that extracted it, and the timestamp. If the source is deleted or revised, the derived fact is invalidated. Model memory — anything baked into weights — is treated as immutable and is the version-pinned artefact you sign off on at deployment.
The rule I enforce is that no worker may use a fact from tenant memory without surfacing the provenance pointer in its output. If the planner sees a worker citing a fact with no provenance, the plan fails. This is annoying to develop against and it's the single biggest reason the system stays auditable. For the longer treatment of how this fits the operational layer, see the methodology section of the Intelligence Brain.
Tool calls, blast radius, and the two-key rule
Read-only tools are easy. The agent queries something, gets data, moves on. Write tools are where governance earns its keep. Any tool that mutates a record, sends a communication, files a return, or moves money is a write tool, and write tools need a different regime entirely.
For write tools I apply a two-key rule borrowed from operations: the worker proposes the mutation, the auditor verifies it against the plan, and a separate human or service identity commits it. The worker never holds the credentials to execute the mutation directly. This is implemented at the network layer — the worker's service account literally cannot reach the write endpoint. It can only post a signed mutation request to a queue, which the commit service drains after auditor approval.
The blast radius question — how much damage can a single misbehaving agent cause before something stops it — is the one I ask first when reviewing any new agent design. If the answer is "it could send an email to a client", the controls are different than if the answer is "it could amend a tax filing". The tooling has to make the small-blast-radius path the easy path, otherwise developers route around it.
Logging that stands up to a regulator
Every Irish regulator I've dealt with — and the EU bodies behind them — eventually asks the same question: show me, for this specific decision, what the system saw, what it did, and who approved it. If your answer involves grepping logs across three services and reconstructing a session from timestamps, you have already lost.
The log format I use is one record per plan, with every worker invocation and every tool call as a child record under it. Each record includes the full input, the full output, the model and prompt version, the worker code version, the tenant identity, the user identity, and any human approval signatures. Logs are written append-only to storage that the running services cannot delete from. Retention is set per tenant according to their sectoral rules — accountants and solicitors in Ireland have specific minima, and the system enforces them rather than relying on policy documents.
One detail that matters more than it should: log the plan that was rejected as well as the plan that ran. When a regulator asks why the system did X, half the time the real answer is "because it considered Y and Z and chose X for these reasons". If you only log the chosen path, you can't answer that question, and the absence is itself a finding.
Running this on-premise in Ireland
Most of what I've described above can in principle run in any cloud. In practice, for Irish regulated firms, on-premise or sovereign deployment is what makes the governance story credible. If the model weights, the tenant memory, and the audit logs all live inside the firm's own perimeter, the data protection analysis collapses to something a DPO can actually sign. If any of those three sit in a third-party cloud, the analysis gets longer and the answers get softer.
This is the bet behind the Intelligence Brain: that Irish AI governance for regulated work is easier when the system runs where the data already lives, rather than shuttling the data out to where the model lives. Multi-agent governance amplifies that bet, because every agent hop is another data movement to account for.
Where to start this week
If you're running multi-agent prototypes and the governance question is starting to be asked, do one thing this week: take your current agent flow and write down, on paper, the planner, the workers, and the auditor. Most teams discover they have workers calling workers in ways nobody designed, and no auditor at all. You don't need to rewrite anything yet — just draw the shape you have and the shape you want. The gap between the two is your governance backlog, and it's almost always smaller than it looks once you can see it.