Why-did-the-agent-do-that: a one-table explainability button

It is Tuesday afternoon. Your support lead pastes a screenshot into Slack: the chat agent told a customer their order was non-refundable. It was refundable. The customer is annoyed, the support lead is annoyed, and you, the person who shipped the agent, have no idea why it said what it said. You open the database. You scroll the messages table. There is the assistant turn, sitting there in plain text, with no tool calls, no system prompt snapshot, no retrieved context, nothing. Just the wrong answer.

This is the worst part of running an agent in production. Not the hallucinations themselves, but the inability to look one in the face and ask why. So here is the small, boring fix we keep installing on client agents. One log table, three lines of UI, and a panel that pops out of every assistant message saying exactly what happened. It takes an afternoon. It is the single highest-leverage thing you can do to a chat agent that already works.

The shape of the problem

A modern chat agent is not one call. It is a small pipeline: the user message arrives, you retrieve some context (RAG, CRM lookup, order history), you build a system prompt, you call the model with a set of tool definitions, the model decides to call a tool or reply, you run the tool, you feed the result back, eventually you get a final assistant message. Five to ten steps, sometimes more.

By default, you store the user message and the assistant message. That is what your chat UI renders. Everything in between, the part that actually decides what the assistant says, evaporates the moment the request returns. When something goes wrong, you have the question and the answer and nothing connecting them.

The fix is not observability platforms. Those are useful, but they are for you, in a separate tab, with a login. The fix is to put the trace inside the product itself, attached to the message that came out of it, gated behind a small button only your team can see. Then the next time your support lead pastes a bad answer into Slack, the first reply is: click the why button.

One table, eight columns

Add one table next to your existing messages table. Call it agent_traces. It stores one row per assistant turn, keyed to the message it produced.

create table agent_traces (
  id            uuid primary key default gen_random_uuid(),
  message_id    uuid not null references messages(id) on delete cascade,
  conversation_id uuid not null,
  created_at    timestamptz not null default now(),
  model         text not null,
  system_prompt text not null,
  retrieved     jsonb not null default '[]'::jsonb,
  tool_calls    jsonb not null default '[]'::jsonb,
  raw_response  jsonb not null,
  latency_ms    integer not null,
  input_tokens  integer,
  output_tokens integer
);

create index on agent_traces(message_id);
create index on agent_traces(conversation_id, created_at desc);

That is the whole schema. Eight content columns. retrieved is the list of chunks your RAG layer pulled in, with their source ids and similarity scores. tool_calls is the ordered list of function calls the model made, each with arguments and result. raw_response is the full provider response object, untouched, so you can replay it later. system_prompt is a snapshot of the prompt as it actually went to the model, including any per-tenant variables, because the version in your codebase today is not the version that ran last Tuesday.

Where the writes go

You already have a function that calls the model. Wrap it. Before the call, start a trace object. After the call, write the row in the same transaction that writes the assistant message. If the model call fails, write the row anyway with the error in raw_response; a failed turn is exactly the kind of thing you want explainable.

async function runAgentTurn(conv: Conversation, userMsg: Message) {
  const t0 = Date.now();
  const retrieved = await rag.search(userMsg.content, { conv });
  const systemPrompt = buildSystemPrompt(conv, retrieved);

  const response = await model.chat({
    system: systemPrompt,
    messages: conv.history,
    tools: TOOLS,
  });

  const toolCalls = await runTools(response.tool_calls, conv);
  const assistantText = response.text;

  await db.transaction(async (tx) => {
    const msg = await tx.messages.insert({
      conversation_id: conv.id,
      role: 'assistant',
      content: assistantText,
    });
    await tx.agent_traces.insert({
      message_id: msg.id,
      conversation_id: conv.id,
      model: response.model,
      system_prompt: systemPrompt,
      retrieved,
      tool_calls: toolCalls,
      raw_response: response,
      latency_ms: Date.now() - t0,
      input_tokens: response.usage?.input_tokens,
      output_tokens: response.usage?.output_tokens,
    });
  });
}

If you are calling the Anthropic SDK or OpenAI directly, the raw_response object is already a JSON-serialisable thing. Just stringify it and store it. Storage is cheap. A typical agent turn writes 4 to 20 kilobytes of trace. At a thousand turns a day that is twenty megabytes a month. You will not feel it. If you do, partition the table by month and prune after ninety days.

The three lines of UI

The UI is genuinely small. Next to every assistant bubble, render a tiny button. Only show it to authenticated team members, not customers. On click, fetch the trace and render it in a side panel.

{isStaff && (
  <button onClick={() => openTrace(msg.id)} className="why-btn">why</button>
)}

That is the three lines. The panel itself is also small, because the table is small. Four sections: the system prompt as it ran, the retrieved chunks with their scores, the tool calls in order with inputs and outputs, and the raw model response collapsed by default. No charts. No timeline. Plain monospace. The point is not to be pretty, the point is to answer the question why did it say that in under thirty seconds.

What you will actually find

The first week after you ship this, you will discover three things, in this order. First, your retrieval is worse than you thought. The chunk the agent needed to answer correctly was sitting at rank seven, below six near-duplicates. Second, your system prompt has a clause from four months ago that no longer makes sense, and the agent has been quietly following it. Third, one of your tools returns an error message that the model interprets as a polite refusal. None of these are exotic failure modes. They are the normal furniture of a production agent, and you cannot see any of them without the trace.

There is a wider point here. Anthropic's engineering team published a piece on building effective agents, and one of the recurring threads is that workable agent systems depend on keeping the model's behaviour legible to the humans operating it. A why button is the cheapest possible version of that. You are not aligning a model. You are giving yourself the ability to notice when it has drifted.

What not to put in the trace

Two things to keep out. PII you would not want sitting in a JSON blob for ninety days; if your retrieved chunks contain customer records, hash or redact the obvious fields before they hit the table. And secrets that leak in through tool results, especially API tokens echoed back in error messages. Both are easy to scrub at write time and very annoying to scrub after the fact. If you are in the EU, treat the trace table like any other personal-data store and put it in your Article 30 record of processing.

When the afternoon is over

You will have a table, a wrapper around your model call, and a small button. Your support lead will stop sending you screenshots and start sending you trace links. Your prompt changes will become evidence-based instead of vibe-based. And the next time someone asks you why the agent said what it said, you will click a button and read the answer.

When we built the inbox-triage agent for a Rotterdam logistics client, the thing we ran into was that a single ambiguous clause in the system prompt was causing the agent to defer to a human on every shipment status question. We only spotted it because the trace showed the same retrieved chunk arriving every time and being ignored. Deleting one sentence cut the human-handoff rate by a third. That kind of fix is what explainable agents are for: small edits, justified by evidence, made in minutes.

If your agent is live and you cannot answer why for its last ten replies, the smallest useful thing you can do today is create the agent_traces table and start writing to it. The UI can wait until tomorrow. The data cannot.