Idempotency keys: the bug that double-invoiced 73 clients

The Friday afternoon when the phone wouldn't stop ringing

The first call came in at 14:47 on a Friday in May. The operations lead at a 28-person recruitment agency in Almere. Voice flat, opening line: "I just got an email from one of our biggest clients. They received two identical invoices for the same placement. Different invoice numbers. Same amount. Same week. Are we hacked?"

No, not hacked. We told her we would call back in ten minutes. We did not. We called back in two hours, because by then a second client had emailed, then a third, then a fourth, and we had stopped trusting any answer we had not verified in the database.

By 17:30 the count was eleven affected clients. By Monday morning, after a full weekend of forensics across Bullhorn, Inngest, Stripe and the agency's own dashboard, the count was 73. These were invoices we had sent. We had built the automation that sent them. And the agent had been silently double-invoicing every single placement for a full week.

How a placement becomes an invoice

The agency uses Bullhorn as their ATS. When a recruiter places a candidate at a client, a Placement record gets created. It holds the start date, the bill rate, the markup, the contract terms, and the link back to the candidate and the client account.

The automation we built listens for new placements through Bullhorn webhooks, validates the data, drafts an invoice in Stripe, and queues it for human approval. The agency's finance lead reviews the queue every Wednesday afternoon and approves the batch. The agent does everything else, including the math on partial weeks, holiday surcharges, and the per-client markup table.

Here is the Bullhorn webhook subscription we had registered. It is what every Bullhorn integration tutorial on the internet looks like, which is part of the problem.

curl -X PUT https://rest.bullhornstaffing.com/rest-services/{corp}/event/subscription/placementChanges \
  -H "BhRestToken: $TOKEN" \
  -d '{
    "type": "entity",
    "names": ["Placement"],
    "eventTypes": ["INSERTED", "UPDATED"]
  }'

The extra UPDATED was deliberate. We wanted to know when a recruiter corrected the bill rate or shifted the start date after the fact, because invoicing logic depends on it. Useful in theory.

What we missed: Bullhorn fires INSERTED the moment the Placement is saved, and then very often fires UPDATED less than a second later, because the platform itself writes derived fields on save. Commission calculations. Rate cap normalization. Status transitions. For every real-world placement, our subscription emitted two events in rapid succession, usually within the same second, sometimes within the same hundred milliseconds.

The Inngest function that did not know it was running twice

Our invoicing logic runs on Inngest. We use it for everything where a step needs to be durable, retryable, and observable. It is, in our opinion, the cleanest workflow engine for AI agent and webhook fan-out work you can put in front of a small team.

Here is the function, lightly anonymized, as it was deployed.

export const draftInvoice = inngest.createFunction(
  { id: "draft-invoice-from-placement" },
  { event: "bullhorn/placement.event" },
  async ({ event, step }) => {
    const placement = await step.run("fetch-placement", async () => {
      return bullhorn.placements.get(event.data.placementId);
    });

    const draft = await step.run("create-stripe-draft", async () => {
      return stripe.invoices.create({
        customer: placement.clientStripeId,
        collection_method: "send_invoice",
        days_until_due: 30,
        metadata: { placementId: placement.id },
      });
    });

    await step.run("queue-for-approval", async () => {
      return db.invoices.insert({
        stripeInvoiceId: draft.id,
        placementId: placement.id,
        status: "pending_approval",
      });
    });
  }
);

Spot the bug. It is what is missing, not what is there.

There is no idempotency field on the function. There is no deduplication on the placement ID. Every event that lands in Inngest spawns a fresh run. INSERTED arrives, run starts, Stripe draft created. UPDATED arrives one second later, run starts again, second Stripe draft created. Different Stripe invoice IDs. Same placement metadata. No warning anywhere in the stack.

The finance lead approves both on Wednesday because her dashboard shows two distinct drafts, both with valid line items, both with sensible totals, and she trusts the agent to have already validated them. The client receives two identical PDFs the next morning.

Why we did not catch it in testing

This is the part that hurts. We tested the integration end-to-end against the Bullhorn sandbox with placements we created by hand through the UI. Those placements only fire INSERTED. The UPDATED-on-save pattern only kicks in when the production data validation rules run against real client and candidate records, and we did not have realistic ones in the sandbox.

Our Inngest dashboard had been showing all the runs the entire time. Two runs per placement. We had built the team's default dashboard view grouped by date instead of by placement ID, so the duplication never stood out. Forty placements in a week looked like eighty runs, which is exactly what you would expect for a workflow with one retry pass built in.

The signal was there. We were not looking at it. That is the honest sentence.

The diagnostic on Friday night

Once we had a fourth client report, we stopped guessing and ran the join. One query against the placements table and the invoices table:

SELECT placement_id, COUNT(*) AS invoice_count, ARRAY_AGG(stripe_invoice_id)
FROM invoices
WHERE created_at >= NOW() - INTERVAL '7 days'
GROUP BY placement_id
HAVING COUNT(*) > 1;

73 rows. Every one of them had exactly two Stripe invoice IDs. Created within seconds of each other. Identical totals. The pattern was uniform enough that within five minutes of seeing the result we knew it was not a race condition or an LLM hallucination or a Stripe-side ghost. It was the webhook source firing twice and our consumer treating both events as new work.

The fix in one line of config

Inngest supports function-level idempotency keys natively. You give it an expression that evaluates against the incoming event, and it deduplicates anything matching that key within a configurable window. Their documentation spells it out clearly. We just had not read it carefully enough when we shipped.

The fix:

export const draftInvoice = inngest.createFunction(
  {
    id: "draft-invoice-from-placement",
    idempotency: "event.data.placementId",
  },
  { event: "bullhorn/placement.event" },
  async ({ event, step }) => {
    // body unchanged
  }
);

That single line tells Inngest: if you receive another event with the same placementId within the next 24 hours, do not start a new run. Drop it. Log it. Move on.

We also added a second guarantee at the Stripe layer, because two layers of idempotency are cheaper than one weekend of refunds. Stripe's Idempotency-Key header is one of the most battle-tested implementations in the payments industry, and it costs nothing to use.

const draft = await step.run("create-stripe-draft", async () => {
  return stripe.invoices.create(
    {
      customer: placement.clientStripeId,
      collection_method: "send_invoice",
      days_until_due: 30,
      metadata: { placementId: placement.id },
    },
    { idempotencyKey: `invoice-placement-${placement.id}` }
  );
});

Now even if our Inngest layer somehow lets a duplicate slip through (operator error, an accidental replay during debugging, a future code change that subtly breaks the dedupe), Stripe will return the original invoice instead of creating a new one. Two layers. Two guarantees. The cost is twelve characters of code.

What we sent the agency on Monday morning

Before any of the technical fixes shipped to production, we drafted a one-page note for the agency to send their clients. Straight apology, the duplicate invoice IDs side by side, the credit memos we had already generated in Stripe, the timestamp when the fix went live.

The operations lead asked for one change. She wanted a paragraph explaining, in human terms, what idempotency means and why a simple webhook subscription had blown up. We wrote it, she sent it, and out of 73 affected clients exactly one moved their account elsewhere. The other 72 stayed.

That ratio is not luck. It is the dividend of writing the explanation as if the recipient runs a business, not as if they wrote the code.

The defaults we changed across the studio

After this we rewrote our internal template for every webhook-triggered Inngest function. Four changes, all of them small.

1. Every function gets an explicit idempotency expression. The PR reviewer must justify the chosen key during review, not silently let it ship without one.
2. Every external write step (Stripe, Xero, Exact, Twinfield, Resend, Postmark) gets an idempotency key derived from a stable business identifier. Never a UUID generated inside the step.
3. The default Inngest dashboard view groups by the idempotency expression, so duplicate runs are visible the moment they happen instead of a week later.
4. Every new webhook integration ships with a one-week monitoring rule: alert on any business entity that produces more than one downstream run in a 60-second window.

None of this is novel. The Inngest documentation has covered it for years. Stripe has shipped idempotency as a first-class concept since 2015. The reason we missed it was not technical ignorance, it was the speed at which the first version felt finished. The webhook was firing. The invoices were arriving. The dashboard was green. The contract with the downstream system was the thing nobody had written down.

What you can do in the next ten minutes

Open the dashboard of whatever workflow engine you use. Inngest, Temporal, Trigger.dev, AWS Step Functions, a homegrown queue. Pick one function that writes to an external system. Find the line where it makes that write.

Ask the question. If this function ran twice with the same input, would the external system end up with two of something it should only have one of? An invoice. An email. A Slack message. A payout. A booking.

If the answer is yes, or worse, "I don't know," you have your next pull request.

When we built the invoicing AI agents for the Almere recruitment agency, the thing we ran into was a webhook source that fired the same logical event under two different names. We ended up solving it with a one-line idempotency expression on the Inngest function and a second guarantee at the Stripe API layer. The week we spent issuing credit memos and writing apology letters is the reason that line is now non-negotiable in every template we ship.