Bol.com fair-use ban: how 240 fetches torched our prijsmonitor

Tuesday, 09:14. The first Slack ping came from monitor-bot: prijsmonitor down — 429s on every call. The second came from the client lead at a 19-person furniture retailer in Almere, who could watch his own dashboard go grey from a Hema parking lot. By 09:18 we had the Bol.com partner-API console open and the answer was sitting there in plain language: Fair-use threshold exceeded. Account temporarily suspended.

This is what we changed before lunch, and what we now run in front of every concurrent scrape that leaves our partner-API tunnel.

What the prijsmonitor does on a normal day

The retailer sells assembled meubels — sofas, dining tables, kasten — and competes with about a dozen Bol.com sellers on a catalogue of roughly 1,800 SKUs. The prijsmonitor reads the Bol.com offer page for each SKU, pulls the lowest competitor price, and writes back into the retailer's Magento store. If they are no longer in the top three results, the agent flags the SKU for manual review. If they have slipped to position seven and the next-lowest seller is more than 8% under, the agent re-prices automatically inside a band the owner controls.

The whole loop runs every 30 minutes during business hours. 1,800 SKUs every 30 minutes is 3,600 page reads an hour. That had been fine for fourteen months.

The 1 April change we missed

Bol.com quietly halved the per-account fair-use ceiling on its partner platform on 1 April 2026 — from 480 requests per minute per account to 240. The change landed in a release-notes paragraph two layers deep that we did not parse fast enough. Our scheduler kept assuming the old ceiling and kept fanning out 240 concurrent fetches at the start of every cycle.

On 1 April that put us exactly at the new line. On 2 April it put us over, because retries from a single slow fetch piled on top of the next cycle's burst. Bol's enforcement is not instant — it tolerated us for several weeks, then on Tuesday 21 April the fair-use evaluator caught up, drew a line under our rolling average, and suspended the account. By the time we saw it, we had been over the line for nearly three weeks of business mornings.

Why the scheduler kept hammering

The bug, in retrospect, was not the concurrency number. It was that the scheduler had no feedback loop from actual response status into the next fan-out. Pseudocode of the bad version:

// before: fixed-width fan-out, no backpressure
const skus = await loadSkus()                       // ~1,800
const concurrency = 240
await pMap(skus, fetchOffer, { concurrency })       // pMap from 'p-map'
await writeBack(results)

240 was chosen because it fit comfortably under the old 480/min ceiling, and because Node could keep that many sockets open without choking. There was no awareness of HTTP status. A burst of 429s was treated the same as a burst of 200s — the scheduler ploughed on and queued the next cycle on time. The retry layer wrapped each request individually, which made things worse: a 429 on SKU 412 produced two more fetches before the cycle even noticed.

The right shape was obvious in hindsight: a token bucket sized to the current ceiling, plus a circuit breaker that opens on the first run of 429s and stays open long enough for the rolling window to forgive us. The harder question was where the egress identity should live.

Cloudflare ephemeral accounts as the egress

While we were rebuilding the backpressure layer, the other half of the fix went in at the network edge. We had been reading a recent Cloudflare piece on temporary accounts for AI agents — the one that has been on and off the Hacker News front page for months. The gist is that an agent can spin up a sandboxed, short-lived Cloudflare account, run its workload, and tear it down. For our prijsmonitor that maps to a simple pattern: every scrape cycle gets its own ephemeral egress identity, scoped to a Worker that proxies the Bol partner-API call, and that identity is torn down as soon as the cycle commits.

That gives us two things. One: we no longer present as the same fingerprint cycle after cycle, which reduces the chance that any fair-use evaluator treats our 30-minute pulses as a single sustained scraper. Two: when an ephemeral identity does pick up a 429, we throw the whole identity away with it. We do not carry the warning forward into the next cycle.

The Worker itself is short. It takes a signed request from our scheduler, forwards it to api.bol.com on the partner endpoint, attaches the rotating account's credentials, and returns the response upstream. Nothing about the upstream client knows or cares that the egress identity rotated under it.

// worker/egress.ts — runs per cycle on an ephemeral Cloudflare account
export default {
  async fetch(req: Request, env: Env): Promise<Response> {
    const signed = await verify(req, env.SCHEDULER_PUBKEY)
    if (!signed) return new Response('forbidden', { status: 403 })

    const upstream = new Request('https://api.bol.com' + signed.path, {
      method: signed.method,
      headers: {
        ...signed.headers,
        authorization: `Bearer ${env.BOL_PARTNER_TOKEN}`,
      },
      body: signed.body,
    })

    const res = await fetch(upstream)
    // surface 429s verbatim — the gate upstream needs to see them
    return new Response(res.body, { status: res.status, headers: res.headers })
  },
}

We rotate at cycle granularity, not per request. Per-request rotation looked synthetic enough in our staging traces to attract its own attention, and the ops cost was real. Cycle-level rotation gives us a fresh identity 18 times a day during business hours and that has been enough.

The 90-second backoff gate

The other piece that now sits in front of every concurrent scrape is what we call the backoff gate. It is not exotic — it is a small Redis-backed lease — but it is the thing that turned out to matter most.

Before any worker is allowed to fetch, it asks the gate: am I allowed out? The gate keeps two pieces of state per upstream: the count of 429s seen in the last rolling minute, and a do-not-enter-until timestamp. On the first 429, the gate sets the do-not-enter timestamp to now + 90s and refuses every subsequent worker until it passes. Ninety seconds is chosen against Bol's published fair-use window — long enough to let the rolling counter drop, short enough that the half-hour cycle still completes.

// after: token bucket + backoff gate + ephemeral egress
import { fetchViaEphemeralWorker } from './egress'
import { gate } from './backoff-gate'

async function fetchOffer(sku: string) {
  await gate.acquire('bol.partner')                  // blocks if gate is closed
  try {
    const res = await fetchViaEphemeralWorker(sku)   // rotates egress identity per cycle
    if (res.status === 429) {
      await gate.trip('bol.partner', { holdMs: 90_000 })
      throw new RateLimitError(sku)
    }
    return res.body
  } finally {
    gate.release('bol.partner')
  }
}

const skus = await loadSkus()
const concurrency = 120                              // half the new ceiling, on purpose
const results = await pMap(skus, fetchOffer, {
  concurrency,
  stopOnError: false,
})

Three things to call out in that snippet. The concurrency cap is now 120, half of the published ceiling, because the published ceiling is a hard line and we want headroom for retries. The gate.trip call is global per upstream, not per worker — one 429 anywhere closes the gate for everyone. And SKUs that throw inside the gate go onto a small follow-up queue that drains at the end of the cycle on a fresh identity, rather than being retried inline.

The release-notes agent that pings us on Fridays

The whole incident only happened because a numeric change in a release-notes paragraph slid past us for three weeks. We did not want that to repeat across the other rate-limited APIs we touch — Mollie, PostNL, two marketplaces — so the smallest thing we built on Wednesday afternoon was a release-notes agent that does one job. It scrapes the partner release-notes feed every Friday morning, diffs it against last week, runs the diff through a model that has been told flag any change to a numeric value in any sentence that contains the words rate, limit, quota, fair use, concurrent or throttle, and pings the on-call channel if anything trips.

It is not glamorous. It catches roughly one real change per quarter. That is one alert per quarter we would otherwise miss.

What it cost to ship

The bones of the rebuild took eight working hours over Tuesday and Wednesday. The Bol.com account suspension lifted after we filed a written explanation on Tuesday evening — about six hours of downtime in total. The retailer's owner cared more about the explanation than the downtime; he had been a Bol.com partner for nine years and his first question was whether his standing was permanently damaged. It was not.

The ephemeral-account rotation pays for itself at well under a euro per cycle, even at the new partner-API volume. The Redis gate runs on the same Upstash instance the scheduler already used. The only meaningful ongoing cost is paying attention to Bol's release notes, and the Friday agent now does that.

Lessons that survived the post-mortem

Two are worth keeping. First: concurrency limits set against vendor ceilings should leave headroom for the vendor moving the ceiling. Half is a defensible default. We had been running at the ceiling for fourteen months and it felt fine because the ceiling did not move. The day it moved, we had no slack.

Second, and this is the one we keep coming back to in design reviews: your scraper's egress identity is part of its rate-limit budget. We used to think of egress as a fixed property of the workload. With cheap ephemeral accounts it becomes a variable you can spend. Spend it.

When we built the prijsmonitor for this Almere retailer, the thing we ran into was that fair-use enforcement is rolling, not instant, so a config drift is invisible until it is fatal. We ended up solving it with the ephemeral-egress and backoff-gate pattern above, which is now the default shape for every AI agent we ship that touches a third-party rate-limited endpoint.

The smallest thing you could do today: grep your scheduler config for any concurrency number that matches a vendor's published ceiling exactly. If you find one, halve it before standup tomorrow.