Cloudflare temp accounts for scrapers: 19 quirks ranked

It's 23:14 on a Tuesday in May. The scraper has just finished its nightly pass over 4,800 product pages for a prijsmonitor we built for an Eindhoven distributor. Every request returned 200 OK. Every price under €1,200 parsed correctly. Every price above €1,200 came back null.

We had been routing that agent through Cloudflare's temporary accounts for AI agents for six weeks. They are a real improvement on residential proxy pools: cleaner trust, cleaner quota, cleaner audit. They are also a new surface, and the failure modes are not the failure modes you already know.

This is the cheatsheet of nineteen quirks we logged, ranked by how silently each one corrupts your data. Numbers one through five will lose you prices without raising an alert. Numbers seventeen through nineteen will only annoy you.

The setup

Fourteen storefronts. Eight on Magento 2.4 with full-page cache via Varnish plus an ESI layer for the variant pricing block. Two on Shopware. Four on bespoke PHP. Cart and checkout live on separate subdomains for half of them — a multi-domain handover the merchants had inherited from a 2017 redesign and never untangled.

The agent's job: pull current price, stock, and variant-specific bundle pricing once an hour, dedupe, and ship deltas into the client's Snowflake. About 4,800 page-loads per week, plus retries on transient 5xx.

We picked temporary Cloudflare accounts for three reasons. Per-agent identity, so a buggy run can be isolated and revoked. No proxy-pool tax. And the bonus that Cloudflare's bot detection treats traffic from its own temp accounts more leniently than it treats unknown residential IPs. That last point is exactly why this surface is interesting, and also exactly why the failures are quiet.

Why we ranked instead of listed

A flat list of quirks is useless. What you need is which of these will corrupt the rows you feed into a price-history table, and which will only cost you latency or money. The first kind is dangerous because the alert never fires. The second kind shows up in a Grafana panel an hour after it starts.

Tier 1 — silent data loss

These are the five we have lost real money to. If your scraper writes directly to a downstream system, address these before you touch anything else on this list.

1. The €1,200 ESI fragment. Magento renders the high-value variant block via a fenced ESI hole. Cloudflare temp accounts honor Cache-Control: private on the parent, but the fragment is keyed independently and lingers past its declared TTL. Result: the parent page is fresh, the price block inside it is twenty minutes stale. We only caught this because a sales lead asked why our dashboard said €1,149 on an item the merchant had just raised to €1,289.
2. Multi-domain session-cookie drop on handover. A session issued on shop.example.com does not reliably travel to checkout.example.com under a temp account, because __cf_bm is bound to the issuing host rather than the eTLD+1. The handover succeeds, the session id silently rotates, the agent thinks it's logged out, retries, and you double-fire.
3. JS-challenge HTML cached at the page URL. When the temp account hits a soft challenge, the challenge HTML can land in cache under the real page's URL key. The next request returns a 200 OK on the challenge document. Your parser sees no itemprop="price" and writes null. No 4xx, no log line, no Sentry event.
4. Duplicate Set-Cookie collapse. If the upstream sends two cookies with the same name (Magento sends two form_key cookies on cart pages), the worker layer keeps only the last. Half your B2B-pricing variants depend on the dropped one.
5. Sibling-domain Origin stripping. Cross-origin fetch() from a temp-account context to a sibling domain silently strips Origin when the temp account does not have an explicit binding for both hosts. Magento's GraphQL price endpoint then refuses with a 200-OK empty body, which your retry logic will not catch because the status code is fine.

Tier 2 — cookies, headers, fingerprints

6. SameSite downgrade. SameSite=Lax cookies behave like Strict when the temp-account proxy injects its own X-Forwarded-* headers and the upstream reads the request as a top-level navigation.
7. Varnish session re-key on rotation. Magento sits behind Varnish; sessions get a fresh key at every temp-account credential rotation. Your authenticated price view turns guest mid-run, and guest prices drop the loyalty discount band entirely.
8. Per-eTLD+1 jar leakage. Cookie isolation across temp-account contexts is per-registrable-domain only. shop.example.fr and shop.example.nl share a jar, so currency and locale bleed between agents you thought were independent.
9. Secure-flag strip on scheme upgrade. If a temp account upgrades HTTP to HTTPS mid-flight, cookies returned in that hop lose Secure. The next request can downgrade them to a plain channel without a warning.
10. JA3 shift inside one session. The TLS fingerprint shifts between requests under load. Bot fight mode reads that as the same IP suddenly behaving like two clients. You get soft-blocked for an hour, and the symptom is not a 403; it is slow-rolled 200s with the wrong currency selected.
11. Referer rewriting on cross-origin scripts. Magento's legacy JSONP-style price fetches break because the temp account rewrites Referer on cross-origin script tags. The price endpoint returns 200 OK with a JS error string instead of JSON.

Tier 3 — latency, quota, geography

12. Spin-up cost. First request on a fresh temp account costs 400 to 900ms in our measurements. Tolerable at human pace; painful at 4,800 pages a week with retries multiplying the cold-start tax.
13. Parent-scoped quota. Request quota is shared across all temp accounts under one parent. One runaway scraper starves the rest, and the per-temp-account dashboard will tell you you have headroom you do not actually have.
14. Pinned colo. A temp account is pinned to one Cloudflare colo for its lifetime. EU-hosted Magento with a US-routed temp account adds about 120ms per round trip. We forced colo selection by region tag after week two.
15. WebSocket upgrade fragility. The 101 Switching Protocols path through a temp-account proxy fails when the upstream sends a non-empty body with the upgrade. Magento's admin pub-sub does that; the storefront does not, so this one only bites internal tooling.
16. HTTP/2 push stripped. Server-pushed resources are dropped at the proxy layer. Magento's preload hints turn into a full waterfall, which shifts your render-budget math if you depend on Above-the-Fold timings for screenshot diffing.

Tier 4 — merely annoying

17. User-Agent drift. The temp-account UA string changes on no public schedule. If you regex against it for tagging, your tagger breaks quietly.
18. Audit log lag. Logs are delayed 6 to 12 hours. Debugging "what did my agent send at 03:14 last Friday" becomes a faith exercise.
19. Wrong rate-limit header. X-RateLimit-Remaining reports the parent account's pool, not the temp account's. The number you trust is the wrong number.

The assertion that fixed half of this

The single highest-ROI change we made was refusing to write any row whose price field failed a per-template presence check. Status codes became advisory. The parser became authoritative.

def assert_price(html: str, template: dict) -> float:
    """Return parsed price or raise. HTTP 200 is not a guarantee."""
    m = PRICE_RE[template["id"]].search(html)
    if not m:
        raise ScrapeError("price_block_missing", template["id"])
    value = float(m.group(1).replace(".", "").replace(",", "."))
    if not 0 < value < 1_000_000:
        raise ScrapeError("price_implausible", value)
    if template["variant_required"] and "data-variant-sku" not in html:
        raise ScrapeError("variant_block_missing", template["id"])
    return value

The variant-block assertion alone caught quirk one in production within forty minutes of deployment. It would have caught it six weeks earlier.

Wider context worth wiring in

Two adjacent shifts belong on the same plan. First, Anthropic begins requiring ID verification for certain capabilities on 8 July 2026; if your scraper pipes parsed HTML into Claude for variant extraction, your provider-side auth posture and your fetch-side auth posture now move on different clocks, and you want a single dashboard for both. Second, the broader conversation on building reliable agentic AI systems is converging on the lesson this cheatsheet teaches: the failures that matter are the ones that return a successful status code.

What we would do differently from day one

Validate every parsed page against a per-template price-presence assertion before insertion. Pin temp accounts to a region tag from the first deployment. Stamp the resolved colo, the JA3 hash, and the parent-cookie hash on every request log line so you can correlate quirks five to ten without re-running anything. Treat a 200 OK with an empty price field as a hard failure, not a row to retry.

When we built the price-monitor for that Eindhoven distributor, the thing we kept running into was item one on this list — the ESI fragment going stale under a fresh-looking parent. We solved it by attaching a Worker rule that forced a Vary on the variant cookie, plus a periodic AI agent sweep that re-fetched any product above the price ceiling with cache busting before considering the row complete.

Five-minute audit you can run today: grep your scraper's last week of output for rows where price is null but stock is positive. That ratio is your silent-failure floor.