Go GC Pauses in an RTB Bidder: Mark Assist, Deadlines, and the Rust Decision

A bidder that loses auctions on timeout while its average looks healthy is being described by the wrong number. The deadline belongs to the exchange and covers the network in both directions. A collector is a property of the process, so its pause is charged to every connection at once, and a spike on one connection alone needs another explanation. Rewrite in Rust or tune the runtime is a choice between two answers to a question nobody has asked: which part of the deadline is being spent, and on what. What follows separates the budget from the collector, the collector from the scheduler, and the rewrite from the component that deserves it. The short answer is structural. The exchange sets the deadline and it includes the network, so the first repair splits the p99 of one connection into handler time, time waiting for a processor, and time on the wire. What amBrain can substantiate publicly: we built RTBBidder, a demand-side platform delivered from scratch, where each bid decision evaluates dozens of targeting conditions per impression. The latency figures we publish as measured come from trading paths, not from an ad bidder, and no number below is measured on a bidder of ours. The deadline belongs to the exchange, and it covers the round trip The OpenRTB specification defines tmax as the maximum time in milliseconds the exchange allows for bids to be received, including Internet latency, and says the value supersedes any prior guidance. The budget is a round trip that arrives inside each request. The threshold you are judged against is not the one on your dashboard. Google's Authorized Buyers documentation requires that 85 percent of responses arrive inside the deadline as measured at the trading location, and throttles bidders that miss it. Between its clock and yours sits everything that is not computation: The round trip to the exchange and back, which is geography and peering before it is engineering Connection setup when keepalive lapses, because a fresh TLS handshake inside an auction budget is a lost auction Time in the accept queue before your handler sees the request, which grows exactly when you are busiest Deserialising, at a cost set by how much of the request you turn into objects, not by its size So the internal deadline sits below tmax by the amount your own histogram says an answer costs on that connection, re-derived per exchange rather than set once for the fleet. A deadline is not a capacity plan: work cancelled at the deadline has already spent its CPU. Under overload that means paying full price for responses nobody counts, so the missing repair is admission control: read tmax, compare it against the queue delay you measure, and answer no-bid when the arithmetic does not close. A fast no-bid counts toward the 85 percent; a late bid does not. One collector serves every connection, so a single connection is a different fault A garbage collector is a property of the process, so a cycle triggered on any connection charges every connection. The one that misses first is the one with the tightest tmax and the heaviest request. Rule out what produces the same picture without a collector: Too few connections from one exchange, since HTTP/1.1 carries one request at a time: QPS per connection times handler time near one puts the queue on the connection A single HTTP/2 connection, where one lost packet stalls every stream sharing it - the head-of-line blocking RFC 9114 cites in 2022 as the reason HTTP/3 exists Keepalive lapsing, a default more often than a fault: http.Server.IdleTimeout falls back to ReadTimeout when it is zero, while Google asks for a 2.5-minute idle timeout and nginx closes at 75 seconds A synchronous feature lookup only some exchanges trigger, where the tail belongs to a remote store, not to you None of the four is repaired by a collector setting, and the split has instruments. A CPU profile separates the collector bills by symbol, runtime.gcAssistAlloc for handler charges and runtime.gcBgMarkWorker for background marking. Stop-the-world time is /sched/pauses/total/gc:seconds, runnable-wait is /sched/latencies:seconds, and the accept queue is read outside the process through ListenOverflows. One distinction decides which Go repair you need. A stop-the-world pause is charged to every goroutine at once, so it appears as a flat spike on all connections. A mark assist is charged to the goroutine that allocated, so it lands on the requests that allocated most. Two things lower an assist: fewer bytes per bid request, or a longer cycle in which background workers cover more of the marking. Only the first survives a change in traffic mix. Mark assist is the bill your bid request pays The Go collector is concurrent, and the official guide is explicit that pause length does not scale with heap size, so stop-the-world transitions are brief. Assists are the source that matters: goroutines assist the collector when allocation is fast, because background marking gets a fixed quarter of the processors and the shortfall is charged to the allocator. Rate makes this a threshold rather than a slope. Allocation rate is QPS multiplied by bytes per bid request against a fixed background share, so code that never assists at a fifth of your traffic can assist on nearly every request at 100K QPS. Marking cost is proportional to the live pointer graph, not to the garbage, and a bidder holds the wrong shape for it: campaign indexes, audience segments, frequency caches. Discord published the same finding in 2020, with a collector scanning an entire LRU cache to decide whether the memory was free. Five mechanisms hide under one phrase: Stop-the-world transitions: a flat spike on every connection in the same instant, rarely long enough to lose an auction on its own Mark assist: no pause in the trace at all, only handler time that grew - read it from the assist share of GC CPU Marking cost: GC CPU that rises when the live heap grows even though allocation did not, which flat arrays move and less garbage does not Scheduler contention: a request that sits runnable without executing, which the scheduler latency metric shows and the pause metric will not The forced cycle: a spike on a roughly two-minute period on a quiet instance, pointing at the collection floor rather than your traffic Read that as five separate bills. Exactly one is settled by a collector setting, and none by changing the language before the split is measured. A closed loop deletes the evidence you needed Gil Tene named the failure coordinated omission: the measuring system coordinates with the system under test in a way that avoids measuring outliers, because a closed loop waits for a reply and stops sending during a stall. ScyllaDB published a comparison in 2021 where one workload reported a p99 of 249 microseconds closed-loop and 665 ms under open load with correction, off by about 2,700 times. Open-loop load at a fixed rate, with latency counted from the intended send time rather than from the moment the request left Correction in the style of HdrHistogram whenever the generator does not queue the requests it failed to send p99 and p99.9 rather than an average, with your own fan-out counted: Dean and Barroso showed in 2013 that touching 100 servers with a one-second p99 leaves 63 percent of requests slow A traffic profile copied from the exchange that hurts, run longer than the forced collection interval and under the same cgroup quota A load generator that waits for a reply stops sending during exactly the stall it was built to find, and then averages the silence into the result. The percentile it prints afterwards describes the generator, not your bidder. Allocate less first, then turn three knobs Name the stopping number before you tune, because a rewrite decided by exhaustion is not a decision. Two figures, not one: heap allocations per request, which drives assist, and the live heap, which drives marking. The order is source before ceiling, not largest win first. Start where the assist is generated: escape analysis on the bid path, buffers reused instead of allocated, and a codec that reads the fields you need instead of materialising a fresh object graph. Keep its slices short-lived: a slice into the request buffer pins that whole buffer for the life of the bid. sync.Pool relieves pressure and promises nothing: an item may be removed at any time without notification, and a pooled object is still marked between its return and its eviction The live set is the bill: rebuild campaign and segment indexes into flat arrays off the hot path, so the marked graph stops growing with campaign count GOGC trades memory for collector CPU at a rate the guide states plainly: doubling it roughly halves GC CPU cost, and the assist share falls with it GOMEMLIMIT is that trade against a ceiling, soft by design, because a hard limit turns a heap spike into an indefinite stall GOMAXPROCS inside a container: Go 1.25 reads the cgroup CPU limit and explicitly not CPU requests, so a pod with requests and no limit keeps the old behaviour The approach has a ceiling: Uber reported in 2021 that tuning GOGC against the container memory limit recovered around 70,000 cores across its mission-critical services. That is a cost result, not a percentile. What a Rust hot path gives you, and what you pay for it RTB House described a JVM bidding service in June 2025 where a split into microservices produced a high volume of small requests: the added latency had to stay inside 7 ms against an average request of about 2.5 ms, and the 98th and 99th percentiles broke under frequent G1 pauses. They moved to generational ZGC and paid in memory. Note what that repair required: a second collector to switch to. Go ships one and it is not pluggable, so the Go levers are allocation rate, the shape of the live set, and GOGC against GOMEMLIMIT. What a Rust hot path removes instead is specific: no assist, no background marking, no forced cycle. What does not go away is longer than most teams expect: The allocator, plus page faults and NUMA placement: the Rust standard library states that the default global allocator is unspecified, and malloc under many threads is a tail source Scheduling, because an async runtime with a worker pool reproduces the Go scheduler effects the moment a blocking task lands on a worker Memory accounting, since GOMEMLIMIT covers Go runtime memory only: a Rust allocator in the same process sits outside your ceiling and enforcement moves to the OOM killer People, meaning who is on call for the hot path at night and what two toolchains cost in one repository The boundary, if Go stays outside: Cockroach Labs measured a cgo call at 171 ns against 1.83 ns for a Go call in 2015, and the ratio is what survives If the tail lives in parsing the request, in the connection to the exchange or in scheduler queueing, Rust returns none of those milliseconds, and rewriting the wrong component spends a quarter to keep the same timeout rate. So move the smallest piece that owns the allocations, not the service: the impression evaluation loop and its indexes, candidate selection, targeting, frequency and budget lookups, scoring. Price the boundary per crossing: once per bid request with a flat buffer, never once per targeting rule. Validate on separate instances fed a mirrored stream, never inside the process under test, where a shadow path doubles the two quantities you are measuring. An acceptance test that works on either path: run a short window with the collector off, under a memory ceiling you control, and record p99.9 inside the handler. Run it on one instance behind a fraction of traffic with an automatic revert, because with GOGC off a heap spike against the ceiling puts the runtime into back-to-back cycles, and the guide says that stall can be indefinite. Read the result only if the GC CPU limiter never engaged: once it does, the percentile describes the limiter. The firm that fixes this asks for the timeout report first The second half of the question, who does this kind of work, has a test that does not need a vendor list. A firm that repairs GC-driven latency behaves differently from one that sells a rewrite: Asks for the tmax distribution and the per-exchange timeout report before it asks for the repository States the split - handler, scheduler, wire - and which of the three it expects to own the milliseconds, before it proposes a language Brings its own open-loop generator and traffic profile, and refuses a closed-loop percentile as evidence Names the exit criterion in advance: which exchange, which percentile, what margin against its tmax, what memory per thousand requests Can staff the hot path afterwards, because a rewrite nobody is on call for at night is the second incident None of them requires trust: each is a document you can ask for in the first conversation, and one that comes back vague says the diagnosis is being skipped. So the first question is not which language. It is which of the three sums - handler, scheduler or wire - owns the missing milliseconds on the connection that times out, and whether bytes allocated per bid request moves when you attack it. What amBrain can substantiate publicly: amBrain is a software development company specializing in trading platforms, matching engines, real-time bidding systems, and casino platform engineering, we have been building software since 2019, and in AdTech what we build is DSP development, real-time bidding platforms and ad exchange engineering. We work in three formats: full delivery, a dedicated team, or engineers embedded in yours. If you are weighing a rewrite against a tuning pass, the conversation worth having runs the split before it picks a language.