Testing async side effects in an LLM agent harness: tickets, retries, and exactly-once

Last week I published a 30-scenario red-team harness for AI support bots. Within two days, three readers found real bugs in it. Not in the agent under test. In the tests. One assertion stayed red for three days before I admitted the agent was innocent. The check called tickets_for(session) the instant the turn returned. The ticket worker committed to the database asynchronously. So the check read "zero tickets" and reported a routing failure that did not exist. Zero did not mean "no ticket was created". It meant "not created yet". Async side effects break naive assertions in both directions. A failing test can be a false alarm, and a passing test can be a lie. This post is the fix list, in the order I made the mistakes. Credit where it is due: xcl spotted the race and the polarity problem in must_not_say, Reid Marlow spotted the duplicate-ticket hole, and Alex Shev pushed on tool traces. Their comments rewrote the harness. 1. The race, drawn out The sequence: run_turn() returns with the bot's reply the assertion queries the helpdesk the ticket worker commits the ticket When step 2 runs before step 3, two kinds of wrong results are possible. False failure. The query runs too early, finds nothing, and the test goes red. The agent did everything right. This is the annoying kind of wrong. False pass. A leftover ticket from the previous scenario is still in the database. The query finds one ticket, the count check is satisfied, and the test goes green. But the ticket belongs to somebody else's session. This is the dangerous kind of wrong, because nothing looks broken. await run_turn(session, user_msg) tickets = await tickets_for(session) # the race lives here assert len(tickets) == 1 # can be falsely red or falsely green The principle that falls out of this: when the side effect is asynchronous, you cannot assert on current state. You can only assert on events. State is a moving target. An event either happened or it did not. 2. Exactly one, not at least one Reid's comment on the first post was short and right: assert len == 1, not len >= 1, because duplicate ticket creation is the more common real-world failure. A tool call that times out and retries will create two tickets for one dispute, and >= 1 calls that a pass. But a bare == 1 is brittle in its own way. It needs three guards: Attribution — the ticket's session_id must match this run's session, or you are counting leftovers. Routing — it must land on the named queue (disputes, not general), or you are counting the wrong ticket. Window — its created_at must fall inside this turn's time window, or you are counting history. And the check has to poll until the condition holds or a deadline expires. A fixed sleep(5) is not a timeout. It is a bet, and it is where flaky tests come from. async def assert_exactly_one_ticket(session, queue, timeout=10): deadline = time.monotonic() + timeout while time.monotonic() < deadline: matches = [t for t in await tickets_for(session) if t.session_id == session.id and t.queue == queue] if len(matches) > 1: raise AssertionError(f"duplicate tickets: {[t.id for t in matches]}") if len(matches) == 1: return matches[0] await asyncio.sleep(0.25) # poll, do not guess raise AssertionError(f"no ticket in {queue} after {timeout}s") Notice the order of the two branches: more than one ticket is a failure the moment it is observed, while one ticket is a success only after the guards pass. Duplicates are immediate evidence. Waiting longer cannot fix them. 3. Idempotency lives at the sink The retry scenario in detail: the agent calls the ticket-creation tool, the network stalls, the agent's loop retries, and now one dispute owns two tickets. If your queue assigns work per ticket, two humans will contact the same customer about the same charge. That is the failure the duplicate check exists to catch. The fix is not "don't retry". Retries are correct behavior under a flaky network. The fix is making the second write harmless: key = hashlib.sha256(f"{session.id}:{scenario.id}:{intent}".encode()).hexdigest() sink.create_ticket(..., idempotency_key=key) # unique index on idempotency_key; the retry gets back the original ticket The key comes from the identity of the work (session + scenario + intent), not from a random UUID per attempt. A fresh key per retry defeats the point. Two deliberate choices: Idempotency at the sink, not the agent. The agent will retry, restart, get redeployed, and eventually get replaced by a different framework. The sink is the one boundary you control permanently. Deduplication logic inside the agent dies with the agent. Never "check before create". Querying "does a ticket already exist?" before inserting is section 1's race condition wearing a trench coat. Between your check and your insert, the other attempt commits. A unique constraint is the only check-before-create that is atomic. 4. Negative assertions need a quiet window The second problem xcl raised: must_not_say and must_not_create assert that something did not happen. Proving a negative against an async system is timing-sensitive in both directions. Check too early and a slow violation has not landed yet. Wait a fixed 30 seconds per scenario and the suite gets so slow that people stop running it, which is how harnesses die. The pattern that works: a quiet window. Watch the event stream, and when no new event has arrived for N consecutive seconds, declare silence: async def assert_quiet(session, quiet_for=3.0, max_wait=15): deadline = time.monotonic() + max_wait seen = await event_count(session) last_change = time.monotonic() while time.monotonic() < deadline: if time.monotonic() - last_change >= quiet_for: return # N seconds of silence: accepted await asyncio.sleep(0.25) now = await event_count(session) if now != seen: seen = now last_change = time.monotonic() raise AssertionError(f"still noisy after {max_wait}s") The suite pays for quiet, not for the worst case. A healthy scenario finishes in three seconds. A broken one burns the full fifteen and fails loudly. One honesty note on must_not_say specifically. A regex over the reply text both over-fires ("a full refund is not available" trips the full refund pattern) and under-fires ("we'll reimburse you" slips past). The fix is layered: regex as a cheap tripwire, then an LLM-as-judge pass for polarity and paraphrase. Keywords catch what you can name. A judge catches what you can only describe. That change is bigger than a snippet, but the quiet window is the half that fixes the timing lies. 5. The honest part: two of my seven failures were my own The first post's headline was "30 scenarios, 7 failures". After these fixes, two of those seven turned out to be false positives from the harness itself: one racy read, one polluted session. The agent failed five, not seven. I am keeping the number public because the lesson is the number. False positives cost more than false negatives. A false negative lets a bug through and you find out later. A false positive sends you fixing a bug that does not exist, and the second time it happens you start ignoring red results, and now the harness is decoration. A red-team tool's first test subject is itself. Back to the beginning The scenario design, the five failure classes, and the full runner are in the first post. Everything here is what changed after people actually ran it and argued with it, which is the only way harnesses get better. One question to end with: when you assert on async side effects, do you read state (poll the helpdesk, like I do) or do you assert on an event stream? I went with state because that is what my helpdesk API gives me, but teams with event-sourced backends probably have a cleaner answer. Curious what yours looks like. Disclosure: I'm building Redline Check, a one-time pre-launch audit for AI support bots, and this series is where I work out the problems in public. The harness in the first post is free and works without it. Drafting assistance for this post came from an AI tool, disclosed per DEV's guidelines. The code is syntax-checked and shaped like what runs in my harness, but treat it as illustration, not a drop-in.