Characterize First, Then Refactor: A Golden-Corpus Harness With a Mutation Gate

Most refactors break behavior that no test ever covered. Adding more hand-written tests is slow and biased toward what you already suspect. A pinned behavior snapshot is faster, and a mutation gate tells you whether that snapshot can actually fail. The rule: pin first, prove the pin bites, then change the smallest thing. If the pin never fails on a deliberate bug, you have documentation, not a safety net. Step 0: Pick one function, not one file Choose a target that is boring and load-bearing. Small input surface, no hidden globals, callable from a script. Write the choice down in the commit message. If the function reads the clock, calls the network, or draws randomness, stop. You need a seam before a snapshot. Pin a wrapper with the bad inputs injected instead. Step 1: Freeze an input corpus Build a JSONL file of real-shaped inputs. One case per line, stable ids, keyword arguments. No customer data, no tokens, no emails. {"id":"c001","kwargs":{"tier":"gold","qty":1,"coupon":null}} {"id":"c002","kwargs":{"tier":"gold","qty":0,"coupon":"SAVE10"}} {"id":"c003","kwargs":{"tier":"free","qty":-1,"coupon":null}} Three sources work well. Existing fixture files, call sites in the repo, and log payloads you have already stripped. Twenty to forty cases covers most small functions. Include the ugly ones: empty string, zero, negative, missing key, duplicate id. The golden file should pin the error paths too, not just the happy path. Step 2: Normalize the output Raw output rarely compares cleanly. Dict ordering drifts, floats repeat, timestamps move. Normalize before hashing. # char_harness.py (part 1) def normalize(value): if isinstance(value, dict): return {str(k): normalize(v) for k, v in sorted(value.items(), key=lambda kv: str(kv[0]))} if isinstance(value, (list, tuple)): return [normalize(v) for v in value] if isinstance(value, (set, frozenset)): return sorted(normalize(v) for v in value) if isinstance(value, float): return round(value, 9) if isinstance(value, bool) or value is None or isinstance(value, (int, str)): return value try: attrs = normalize(vars(value)) except TypeError: attrs = {} return {"__type__": type(value).__name__, "__attrs__": attrs} Normalization is a tradeoff, not a free win. Rounding floats hides small drift. Falling back to type name hides everything inside slot-only objects. Log both decisions in the harness header. Step 3: Pin a hash per case The harness fingerprints each case and stores a sha256. Exceptions are pinned by class name and message. # char_harness.py (part 2) import argparse, hashlib, importlib.util, json from pathlib import Path def load_target(path): spec = importlib.util.spec_from_file_location("target", path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) return mod def fingerprint(mod, fn_name, case): fn = getattr(mod, fn_name) try: payload = {"ok": True, "out": normalize(fn(**case["kwargs"]))} except Exception as exc: payload = {"ok": False, "err": type(exc).__name__, "msg": str(exc)} blob = json.dumps(payload, sort_keys=True, separators=(",", ":")) return hashlib.sha256(blob.encode()).hexdigest(), payload def main(): ap = argparse.ArgumentParser() ap.add_argument("--target", required=True) ap.add_argument("--function", required=True) ap.add_argument("--corpus", default="corpus.jsonl") ap.add_argument("--golden", default="golden.json") ap.add_argument("--mode", choices=["write", "check"], default="check") args = ap.parse_args() mod = load_target(args.target) lines = Path(args.corpus).read_text().splitlines() cases = [json.loads(l) for l in lines if l.strip()] results = {c["id"]: dict(zip(("sha256", "payload"), fingerprint(mod, args.function, c))) for c in cases} if args.mode == "write": Path(args.golden).write_text(json.dumps(results, indent=2, sort_keys=True)) print(f"pinned {len(results)} cases -> {args.golden}") return 0 golden = json.loads(Path(args.golden).read_text()) drift = [cid for cid in golden if results.get(cid, {}).get("sha256") != golden[cid]["sha256"]] missing = [cid for cid in golden if cid not in results] if drift or missing: print(json.dumps({"drift": drift, "missing": missing}, indent=2)) return 1 print(f"ok: {len(golden)} cases unchanged") return 0 if __name__ == "__main__": raise SystemExit(main()) Run it twice. Once to write, once to confirm the harness is stable. A pin that drifts on a second run is not a pin. python char_harness.py --target ./src/pricing.py --function normalize_offer \ --corpus corpus.jsonl --golden golden.json --mode write python char_harness.py --target ./src/pricing.py --function normalize_offer --mode check Step 4: Prove the harness can fail This is the step most people skip. A snapshot nobody has ever seen fail is unverified. So break the code on purpose, one behavior at a time. Keep each mutant as a tiny patch file. Each one must change real behavior, not formatting. # mutants/01-boundary.patch --- target.py +++ target.py @@ - if qty >= 1: + if qty > 1: # mutant probe: a failed check is a pass here for m in mutants/*.patch; do cp target.py /tmp/target.bak if ! patch -s -p0 < "$m"; then echo "SKIP $m (does not apply)"; continue; fi if ! python -c "import target" 2>/dev/null; then echo "SKIP $m (mutant does not import)"; cp /tmp/target.bak target.py; continue fi if python char_harness.py --target target.py --function normalize_offer \ --mode check >/dev/null 2>&1; then echo "MISSED $m