Give the Model One JSON Job. Run Everything Else Yourself.

Most agent messes are not “the model is dumb.” The model was allowed to drive the whole loop. Planning, file edits, test commands, retries. All of it. That is how invented paths and silent extra files show up. So I stopped doing that. The model gets one job: fill a JSON contract. Then a local validator, a path allowlist, and ordinary git/test commands do the rest. If the contract is wrong, nothing is written. Fail closed. Boring on purpose. Why wrap a language model around git and pytest at all? Those tools already know how to say no. What this tutorial builds A from-zero pipeline you can run in a throwaway repo: Snapshot a git worktree so the agent never sits on main. Freeze a JSON Schema for the only model output you will accept. Call a model once to fill that schema. Reject extra keys, path escapes, and test commands you did not allow. Apply the change with a dumb Python writer. No shell interpolation. Run the test command from the plan only after it matches an allowlist. Print a receipt. Keep or throw away the worktree. Each stage has a verification command. If a stage fails, stop. Do not “let the agent retry.” This is a recipe, not a production war story. I am not claiming timings, pass rates, or a hosted benchmark. Copy the files. Break them on purpose. That is the point. Stage 0 — empty repo, one failing test You need a target small enough that you can see every file the model might invent. mkdir /tmp/schema-gate-demo && cd /tmp/schema-gate-demo git init -b main python3 -m venv .venv . .venv/bin/activate Write a tiny function with no implementation, plus a test that should fail: # src/slug.py def slugify(text: str) -> str: raise NotImplementedError # tests/test_slug.py from src.slug import slugify def test_spaces_become_hyphens(): assert slugify("Hello World") == "hello-world" mkdir -p src tests # paste the two files, then: touch src/__init__.py tests/__init__.py python3 -m pytest tests/test_slug.py -q; echo "exit:$?" Verify: pytest exits non-zero. If the test already passes, you have nothing to gate. Pick a different function. Stage 1 — worktree, not a hope and a stash Do you really want a model editing the branch you push? I do not. git add src tests git commit -m "failing slugify" git worktree add /tmp/schema-gate-wt -b agent/slugify cd /tmp/schema-gate-wt Verify: git rev-parse --abbrev-ref HEAD # agent/slugify git status --porcelain # empty If HEAD is still main, stop. The rest of this tutorial assumes isolation. Stage 2 — the contract the model must fill The contract is the product. Not the prompt. Not the chat log. { "$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "additionalProperties": false, "required": ["goal", "files", "test_command"], "properties": { "goal": { "type": "string", "minLength": 8, "maxLength": 200 }, "files": { "type": "array", "minItems": 1, "maxItems": 3, "items": { "type": "object", "additionalProperties": false, "required": ["path", "content"], "properties": { "path": { "type": "string", "pattern": "^(src|tests)/[A-Za-z0-9_./-]+\\.py$" }, "content": { "type": "string", "minLength": 1, "maxLength": 8000 } } } }, "test_command": { "type": "string", "enum": ["python3 -m pytest tests/test_slug.py -q"] } } } Save it as contract.schema.json. Notice what is missing. No shell. No package installs. No second test file outside tests/. No ../. The enum on test_command is the whole security model for that field. One allowed string. Verify the schema is actually schema, not a blog comment: python3 -c "import json,pathlib; json.loads(pathlib.Path('contract.schema.json').read_text())" Stage 3 — a validator that fails closed Install a schema library in the venv. Then write a checker that also enforces repo-relative paths. pip install jsonschema # tools/check_plan.py from __future__ import annotations import json import sys from pathlib import Path import jsonschema ROOT = Path(__file__).resolve().parents[1] ALLOWED_TOP = {"src", "tests"} def load(p: Path): return json.loads(p.read_text(encoding="utf-8")) def assert_safe_path(rel: str) -> Path: candidate = (ROOT / rel).resolve() if not str(candidate).startswith(str(ROOT.resolve()) + "/") and candidate != ROOT.resolve(): raise SystemExit(f"path escapes repo: {rel}") if candidate.parts[len(ROOT.resolve().parts)] not in ALLOWED_TOP: raise SystemExit(f"path not in allowlist: {rel}") if not rel.endswith(".py"): raise SystemExit(f"refusing non-python path: {rel}") return candidate def main(plan_path: str) -> None: schema = load(ROOT / "contract.schema.json") plan = load(Path(plan_path)) jsonschema.validate(plan, schema) for item in plan["files"]: assert_safe_path(item["path"]) print("plan ok") print(json.dumps({"files": [f["path"] for f in plan["files"]], "test": plan["test_command"]}, indent=2)) if __name__ == "__main__": if len(sys.argv) != 2: raise SystemExit("usage: check_plan.py plan.json") main(sys.argv[1]) Verify with a bad plan first. A validator you have never failed is decoration. { "goal": "hack", "files": [{ "path": "../secrets.env", "content": "x" }], "test_command": "python3 -m pytest tests/test_slug.py -q" } python3 tools/check_plan.py /tmp/bad-plan.json; echo "exit:$?" You want a non-zero exit. Schema should reject ../secrets.env via the path pattern. If it does not, fix the pattern before you ever call a model. Stage 4 — one language call, then disk Here is the only place a model belongs. Write the prompt as a file. Keep it short. The schema is the real instruction. Fill plan.json for this repo. Implement slugify() so tests/test_slug.py passes. Return ONLY JSON that matches contract.schema.json. Do not invent files. Do not change the test command. How you obtain that JSON is interchangeable. A local script, a curl to an endpoint you already pay for, or a free-tier host. The gate does not care. Disclosure: This article was prepared as part of MonkeyCode's product outreach. For the language step I pointed this recipe at MonkeyCode, an open-source project with free model access and a free server option. That is the only product-shaped part of this pipeline. If you already have another endpoint, swap the HTTP call. Keep the schema. I am not going to invent model names, token ceilings, or hardware. Put the base URL in an env var and refuse to log the response body to a shared chat. # labeled example: wire this to whatever endpoint you control export MODEL_BASE_URL="${MODEL_BASE_URL:?set me}" # your one-shot call writes plan.json — then: python3 tools/check_plan.py plan.json Verify: check_plan.py prints plan ok and a file list you recognize. If the model added src/utils_helper2.py and you did not ask for it, delete plan.json. Do not negotiate. No plan file? Stop. Do not fall back to “just let it edit.” Stage 5 — a dumb writer, not an agent apply The writer overwrites only paths the plan already passed. # tools/apply_plan.py from __future__ import annotations import json import sys from pathlib import Path ROOT = Path(__file__).resolve().parents[1] def main(plan_path: str) -> None: plan = json.loads(Path(plan_path).read_text(encoding="utf-8")) written = [] for item in plan["files"]: dest = ROOT / item["path"] dest.parent.mkdir(parents=True, exist_ok=True) dest.write_text(item["content"], encoding="utf-8") written.append(item["path"]) print("wrote: " + ", ".join(written)) if __name__ == "__main__": main(sys.argv[1]) Run check, then apply, then status: python3 tools/check_plan.py plan.json && python3 tools/apply_plan.py plan.json git status --porcelain git diff --stat Verify: porcelain only lists files under src/ or tests/ that you expected. An untracked notes.md means the writer is not the only thing touching disk. Hunt that down before tests. Stage 6 — run the enum, not a free-form shell The plan’s test_command is already an enum of one string. Still, do not os.system a model string. Duplicate the allowlist in the runner. # tools/run_allowed_test.py import json import subprocess import sys from pathlib import Path ALLOWED = ["python3 -m pytest tests/test_slug.py -q"] plan = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8")) cmd = plan["test_command"] if cmd not in ALLOWED: raise SystemExit(f"command not allowed: {cmd!r}") raise SystemExit(subprocess.call(cmd.split())) python3 tools/run_allowed_test.py plan.json; echo "exit:$?" Verify: exit 0, or you reject the worktree. A failing test is not a signal to “ask the model to try more files.” It is a signal that this plan is done. Want a second attempt? New worktree. New plan.json. Same schema. That constraint is the whole tutorial. Stage 7 — receipt, then you decide { echo "branch: $(git rev-parse --abbrev-ref HEAD)" echo "head: $(git rev-parse --short HEAD)" echo "plan files:" python3 -c "import json; print('\n'.join(f['path'] for f in json.load(open('plan.json'))['files']))" git diff --stat } | tee receipt.txt If the diff is the slug implementation you wanted, commit on agent/slugify and open a normal review. If it is not, cd /tmp/schema-gate-demo && git worktree remove --force /tmp/schema-gate-wt. No debate with the model. The worktree is cheap. Decision table I actually use Symptom Action Schema fail Delete plan.json. Do not apply. Extra path in allowlist shape you did not ask for Delete plan.json. Tighten maxItems or the prompt. Path escape or non-.py Treat as hostile. Stop the session. Test command not in enum Treat as hostile. Stop the session. Tests fail after apply Discard worktree. New attempt, new plan. Tests pass, diff includes surprise files Discard worktree. The writer or the shell leaked. See the pattern? The model never gets a vote after JSON lands on disk. Limitations This gate does not prove the implementation is right beyond one test. A model can hard-code return "hello-world" and still pass test_spaces_become_hyphens. Add cases before you widen the allowlist. JSON Schema will not catch a clever path that your regex permits. Keep the pattern tight. Keep maxItems tiny. A free-tier language step can be slow, rate-limited, or down. That is fine. The pipeline should idle, not degrade into “edit locally with extra retries.” I did not wire secret scanning, dependency installs, or multi-file refactors. Those need different contracts. Do not stretch this one. Who should not use this Skip this if you need a long-running agent that shells out, installs packages, or browses the web. This recipe will fight you, on purpose. Skip it if you cannot run pytest locally. A schema gate with no test runner is just a linter for fiction. Skip it if your change set is a 40-file migration. One JSON job with maxItems: 3 is the wrong shape. Split the work or write the patch yourself. And skip it if you were hoping the model would also “own git.” Git stays yours. Keep the loop small The interesting AI news this week is still the same failure mode in new clothes: a loop that assumes, retries, and touches too much. You do not fix that with more tools. You fix it by taking tools away. One schema. One call. One allowlisted test. Then a human. If you need a free-tier endpoint for that single language call, MonkeyCode’s free model access and free server option are what I pointed the recipe at. Leave the rest of the pipeline ordinary Python.