Set up model-radar: catch free LLMs that really make tool calls

model-radar is two small commands I run for my AI agents: model-radar lists the free models on OpenRouter and actually asks each new one to make a tool call. Loads of free models say they support tools and then answer in prose. model-canary watches the paid model my agents use, and the credit balance, and fails over to a cheaper model or a local one when either runs out. Neither asks a model for an opinion. They measure and act. Setup is about ten minutes, and the guide is run on a fresh Ubuntu machine every week. Install Python 3.11 or newer, nothing to pip install. git clone https://github.com/casareanderson/model-radar.git && cd model-radar cp config.example.toml config.toml ./bin/model-radar --help Check it fails safely first Before adding a key, run both with no credentials: ./bin/model-radar --dry-run # both sources "no credential, skipped", empty baseline ./bin/model-canary --dry-run # "no OPENROUTER_API_KEY -- nothing to check", exit 1 That exit 1 matters. If the canary can't check anything, a timer should notice, not shrug. The real run export OPENROUTER_API_KEY=sk-or-... ./bin/model-radar --dry-run One line per model: PASS, FAIL or BUSY. BUSY means rate-limited or timed out, and it's never treated as a verdict, it just gets tried next run. On 8 October that gave me 4 passes out of 8 probed, two 403s and two busy. One free model I'd tried earlier answered every request and never once called the tool. Run it once without --dry-run to save a baseline. From then on it only probes models it hasn't seen. The canary Point it at whatever your agents use (a JSON file, or hermes-agent profiles), add a paid fallback ladder and a local model in config.toml, and run it every few hours from cron. Two hard failures in a row and it walks the ladder. Timeouts and 429s never count. What CI can't prove Anything with a key. The weekly run has none, so it proves install, config, help and the no-key behaviour. The probes and the canary's failover need your OpenRouter account. Full guide: github.com/casareanderson/model-radar/blob/main/docs/SETUP.md How do you pick models for your agents, and how do you find out when one quietly stops working? 🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.