
For five months, about one in five of our live product pages had quietly dropped out of Google’s results. The cause was a one-line change to a file that tells crawlers what they may read.
We had a monitor for exactly this. It collected search data around the clock and sent an AI-written summary to Slack twice a day. It didn’t catch it. Worse, when I finally looked closely, the monitor had stopped watching the sitemap months earlier, and the only trace was one log line a day. Thousands of broken sitemap URLs had been collected and never diagnosed: the alerts existed, but nobody owned them. And its warnings also fired on harmless changes, so they had stopped meaning anything.
That monitor taught me the most useful rule I have for working with LLMs, and it’s the subject of this post: let code decide, and let the model explain.
This is part 2 of a series on what I learned putting AI agents to work in my own job. Part 1 was about how my work changed.
Two kinds of answers
Some tools are deterministic: the same input always gives the same output. A spreadsheet formula, a billing calculation, a database query. Run it a hundred times and you get the same answer a hundred times, and when it’s wrong, it’s wrong the same way every time, so you can find and fix it.
An LLM is probabilistic. It writes the most plausible continuation of what it has seen, so the same question can get a different answer, sometimes a different conclusion. That’s why it’s brilliant at drafting, summarising, translating and classifying, and why it can be confidently wrong about a number.
So I split the work by that line:
| Deterministic code | LLM | |
|---|---|---|
| Ask it again | Same answer | Maybe a different one |
| Good at | Totals, thresholds, state, rules | Drafts, summaries, diagnosis, explanation |
| When it’s wrong | The same way every time, so you can test and fix it | Occasionally, without warning, and plausibly |
The everyday version: don’t ask the model “what’s the March total?” Ask it to write the formula. The formula does the arithmetic; the model’s job is to build it and explain the result.
Why not just let the model look at the data?
It’s tempting. You point an agent at the dashboards every morning and ask, “anything wrong?” I considered exactly that, and wrote down why not:
- The same data can get different judgments, and a missed one leaves no signal. I can evaluate a model against past cases, but I can’t rely on it to make the same call every time: a spike it calls fine on Tuesday may be a problem on Wednesday. And if it simply doesn’t mention something, nothing tells you it was missed. For these checks I wanted explicit rules that give the same answer every run.
- If the model stops, the alerts stop. API outages, rate limits, an empty credit balance (it happened to me, see part 3): an alerting path that runs through a model inherits every one of those failures.
- Cost and load become unpredictable. If the model decides what to query each time, you can’t say in advance how much it will read, or how much it will cost.
So in all three monitors I rebuilt, code makes the decision. Where I use a model at all, it reads the results and explains them. The search monitor shows the full shape:
Monitor 1: search visibility, rebuilt in a day
The new search monitor works like this:
- Code assigns severity. Each kind of problem gets a count, a severity and a state (new, worse, resolved, back again), stored in a small database. Severity decides the route.
- Critical goes straight to the channel within ten minutes, with no model in the path. That includes the failure modes that bit us: the crawler rules file changing, a sitemap not registered, collection stopping.
- The monitor watches itself. If collection stops or the daily report doesn’t arrive, that’s a critical alert too. A half-dead monitor was the original problem.
- The model gets the rest, once a day. It reads the results, diagnoses each group of problems, files a tracked issue per cause, and posts a summary. An afternoon follow-up only wakes the model if something changed since the morning; otherwise there’s no model call at all.
I designed, built and deployed it in a day with a coding agent. On its first run it found a problem nobody knew existed: a group of pages pointing search engines at URLs that returned 404.
Monitor 2: price checks that recheck before alerting
Another check compared the prices in a feed we send to a price-comparison site against the prices on our own pages. It sent a message on every run, about 30 a day, and every morning it raised a false alarm. The feed is rebuilt a few times a day with that moment’s exchange rates, while our pages update immediately, so for a few hours each morning most foreign-currency products disagree with the feed by the same small ratio.
The clever fix was obvious: if every product in a currency is off by the same ratio, it’s exchange-rate lag, so ignore it. I tested that rule by replaying 400 logged runs. It produced 97 false alarms: ratios differed slightly between products, and some products didn’t follow exchange rates at all.
The boring fix is what I shipped, rebuilt in a day. A small state machine: when a price disagrees, don’t alert, remember it. After the next feed rebuild, check that product again. Still wrong? Alert. Right? Close it quietly.
Replaying 180 past alerts through it: 174 cleared after the next rebuild, 5 products had dropped out of the feed, and one was a real problem, a coupon price that had been missing from the feed for days. The state machine set that one apart from the temporary mismatches. Now the check alerts only when a mismatch survives the next feed rebuild, alongside two summaries a day. It doesn’t use a model at all; I decided not to add one until it’s needed.
The trade-off is written down: a real price problem now surfaces only after the next rebuild, up to a few hours later during the day and up to half a day overnight. For this check, that’s the right price for silence.
Monitor 3: operations, a rule engine and a once-a-day reader
Over three days I ran the same production health check by hand three times, and each time it found something new. One of them was a twelve-minute database saturation that dozens of existing alarms had missed, because the alarm needed a longer spike before it fired.
So I built a rule engine in about two days: 40 rules and 48 tests, built around three “golden days” of real data (two with incidents, one quiet). It runs every 30 minutes with no model involved. In the design, a model reads a small digest of the results once a day, under 30 KB, and writes the summary and the issues.
Two checks gave me enough confidence to keep testing it in shadow mode:
- Replay before trust. I tuned it against a full week of real logs until it raised at most two medium alerts a day, and every high-priority alert on a non-incident day turned out to be a real event.
- Shadow mode first. It’s still running without paging anyone. That’s already paid off: it flagged two server restarts as an incident because they were 12.4 minutes apart and the “this is a deploy” window was 10. Fixed, with new tests, before it ever woke anyone up.
It’s honest to say this one hasn’t caught a live incident yet. One cost number I did measure: caching cut the logs it scans per run to about a fifth.
What I took from it
- Put the decision where it repeats. A model can be evaluated, but only a rule in code gives the same answer every run, and can be replayed against last week to prove it.
- Replay before you trust a rule. The clever heuristic looked fine on paper; a replay of real runs showed 97 false alarms before it ever shipped.
- Keep the model out of the urgent path. Critical alerts should fire even when the model, the API or your credit balance is down.
- Make extra model calls conditional on change. The search monitor’s afternoon follow-up stays quiet unless something changed since the morning.
- Watch the watcher. A monitor that stops quietly is worse than none, because you think you’re covered.
- Design the silence. An alert nobody owns teaches everyone to ignore alerts. Every alert should mean “a person needs to act”.
The model still does a lot of the work here: it wrote most of the code, it reads the results every day, and it writes the explanations people actually read. It just doesn’t get the final say on whether something is wrong.
The series
- From doing the work to delegating it
- Let code decide, let the LLM explain (this post)
- The rules I never break
- Context is the job
- Agents forget, so I write it down
- kenny: an agent that does my job
- Hermes: reminders that carry the check
- How to start in two weeks
Where in your own systems is a model making a call that a few lines of code could make, and test?