Wednesday Wisdom: When the Alarm Says Broken, Check the Alarm First (Two False Alarms in One Day — Both Mine)

Last night at 20:05 my video agent started a render. It died at step 3 of 10 with a vendor error code. The agent did the responsible thing and probed the API to find out whether the service itself was up. The probe came back HTTP 503.

That is a clean, unambiguous story. The vendor is down. The correct, cautious, money-saving move is to stop paying for renders and ship a static image instead. My agent was one line of code away from doing exactly that.

The service was fine. The render that finally shipped is a real 43-second video. Here is the wisdom, and then the two receipts that earned it — both from the same 24 hours, both from instruments I wrote myself.

The wisdom: a red light is a claim, not a result

Everyone building with agents eventually learns to distrust green. A clean exit code means the script finished, not that the work happened. That lesson is cheap because you learn it the first time an agent reports success over an empty file.

Almost nobody applies the same suspicion to red. When an instrument says broken, we act — immediately, and without auditing the instrument — because acting on a failure report feels like diligence. That is precisely the problem. A false alarm does not look like a bug. It looks like good judgment. It looks like you caught something.

So say it out loud: your monitor is a system too. It makes network calls. It parses responses. It reads caches. It can be wrong, and when it is wrong it is wrong in exactly the same silence as everything else you built.

Receipt one: the 503 that was not an outage

The render failed with status=failed: Director: unexpected error handling prediction (E9828) at step 3 of 10. A real failure — no argument. So my agent probed the prediction endpoint and got a 503, and the probe script called the URL once and raised on that single response.

One read. One verdict: the service is down.

Re-polling twelve seconds later showed that the probe’s own test prediction had succeeded. The 503 was on the read path, not the generate path. The service could produce images the entire time; it just briefly could not tell me about it. A retry produced a clean 10 of 10.

What actually shipped: 1080×1920, 43.1 seconds, narration present at a mean of −22.4 dB, confirmed with ffprobe rather than assumed. It was still serving from the CDN when I checked this morning. Not a slideshow.

Now the part worth sitting with. The fallback was cheaper. A 503 on a read endpoint would have bought permission to skip a paid render and ship something worse — and the run log would have recorded success, because shipping the fallback is a supported outcome. Nothing in my fleet would ever have flagged it. I would have simply published a worse video and felt prudent about it.

Receipt two: three days of a bug that was my own tool

For three separate mornings I wrote in my notes that my SEO plugin needed two API calls to save metadata. The evidence looked airtight: the write returned success, then a plain GET of the live URL showed the raw post title and the excerpt sitting where my custom title and description should be. Write, check, still wrong, call again. I logged it three times as a vendor defect.

Yesterday I re-fetched the same URL with a cache-busting query string and a Cache-Control: no-cache header. Both fields were live and correct — and had been all along.

The bug was the GET. It was reading a cached page. Three days of a documented API defect that I am now fairly confident never existed. The API was never the unreliable component. My instrument was, and my instrument was the only thing I never thought to test.

So this post is the experiment. One metadata write, one verification, cache-buster attached, no second call. If the wisdom is real it should hold on the very post that argues for it.

Update, four minutes later: it held. One write. Then the same URL read two ways in the same breath — the plain GET returned the raw post title and the excerpt, the cache-busted GET returned the exact title and description I had just written. Same page, same second, two different answers, and only one of them was true. The second call I made for three days running was never fixing anything. It was just giving the cache time to catch up.

The mirror: red is not proof, but neither is green

The same morning, my image script printed WARNING: Airtable log failed — UNKNOWN_FIELD_NAME: "Name" and then exited zero. It did that again on the image at the top of this post. Fifth consecutive day.

I read the live table schema this morning instead of guessing at it. The fields are Image ID, Created, Image URL, Image Model, Saved Image, Prompt, Status, Site, Article Title, Task Reference, Prompt Summary, Suggested Image Category, WP Media ID, WP CloudFront URL, Reference Image Used. There is no Name field. The write cannot ever succeed. It is not flaky — it is impossible, and it has been reporting itself as a warning inside a successful run for five days while the table quietly received nothing.

And the structural version of the same defect, measured across my own agent definitions this morning: 39 skill files, 57,438 words, 217 references to retry, except, or fallback — and zero occurrences of the word “assert.” My fleet is exhaustively prepared for a call that fails and has no vocabulary at all for a call that lies. In either direction.

Three lines to steal

1. Read twice, by two different paths, before you believe a failure. A different endpoint or a different cache, ten seconds apart. One read is a rumor. Two agreeing reads are evidence.

2. Ask what “broken” buys you. If a failure verdict routes your agent to the cheaper, faster, lower-quality path, that is exactly where false alarms hide — a cost-saving fallback writes its own justification, and no alarm ever fires for the alarm.

3. Bracket every empty or failing read with a control. My task-checker reported an empty queue last night and I believed it for one reason only: the identical query, same shape, aimed at a target known to be full came back full. If your control also reads broken, your instrument is broken, not your system.

None of this is exotic. It is the difference between an agent that reports what happened and an agent that reports what its tools claimed. You already know not to trust a success that produced nothing; the harder discipline is not trusting a failure that cost you nothing to declare. It is why monitoring comes before automation, why a ten-second health check beats a hopeful retry, and why “set it and forget it” stays the most expensive lie in this field. Yesterday I counted a queue in days instead of rows and found an outage a day before it happened — same principle, pointed at capacity instead of at alarms.

Want the plumbing that makes this boring instead of exciting? Auditing the instruments is most of the real work behind agents you can actually leave alone. Book an automation strategy session and we will go find out which of your alarms are lying to you.