Tool Thursday: Telegram Bot API Alerts — 66 Bad Runs, 1 Alert (I Counted My Own Red X’s)

On 13 August I published a Tool Thursday telling you to wire up Telegram Bot API alerts as your agent’s voice. I stand by the setup instructions. I do not stand by the sentence I wrote underneath them:

“The magic isn’t the green checks — it’s the red X’s. An agent that fails loudly, in your pocket, within seconds, is an agent you can leave alone.”

This morning I counted the red X’s my own stack has sent. Across 1,565 completed scheduled runs since 12 June, 66 runs ended badly and exactly 1 of them told me so.

The channel wasn’t broken. Telegram Bot API alerts were firing the whole time — on the successes. The failures were the ones going out silent, which is the precise opposite of the promise. Here is how I found that out, including the part where my first measurement was wrong.

First, the good news: Telegram Bot API alerts are still being delivered

The build from August still holds. Message @BotFather, get a token, POST to sendMessage at the end of every run. It is free, it has no SDK, and it is three lines of curl. Nothing below is an argument against the tool.

But I needed to know whether it was actually delivering, and that turns out to be a genuinely awkward question. The Bot API has no “messages sent” endpoint. I called getChat on my own alert chat: ten fields come back, and not one of them is a counter.

There is exactly one way to ask. In a private chat, message_id is a per-chat sequence number, so you send one message and read the number on the receipt:

curl -s -X POST "https://api.telegram.org/bot$TOKEN/sendMessage" \
  -d "chat_id=$CHAT_ID" -d "text=probe" -d "disable_notification=true"
# {"ok":true,"result":{"message_id":1605, ...}}

1,605. That is every message this chat has ever held. My daemon’s session file records 19 operator conversation messages, so near enough all 1,586 of the rest are agent alerts, against roughly 1,500 successful runs. The success path is healthy. (Delete the probe afterwards with deleteMessage; I did.)

You cannot audit Telegram Bot API alerts from the logs of the thing that calls them

My first instinct was to grep my run logs, and my first answer was badly wrong. 524 of 1,566 logs mentioned the alert, so I briefly believed the channel was firing a third of the time.

Then I read one. The line was the agent’s own prose — “Step 7: Telegram alert sent” — not the script’s output. So I checked whether any command output reaches those logs at all:

• Logs containing the alert script’s real stdout, success or failure: 0 of 1,566
• Logs containing any tool-call artifact — a curl, an API hostname, anything: 0
• Logs containing the exact string Telegram alert sent.: 239, every one of them the model describing what it expected to happen

The dispatcher captures the agent’s final written summary and nothing else. The alert script’s own verdict has never once reached the log file. Neither has a failure: zero logs contain the FAILED branch, and zero contain the warning the script prints when the token is missing.

Which matters, because telegram-alert.sh is written to exit 0 when the credentials are unset. It calls that “skipping gracefully.” From inside the container, a correctly configured alert channel and a completely unconfigured one produce identical evidence: none. That is the same failure shape as a run that exits clean and reports nothing — silence being read as success.

The red X’s: 66 bad runs, 1 alert

With the logs ruled out as evidence about the channel, they are still fine evidence about the runs. 1,565 completed dispatches:

• 1,499 exited 0 — each one ends with its alert, and the chat counter agrees
• 50 exited 1
• 16 died before the dispatcher could even write an exit footer

So 66 runs went wrong. Now the number that actually indicts the August post: 0 of those 50 failed runs contain a SKILL_RESULT line. Not one of them reached the end of its own pipeline.

And that is the structural problem, not a bug. I had written the failure alert as the last line of the thing that fails. A run that dies on its first API call never gets to the step where it would have told me it died. The green checks are self-reporting; the red X’s require something outside the run to notice.

I did have such a thing — a reactive classifier in the dispatcher that greps the log after a non-zero exit and fires an alert if the error looks auth-shaped. It caught 1 of the 66.

The root cause is a word order

Here is what those 50 failures actually were, and whether my classifier’s pattern matched:

Failure family	Runs	Alert fired
Authentication failure (401)	21	1
Org subscription access disabled	21	0
Capacity / 529 Overloaded	5	0
Rate limit reached	3	0

Twenty-one genuine auth failures. One alert. The pattern I had written included the alternative authentication.*failed, which reads perfectly sensibly. The CLI emits:

Failed to authenticate. API Error: 401 OAuth access token is invalid.

“Failed to authenticate” is not “authentication failed.” Same words, wrong order, no match. The single alert that did fire got through on a different alternative, oauth.*expir, because that one run happened to phrase itself as “OAuth session expired.”

I had written the detector against the exact string that was in front of me the day I wrote it. Every other wording the same vendor uses for the same condition — invalid, revoked, invalid credentials — walked straight past it. And the largest family of all, 21 runs halted because subscription access was switched off, was a sentence I had simply never seen when I wrote the regex.

I flagged this risk in a post on 1 September after one bad night, and recommended alerting by default rather than by pattern. I never changed the code. Thirty-seven days later the count says the anecdote was, if anything, generous.

Do this today

Three checks, none longer than a coffee:

1. Read your chat’s counter. Send one probe, note the message_id, delete it. Compare it to how many runs you have scheduled. If it is wildly lower, your alerts are not arriving and nothing in your logs will have said so.
2. Move the failure alert outside the run. If the only thing that can report a crash is the process that crashed, you are relying on a dead process to speak. The wrapper that launches the job should alert on any non-zero exit, before it inspects anything.
3. Alert on unrecognised, not on recognised. Fire on every failure and allowlist the boring ones as you meet them. A pattern list only ever catches the errors you have already survived — and the one that takes you down is, by definition, one you have not.

The takeaway, and it is a correction rather than a tip: an alert channel you have never seen fail is not a channel you have evidence about. Mine delivered about 1,500 green checks flawlessly and ate 65 of 66 red X’s, and both of those things were true for 118 days while I read the quiet as good news.

If you want the schedule layer underneath this, the cron write-up is the companion piece, and monitoring before automation is the argument for doing it first. If you would rather someone wire the alerting path properly the first time, book a strategy session and we will go through your stack.