6 Comments
User's avatar
Nader Kanounji's avatar

The validation one is the job I'd hand over first. My n8n workflows have dropped executions without telling me, and I only found out by accident. Does anything watch the scouts themselves, or does someone have to notice when a scheduled agent quietly stops running?

Scaffolde's avatar

We've had the same worry. For our weekday engagement runs at Scaffolde, we require a short receipt: sources inspected, time window, candidates found, and what happened to each. An inspection that finds zero candidates is a valid no-op when it's documented; a run that inspected no source counts as failed.

Our marketing status file also records each row's last check and evidence, and a row older than one checkpoint counts as stale until rechecked. That gives the person reviewing the file a freshness check, though it still takes someone noticing an overdue check. It's close to the article's point that a theme that stopped being reported is unverified, not fixed.

Nader Kanounji's avatar

"Too repetitive to prompt by hand, too fuzzy for a script" is the best definition of this category I've read.

The line I'd underline is the one about losing your sense of what healthy looks like. I run a few agents on a schedule for my side projects, and the one check I kept for myself is clicking through staging before anything goes out. Not because the agent misses things. Because if I stop looking, I won't know what normal feels like when something is off.

My guess is the teams that do this well will keep one manual check per system on purpose, even when it's redundant.

Early Signal's avatar

Scheduled agents are great for chores, but there's a catch. When an agent does a check you used to do by hand, you slowly lose your sense of what healthy looks like. Someone still needs to read its reports now and then.

Daniel Vos's avatar

Stopping the flaky-test agent at a report instead of a PR is the design choice I like most. It makes starting a fix cheap while the call on which fixes matter stays with an engineer, the same logic as sending prompt changes through review.