Scheduled AI Agents — What Weeks of Unattended Runs Teach You

An agent you watch is a tool. An agent that runs at 04:00 while you sleep is a cron job with opinions. I've spent the last few weeks running a handful of the second kind inside Orca, my fork of an open-source agent-workspace app, on top of omp. This isn't a success story with numbers. It's what broke, what held, and what I'd build into every scheduled agent from day one. The thesis is the same as in harness vs. model: the model is rarely the problem. The guardrails around it are.

What's Running

• A daily upstream-sync agent: my fork tracks an upstream project, and every day a fresh git worktree starts an agent that follows a runbook, merges upstream, and opens a PR.
• A daily research agent that searches, filters, and publishes one HTML page. Its only output is that page.
• A smaller twice-weekly variant of the same pattern.

All three get a fresh workspace per run and a prompt that points at a file in the repo instead of containing the logic.

Put the Logic in a Runbook, Not in the Schedule

The automation prompt for the sync job is short: read scripts/sync-upstream.runbook.md from origin/main, follow it step by step, "it is the single source of truth; it wins over anything you assume." Then a few hard rules: never push to main, never merge the PR, open it as the right GitHub account.

That split matters. The prompt lives in a scheduler UI where nobody reviews changes. The runbook lives in git, gets reviewed, and changes with the code it describes. When the procedure changes, I edit a markdown file in a PR, not a scheduler field.

Guard Against Duplicate Work

Step 3 of the runbook is the guard: if a sync PR is already open, comment on it and stop. No second branch, no second PR, no merge attempt on top of unreviewed work.

It worked exactly as designed, and it is also the failure mode. Sync PR #11 stayed open for days. Each daily run correctly did nothing except comment with how far upstream had moved: 126 commits past the PR's base one day, 251 ahead of origin/main by the last run. By then GitHub reported the PR as CONFLICTING, and a trial merge showed five conflicts, including a database-migration snapshot where the runbook says to take upstream's version and regenerate the fork's own migration.

Lessons:

• A one-open-PR guard turns a slow human into a growing backlog. The guard prevents chaos, but it needs a second signal when the PR has been open too long.
• The cheapest recovery was closing the PR and syncing fresh. I allowed the agent to do that. The fresh merge aborted with exit code 2 and roughly 140 conflicting files, so the agent stopped without pushing. That's the right outcome: the job's hard rule is "never push main" because every push to main deploys to production.
• Falling far behind upstream is its own cost. A daily sync is cheap when daily; it gets expensive when skipped for two weeks.

Idempotent State Beats Clever Memory

The research agent runs unattended once per day and tracks what it has already reported in a state file. The rules I'd copy anywhere:

1. Read the state first, write it last. If the run dies halfway, tomorrow redoes the work rather than skipping it.
2. Key on a stable identifier, not on the text of the item.
3. Never let the agent "remember". Memory is a file in the repo or the page itself. A fresh worktree has nothing else.

This is the same idea as an idempotent queue consumer: a rerun must be safe.

Publishing Is the Step That Fails

The research agent's only deliverable is a page published through the Orca CLI. One morning every run failed at the last step with:

Error: No value provided for input HTTP label: Bucket.

It failed for the real page and for a tiny test page with the most restrictive visibility. Auth was valid and the host healthy. The agent's own conclusion, labelled as inference: a missing storage bucket in the pages backend, which it can't fix from inside a run. It reported the failure instead of faking a URL. That's correct behavior and the thing I check first in any unattended agent: does a failed final step produce a loud failure, or a plausible lie?

Cause on my side: the pages feature stores content in object storage behind a Cloudflare Worker, and on the self-hosted instance that storage hadn't been enabled. Enabling R2 needs a payment method on the account even though the free tier covers small use, and the Worker needs wildcard DNS and two matching secrets between Worker and API. That's a deployment checklist, not agent work.

The free-plan TLS trap

While debugging page links I hit ERR_SSL_VERSION_OR_CIPHER_MISMATCH. The Worker's TLS was fine. The cause was the hostname shape:

• <id>-frame.example.com — one label under the wildcard, covered by the free universal certificate.
• <id>.frame.example.com — two labels, not covered, handshake fails.

curl against the Cloudflare IP with each name showed it in a minute. The remaining bug was older builds still generating the dotted form, because the API builds page links from shared code. Rule: on Cloudflare's free plan, keep per-item subdomains to one label.

Notifications: Only on Completion

My notification rule for agents: sound when a task finishes or needs me, never for housekeeping. In Orca I removed the sound on closing a tab or session and on closing all ports. It sounds trivial. With several agents running, noise that isn't a signal trains you to ignore the sound that is one.

Tame the Review Bot Too

The same discipline applies to a review bot. A team's GitHub Copilot review ruleset had review_on_push and review_draft_pull_requests both on. The complaint: handle comments, mark ready, and the bot finds new points again. Reading the docs, there's no "only on new commits" switch. The options were:

1. Turn off draft reviews: drafts stay quiet, one review at ready, then one per push.
2. Turn off review on push: one review at ready, re-request manually after big changes.
3. Remove the rule and request the review by hand.

I recommended option 2 because it fits "bot first, humans second". I didn't reproduce the draft-to-ready behavior; that's the docs plus inference.

Where It Still Breaks

• Silent staleness. A guard that says "stop" produces no failure, only a growing gap.
• Environment drift. The mobile client once called localhost because a root .env was loaded with override: true and beat the command-line values.
• Tooling flakiness. A release upload died on a TLS timeout and had to be retried. One MCP check showed three servers connected and one failed.
• Slow browser bridges. I was chasing a browser relay that made requests feel slow. I don't have a finished root cause to share.

None of that is model quality. For the review habits that apply here too, see Claude Code in real client work.

My Checklist

1. Logic in a versioned runbook, prompt only points to it.
2. A guard against duplicate work, plus an alert when the guard fires repeatedly.
3. State in a file, read first, written last.
4. A final step that fails loudly.
5. Hard rules about what the agent may never touch.
6. Notifications only for completion or needs-you.

If you want agents doing recurring work for your shop or team without surprises, Get in touch.