"Local green" is not evidence. It's the whole thesis Heimdall is built around, and it's also, honestly, how we found out the hard way. Standing up the control plane — the service that dispatches work and the Cloud Run Job that executes it, backed by Firestore instead of a laptop's filesystem — surfaced 29 real bugs before the maintainer ever opened a PR on its own. We didn't hide them; the full 29-bug ladder is on the proof page. This post goes deeper on eight of them, picked because each one has a clean, citable "why the laptop lied" story and a test file you can go read right now.
The pattern that kept recurring: a Cloud Run container is not a developer laptop. Its filesystem is read-only outside a scratch dir. It has no gcloud SDK unless the Dockerfile put one there. Its storage backend is Firestore, not a real directory tree, so anything that assumes a filesystem path() exists is one deploy away from breaking. And a Google Front End load balancer sits in front of every request, with its own opinions about HTTP that a local dev server never enforces. None of that is visible to a unit test run on the machine that wrote the code — the laptop is the false environment.
1 · The container had no gcloud
Dispatch shelled out to a binary that didn't exist
- What we saw
- Jobs queued cleanly but never ran. No crash, no error in the service logs — the task simply stayed queued forever.
- Why local was green
- The dispatcher shelled out to
gcloud run jobs execute. On every developer machine, the Google Cloud SDK is already installed and onPATH— the subprocess call just worked, every time, in every local run. - Root cause
- The Cloud Run image never installed the
gcloudSDK. Dispatch shelled a binary that didn't exist in the container — the exact failure mode the maintainer's own deployed-shape lessons now name as "prove the deployed dep shape, not the laptop's." - The fix
- Dispatch moved off the shell entirely:
JobsClient().run_job(RunJobRequest(...))over the Cloud Run Jobsrun_v2API, authenticated with Application Default Credentials — no subprocess, no external binary. The import is lazy so a local run without the extra dependency installed stays green.
2 · The container had no writable filesystem — and no checkout
A repo slug got joined onto a read-only CWD like it was a checkout
- What we saw
- The maintainer's cloud job crashed at startup:
PermissionErrortrying to create its own planning directory. - Why local was green
- Locally,
--repo owner/nameresolves against a real working checkout already sitting in the current directory. The code joined that repo slug onto the CWD as though it were always a checkout path — and on a laptop, the CWD is always a writable git repo, so the join always landed somewhere real. - Root cause
- The cloud job has no checkout at all. Its CWD is
/app, the read-only application image root. A bare slug likeowner/namejoined onto/appproduced a path that was neither writable nor a real repo, andos.makedirs()raised. - The fix
- A new
resolve_workspace()step: an existing local dir is reused unchanged (so a workstation run stays byte-identical), a bare slug is shallow-cloned into a writable$HEIMDALL_HOME/work/<owner>__<repo>, and a traversing or ill-formed slug is refused fail-loud before any path is ever joined.
3 · path() is refused by design under Firestore
One assumption, six independent call sites, six separate breaks
- What we saw
- The background tick errored every 60 seconds in the live Cloud Run logs —
BackendUnavailable— on a code path that was clean in every local and hermetic test. - Why local was green
- The local storage backend is a real filesystem, so calling
.path()on it legitimately returns a filesystem path — everyopen(backend.path())oros.path.isdir(backend.path())call worked, every time, locally. The Firestore-backed production storage refusespath()by design — a Firestore document has no filesystem path to give you — but nothing in the calling code accounted for that. - Root cause
- This wasn't one bug, it was one bad assumption baked into six independent call sites over time: the tick's owner-registry read, the dashboard's
stored_instancesscan,cp_ingest/cp_notify's result payload, and the CLIstatuscommand all calledbackend.path()and treated the local-only behavior as universal. - The fix
- Each call site was swept, one at a time across several commits, off
path()onto backend-safe primitives —get_record(),exists(),append_line()— that both backends implement without leaking filesystem semantics. The tick incident specifically moved to reading the owner registry throughcp_auth.owner_haids()instead of opening a file at a path.
4 · A doc-id collision quietly dropped a team's presence
The write succeeded; the read landed on a different document
- What we saw
- A team's presence would intermittently vanish from the live roster: visible on write, invisible on read, no error either side.
- Why local was green
- The hermetic presence test ran on the local filesystem-backed storage, where a doc-id containing a literal
__is just two underscores in a filename — it round-trips fine. Nothing about that test could see what Firestore does with the same string. - Root cause
- The production Firestore backend uses
__internally as its own separator, encoding a nested path into a flat document-id namespace. The presence slug generator could itself emit__, so a slug liketeam__xcollided with the backend's own encoding and got read back as a different node than the one written to. - The fix
- Any run of underscores in a generated slug now collapses to a single
_before it's used as a doc-id, so the write path and the read path can never disagree about what the flat key actually is.
5 · The job finished — into a document the service never read
Two containers, two Firestore identities, one "job"
- What we saw
- A dispatched job's own logs reported
done. The service that had dispatched it kept polling, never saw completion, and looked hung indefinitely. - Why local was green
- Locally, the "job" and the "service" are the same process sharing one storage identity by construction. There was no way for them to disagree about which document they meant, because there was only ever one.
- Root cause
- The Cloud Run Job runs as a genuinely separate container from the service. Nothing propagated the service's exact Firestore identity — backend, project, root prefix, and the one that actually bit, the database coordinate, which had silently stayed pinned to
(default)— into the job's environment. Job and service could each resolve a valid-looking but different Firestore document for the same job id. - The fix
- Dispatch now threads the service's complete storage identity into the Job's environment overrides, so the Job writes the literal document the service is polling, by construction rather than by convention.
6 · A swallowed error degraded every roster read to a full scan
It worked — right up until the collection grew
- What we saw
- The live team roster came back empty on Firestore, even though presence writes were landing cleanly. This one is unusual: it worked in production too, for a while.
- Why local was green
- The bounded read tried to build a server-side document-id range query, but two things were wrong with it and both failed silently:
firestore.FieldPath.document_id()isn't actually exported at the top level of the pinned client (google-cloud-firestore==2.16.1exposes onlyFieldFilter), so it raisedAttributeError— and that was caught and swallowed, silently degrading the "bounded" read into a full-collection scan. On a small collection, a full scan is still fast enough to return everything inside the timeout, so it looked correct everywhere — local, staging, and early production. - Root cause
- As the production
heimdall_cpcollection grew past what a full scan could finish inside a 20-second timeout, the degraded scan started timing out and returning[]— a real backend read failure silently reported as "nobody's online." A second, independent bug compounded it: even when the range filter did build, it compared the document-id against a plain string instead of aDocumentReference, which Firestore never matches. - The fix
- The document-id sentinel is now the literal
"__name__"string (no version-fragile import), bounds are built from realcol.document(id)references so the range filter actually pushes down server-side, and a read that fails now writes one loudstderrline —_warn_read_failure— so an empty roster caused by a backend error is never indistinguishable from an idle team again.
7 · Google's load balancer rejects a GET with a body
The request never reached our code at all
- What we saw
- A signed read-back of a job's state worked fine hitting the handler directly, but over real HTTP against the deployed Cloud Run service, the client got a bare
400before any application code ran. - Why local was green
- Local testing drove the handler function directly (or through a plain local HTTP server), which doesn't sit behind Google's Front End (GFE) load balancer — the thing that fronts every Cloud Run service in production. Our authorization code was never wrong; it never got the chance to be, because the request never arrived.
- Root cause
- The client sent the signed
job_idas a JSON body on aGETrequest. HTTP technically allows a body onGET, but GFE rejects it outright at the edge — a platform-level behavior with zero local equivalent. - The fix
job_idmoved to a signed query parameter. The signing scheme now covers the full path-with-query, so aGETcarries an empty body and existing query-lessPOSTroutes verify byte-identically.
8 · A discarded return value faked 15 straight successes
The bug that wasn't about the cloud at all
- What we saw
- For 15 straight maintainer runs, the loop reported opening a pull request —
PR_OPEN— even on runs wheregh pr createhad actually failed. - Why local was green
- This one isn't a deployed-shape bug — it's the silent-failure class, and arguably the more dangerous one:
open_pr()'s return value was discarded by its caller. There was no branch for "it failed," locally or in prod, because nothing ever checked. Early in developmentgh pr createhappened to succeed every time it was exercised, so the missing check never got a chance to matter until a real failure (an expired token, a wrong--repo, a git-identity mismatch — three separate causes fixed individually in bugs #21, #26, and #29) hit it. - Root cause
- A classic discarded-result bug, just in a place where the result was "did we actually tell the truth about what happened." The loop kept reporting the same optimistic state regardless of what
gh pr createactually did. - The fix
- The caller now honors
open_pr's return value. A failed create produces an honestPR_FAILEDstate — branch pushed, PR not created — flagged and re-runnable, with the failure propagated loudly to the job row instead of a fabricatedPR_OPEN.
The lesson wired in
Every one of these bugs is a variation on the same failure: a test that can only pass locally isn't proof, it's a laptop's opinion. The fix wasn't "write more tests" in the abstract — it was making firestore-mode and deployed-shape coverage a structural requirement for any code that stores state or dispatches work, and making every failure surface loudly instead of degrading silently. Two concrete things came out of this saga and are load-bearing in the codebase today:
A static deployed-shape preflight. bin/heimdall-deployed-shape-check is a stdlib-ast static checker that flags the recurring shapes above — slug-onto-CWD paths, unguarded local-state reads, Cloud Run reserved env names in an override dict, captured-but-ignored subprocess stderr — as file:line warnings before a deploy ever runs. It's proven against the reconstructed pre-fix snippets of several of these bugs and their fixed forms, wired warn-only into the deploy scripts today.
The error_tail discipline. Bug 6's silent AttributeError and bug 8's discarded return value are the same shape at different layers: something failed, and the system reported success anyway. Every subprocess and backend read in the maintainer loop now surfaces a scrubbed failure reason on the way out — a stuck task gets a name, not a shrug.
None of this is a claim that the bugs stop coming. It's a claim that the next one gets caught by a test that can actually go red, on the shape that actually ships — not by whoever notices production is quiet. The full ladder — all 29, not just these eight — is on the proof page, alongside the isolation oracle and the first PR itself.
See the full 29-bug ladder and the first autonomous PR.
The headline claim, the isolation oracle, and every bug in the bring-up — listed, not hidden.