Swapping models mid-project

ScanReady was a test as much as a project. I’ve been developing a loop theory for autonomous builds, along with a refinement loop. I’d already run earlier experiments with refinement and reflection loops to improve model performance and output, whether the model was a frontier model or a small local one. This time I wanted to put a different piece under real pressure: the resume ledger. The question was simple. If a model is interrupted mid-build, by a usage limit, a dropped connection or anything else, can a different model read what was left behind and pick up without missing a step? If it can, a build can keep going on its own with very little human intervention.

How the handoff worked

Every model got the same short starting prompt: start at the start-here file, then work through the plan one item at a time. It ran on a heartbeat, a timed loop that re-ran the prompt every few minutes. Each tick started with a fresh read, did one item, then reflected on it and audited it. If the work didn’t meet its acceptance criteria, it went around again until it did. The start-here file pointed to the current sprint sheet, the progress ledger and the next task. With that, a model could build its full context without me re-explaining anything and resume exactly where the last one stopped. My prompts stayed minimal. The files did the work.

What worked

Every model picked up and resumed: Codex models, Claude Code models and a local model, all working from the same ledger. The clearest example came near the end. Codex built and verified the development archive, then hit its usage limit before it could log the work. Claude Opus 5.5 picked it up from the files alone, re-verified the archive and closed out the task. Some models needed me to steer now and then, but it was minimal.

What broke

Different models broke in different ways. The instructions had fail-safes built in, though, and when a model started to drift I steered it straight back to the pickup ledger. Because the ledger was the single source of truth, getting back on track was one instruction away.

The local model

I ran Gemma 4 (gemma4:26b) on a local Ollama instance, served on localhost through my own CLI and wired into VS Code’s chat panel in place of the GitHub Copilot models. It worked great until its 32K context window filled; then it started to hallucinate. That taught me something useful: a small-context model shouldn’t read whole files. Next I’m writing a script that uses the ledgers as living memory. Each entry already leaves pointers for the next model, so a model can load only what the next step needs instead of filling its context with things it doesn’t. I’ll explore that in later iterations of the loop and ledger pickup.

My role

My job was to set the rules the models had to prove against. I shaped the initial definitions, acceptance criteria and definitions of done, and every claim had to trace from evidence to an acceptance criterion to a definition of done. If a claim checked out against that chain and passed the model’s own audit and refinement loops, it didn’t need me. I stepped in when something only the owner could decide came up. The clearest example was the evaluation: the public scans had no labeled defects, so the results fell short of the bar I’d set, and I chose to disclose that rather than bend the data to pass.

Next time

On the next project I’ll write the full plan myself up front, with acceptance criteria and definitions of done at every level. I’ll back the build with deeper research run through an IEEE-style peer-review process and write an IEEE research paper to support the project’s objective. I’ll also set up the build repo, and the production dist/zip as its own worktree, from the start, before publishing, instead of carving a repo out of a finished zip afterwards. The portfolio side has already started with this site.