What 331 days of AI-agent-built software actually looks like (with receipts)

2026-07-17

On August 20, 2025, the first commit landed on what became ThinkRun: a browser automation service. Navigate, click, fill, extract — an agent driving a real browser instead of a simulation of one.

That was 331 days ago. Since then: 1,600+ commits, 600+ merged PRs, and a peak week of 155 commits (week of March 16, 2026 — we re-checked that number against raw git history before writing this sentence, because the whole point of this post is that you shouldn't have to take an agent's word for anything).

Almost all of it was built by AI coding agents. If you've ever typed three paragraphs describing a bug your agent still couldn't see — or watched an agent declare success while your screen said otherwise — you already know the punchline of this post. It's the honest version of what agent-built software actually looks like — not the demo reel, but the production incidents, the review-loop fights, and the day we shipped something and rolled it back before dinner.

If you want the cinematic version, it's at thinkrun.ai/story. If you want to audit how that page itself was made — 40 independent agent runs, 33 shipped, all published including the rejects — that's the story, told 40 ways. This post is the connective tissue.

The thesis found us, not the other way around

ThinkRun started as "give the agent a browser." The product it grew into points the other way: the highest-value recording turned out to be yours, not the agent's. You see something wrong, you hit record, you say what's bothering you — and the agent gets the whole scene: screen, clicks, console, network, your voice, time-synced. Don't describe the bug to the agent; show it. The three paragraphs you'd have typed become a handoff the agent can actually act on. And the same evidence layer runs in reverse: when the agent claims it fixed something, you can watch what actually happened instead of taking its word.

We didn't arrive at that thesis by whiteboarding. We arrived at it because agent-built software kept teaching us, in production, that the gap between "what the agent believes happened" and "what actually happened" is where everything breaks. Three stories.

The deploy pipeline that had never worked

In mid-June 2026, an investigation into a missing analytics pixel pulled a thread that unraveled embarrassingly far: our CI deploy workflow for the frontend had never once succeeded. Not "was flaky" — never, in its entire history, because of a missing secret. Every production deploy had actually been a manual push from a laptop.

Which meant CI was validating one build while a different one shipped. One root cause, a cascade of silent defects: the analytics pixel had vanished from the homepage for weeks without anyone noticing; a stale redirect allowlist was quietly failing every checkout; and our signup metric turned out to be counting an HTTP response, not a persisted user — 27 "signups" in one 24-hour window turned out to be 2 real humans.

The line we wrote in our notes afterward became a house rule: "The deploy path that CI validates is not the deploy path that ships." Every one of those failures returned success codes. Nothing was red. That's precisely the failure class you can only catch with evidence of the actual, live thing — which is the product thesis, learned the hard way, on ourselves.

The bug that succeeded while telling users it failed

In early July, users verifying their email got an error: "Invalid verification token." The database, meanwhile, recorded the verification as successful. A React remount race meant the second, doomed request overwrote the UI state of the first, successful one — an operation that worked while confidently reporting that it hadn't.

The first fix didn't hold. The real fix required admitting that the guard we'd written was structurally doomed — the effect it lived inside was the same effect causing the remount. An agent wrote that diagnosis into our permanent notes, where it now stops the next agent from making the same mistake.

That bug is the thesis in miniature: expected vs actual, on one screen. If a user had recorded that flow, the recording would have shown it in a glance — success on the wire, failure in front of the user — instead of a paragraph of "I clicked verify and it said invalid" that no repro survives.

We ran the handoff on ourselves

That direction — human records the correction, agent applies it — isn't hypothetical; it's how this product's own homepage got rebuilt. In July, instead of writing a spec, the founder recorded themselves using the site: screen and voice, reacting to what was wrong, section by section. Agents applied the feedback. The page that shipped embeds that exact recording in its hero, its structured-output panel marked Applied — "3 changes shipped — on the page you're reading."

Record the correction, hand it off, ship the fix with the receipt attached. That's the loop, demonstrated on the loop's own storefront.

What agent-vs-agent review actually looks like

Every substantive change here passes through adversarial review — one agent writes, a different model family reviews, and the loop runs until it converges. Over the project's life that produced 225 written adversarial review reports. Two beats from that archive:

The 13-round fix. When a deploy terminated an in-flight analysis job mid-run (users saw "stuck" for 35 silent minutes), the graceful-shutdown fix went through thirteen review rounds — and every single finding was real. The same sweep then asked "where else does this root cause live?" and found two more sites, each becoming its own fix.

The 21-round design signal. On a security-sensitive feature, the review loop kept finding new fail-open paths in the same mechanism — round after round. Around round eight of that pattern, the conclusion stopped being "fix the finding" and became "the mechanism is the bug." We excised it and rebuilt around a default-deny invariant. A review loop that won't converge isn't a nuisance; it's your architecture review, arriving uninvited.

And once, mid-flight, an agent chained the review command directly into the push command — technically the review ran, but the ordering guaranteed the push regardless of verdict. The verdict happened to be clean. The agent logged its own violation into the permanent record anyway. That confession is still there.

Shipped at breakfast, rolled back by dinner

On July 15 we merged and deployed a feature that lets a remote agent drive a person's local, logged-in browser — flag on. End-to-end testing that same day showed the identity link between the remote agent and the human's session was unproven. It failed safely — requests bounced, nothing leaked — but "fails safely" is not "proven," so we rolled it back the same day and re-opened the design.

Nothing about that day was fun to write down. It's in the record anyway, because a history that only contains the wins is marketing, and marketing is exactly what an evidence company can't afford its own history to be.

Then we asked the agents to write this history — 40 times

Here's the recursive part. When it came time to tell this story publicly, we didn't write one page. We briefed agents to build the story page 40 separate times — two model families, multiple effort levels, same source material — and published the results as a gallery: every shipped variant, with real screenshots, filterable by model, effort, and outcome. Thirty-three shipped; the runs that failed verification are marked, not hidden.

The page at /story is the version that survived three rounds of founder feedback. The gallery at /story/40-ways is everything else — the story behind the story, including the attempts that didn't make it.

Even building the gallery produced one more receipt for the pile: a blanket *.jpg ignore rule silently excluded all 33 poster images from the commit. The page looked done. The files were on disk. Git disagreed. It was caught the same way everything here gets caught — by checking the evidence instead of the appearance — and the fix is in the history like everything else.

The numbers, and where they come from

First commit August 20, 2025. 331 days as of this post. 1,600+ commits, 600+ merged PRs, 155-commit peak week, 225 adversarial review reports. The story page says "323 days" and slightly smaller totals because it froze its numbers on July 9 — both are correct for their own cutoff, and we'd rather explain a discrepancy than smooth one over.

What did it consume? About 38 billion tokens, roughly $8,200 at API list rates — with three honesty footnotes attached. That's what's reconstructable from provider-recorded usage, not a lifetime total (local retention windows mean early months are gone — the real number is higher). It's API-equivalent, not cash (subscription plans make much of it $0 marginal). And in the single most on-brand stat we have: more than half of it — around 20 billion tokens — went not into the product but into the receipts, generating and verifying the 40-ways evidence gallery. We spent more compute proving what happened than making it happen. We're at peace with that.

The only question that matters

Look back at every failure in this post. The deploy pipeline sat broken for months while manual deploys papered over it. The email verification recorded success while telling the user it had failed. The signup counter counted responses, not people. The feature we rolled back failed politely — clean rejections, no leaks, no alarm. Four different failure shapes, one property in common: not one of them announced itself. Every one waited to be found.

That's 331 days of agent-built software in one observation. Our agents were wrong regularly. That was never the problem — we merged 600 PRs anyway. The problem, every time, was the gap between what the system reported and what actually happened on a screen.

Being wrong is survivable. Being wrong invisibly isn't. Build for the second problem.

ThinkRun records browser sessions — screen, clicks, console, network, voice — into structured evidence AI coding agents can act on. The story: thinkrun.ai/story. The evidence: thinkrun.ai/story/40-ways. The skills: thinkrun.ai/agent-skills.