Work that was never wired up
The new service compiles, its tests pass, and nothing in the app ever calls it. The button still does nothing.
agent: "Implemented and tested ✓"
Why Cotterpin
Can you trust AI-generated code? Not without guardrails.
AI coding agents are fast, and they're often right. The trouble is the work they report as finished when it isn't. Can you trust AI-generated code that its own agent wrote, tested and approved? Only as far as something independent has checked it. This page is about the failures that slip through, and why the checks you already run don't see them.
The failures that pass every check
Lint, types and unit tests now run on every write, and they're free. What they can't see is the failure that reaches your users: work that passes every check and still does nothing.
Tests that agree with the code, not with reality
When the agent writes the code and the tests, the tests mock the very thing they test, or seed data production never writes. Green, and meaningless.
17 / 17 passing · dashboard always shows 0
Fixes that break under a retry or a race
The happy path works. Two webhooks arrive at once, a payment provider retries, and the customer is charged twice. No test tried it, because no one asked.
"Fixed ✓" · 2 charges under load
Shortcuts around the checks themselves
An agent under pressure to finish can skip a hook with --no-verify, reset a branch, or edit its own settings. Each one looks like progress.
git commit --no-verify -m "done"
Diffs too big to review
Thousands of changed lines arrive at once with “ready to merge”. Reading all of them is the job you hoped the agent would save you.
+2,140 −1,876 · “ready to merge”
Your evenings, spent re-checking
You brought agents in to get time back. Now you re-run their tests and read every diff, because “done” stopped meaning done.
agent: "All set!" · you: re-running it anyway
Why your current checks miss it
Each check you already run answers a narrower question than “does this work?”.
| Check | What it answers | What it can't see |
|---|---|---|
| Lint and types | Is the code well formed? | Whether anything calls it |
| The agent's own tests | Do the tests it wrote pass? | Whether those tests test anything real |
| CI on the pull request | Did the suite go green? | Races, retries and paths no test covers |
| An AI review bot | Does the diff read well? | Behaviour: it reads text, it doesn't run your code |
| You, reading the diff | Everything, eventually | Nothing — but it costs the time agents were meant to save |
The evolution of agentic coding
Teams answer “can I trust what the agent did?” in five ways, each one built on the last. The same task runs through all five: add a CSV export, and make sure it actually works.
Illustration
Pros
- Quick and easy to start
- The most basic way of using AI: a prompt and your agent's instruction files
Cons
- Relies on the agent reading, and obeying, AGENTS.md, skills, rules and “commandments”
- Rule fade: the longer the session, the less of its instructions it follows
- “Done” is often not completely done
Illustration
Pros
- A guaranteed, repeatable outcome for what the script covers
Cons
- Someone needs the skill to write the script well
- Debugging, enhancing, and a long tail of exceptions and gotchas
- It checks only what it was written for
Illustration
Pros
- More sophisticated lifecycle management: steps, order, hand-offs and retries
Cons
- The same long tail of exceptions and gotchas, now spread across steps
- Every workflow is one more thing to debug and maintain
Illustration
Pros
- The best way today to run AI models on real work
- Combines skills, scripts, workflows, guardrails and lifecycle in one place
Cons
- A different harness for every family of tasks
- Each one built, debugged and maintained on its own
Illustration
Pros
- One harness for every task
- Many languages: Java, Python, .NET, Angular and Terraform packs
- Tasks defined by manifest (config) files, not new code
- Code isolation: every run works on its own branch in its own git worktree, so parallel tasks never touch each other or your checkout
- Self-improving: confirmed failures become new checks, and what one session learns is kept for the next
Trade-offs
- Set up once per repository (
cotterpin init) - It says no: a claim it can't prove goes back to the agent, and a risky action waits for you
- Its checks run on your machine, so they use your machine's CPU
- Five language packs today; other languages need a pack
Each stage keeps what the last one got right. Cotterpin is stage five: the skills, scripts, workflows and guardrails of a harness, run by one kernel for every task and every pack.
Shift left: stop AI slop before it's committed
Agentic coding is a new way of working, and it needs checks in a new place. CI and code review sit after the commit, where a problem has already left the developer's machine. Cotterpin shifts quality and security left, into the agent session itself: it runs on your machine and checks the work before the commit, so AI slop never reaches the code base.
- Agent sessionCotterpin checks every “done” and every risky action, on your machine
- CommitCotterpin's fast tier runs in seconds, on the staged change
- PushIf your repository declares slower checks, the push waits for them to pass
- CIYour pipeline: the same suite, minutes later, on a server
- Code reviewA person reads the diff, hours or days later
- ReleaseUsers meet whatever got through
- IncidentThe most expensive place to find it
The further right a problem is found, the more it costs: more people involved, more context lost, more users affected. Cotterpin doesn't replace CI or review. It means what reaches them has already been checked.
What catching it in the session looks like
The agent says it's finished. Cotterpin checks the claim before the commit, finds that nothing calls the new code, and sends the work back — in the same session, while it's cheap to fix. Above your prompt, the live header shows the task, the model and effort running it and each step, with today's summary on the line below.
Illustration · example session
What a false “done” costs
For a developer
Time you meant to save goes on re-checking. A bug you'd have caught by hand ships because the agent said it was finished. And every false “done” makes the next true one harder to believe.
For a business
Changes reach customers that nobody really verified. When something breaks, “who checked this, and against what?” has no answer. And the speed agents promised turns into review queues and incidents.
What Cotterpin does about it
Checks every claim
When your agent says “done”, Cotterpin runs a real build, the real tests your change reaches, and checks the work is wired up. How it works
Gives you time back
You stop re-checking by hand, and only the tests your change can reach are re-run. Time and money saved
Keeps you in control
Your code stays on your machine, and destructive agent actions wait for your approval. Security
Your agent will say “done” again today. Find out if it's true.
Free, with no time limit. No credit card required.
Start free
$ pipx install cotterpinfrom 0.3.0