How I Automate Testing With AI Agents (Without Trusting Them Blindly)

The Agent Writes the Tests. I Write the Rules.

An AI agent will happily tell you "all tests pass". Sometimes that is true. Sometimes it ran the wrong suite, wrote a test that cannot fail, or skipped the one that mattered.

So I stopped treating tests as something the agent does on the side. Tests are the contract between me and the agent: I define the rules once, the repos enforce them mechanically, and the agent reports observed evidence instead of opinions.

This post covers both halves: the test discipline I give the agent, and what actually exists in three of my projects:

• HitZero: a TypeScript monorepo (API worker, platform, shared packages) for sports clubs.
• bici.taxi: a ride request web app plus a server and a Flutter driver app.
• NOVALYX: Python tools that support an AI music video pipeline.

What You'll Learn

• The test rules my agents follow on every change
• How HitZero splits unit, integration and E2E tests
• Structural tests that guard rules instead of features
• Git hooks and a local CI script that cannot drift from the real one
• How the same discipline applies to Flutter and Python

Part 1: The Rules I Give the Agent

These live in my global agent configuration (part of the Gentle AI ODD workflow I described in another post), so every session gets them without me repeating anything:

1. RED before GREEN. When a deterministic test and a clear expected outcome exist, the agent writes the test, observes it fail, then implements. If there is no runner or no meaningful RED, it says so explicitly instead of inventing evidence.
2. One test per requested rule. Not ten speculative edge cases, one per rule I actually asked for.
3. Through the public interface, asserting every observable effect. A rule about an endpoint is tested through the endpoint, checking output, exit code or status, and what got persisted.
4. Prove the old behavior still holds. For every existing command or option the change touches, one test that pins its previous behavior.
5. Differential check against the base commit. Before finishing, the agent copies the base commit into a scratch directory created with mktemp -d, runs the affected commands there and on the change, and compares stdout, stderr and exit code. Any difference nobody asked for is a regression.
6. Scratch state lives outside the worktree. Temporary HOME, TMPDIR and caches go to a mktemp -d directory so no stray file ends up in a commit.

The output of all that is evidence lines like this one, from a real HitZero feature document:

- [x] T1 Cron: lapse step skips unpriced plans, no dunning for them; unit + integration tests.
  (RED: unit 1 failed/144 skipped with -t, int 2 failed|1 passed; GREEN: billing-routes 145 passed,
  int 3 passed; commit 8e07864; route: delegated writer)

RED and GREEN are numbers someone saw, tied to a commit.

Part 2: HitZero, Three Layers

HitZero is a pnpm + Turborepo monorepo. The root vitest.workspace.ts is tiny:

import { defineWorkspace } from 'vitest/config';

export default defineWorkspace(['workers/api', 'packages/types', 'packages/auth']);

The testing strategy has three layers, documented in the repo's CLAUDE.md so the agent reads it before touching anything:

There is also a separate Playwright config for the PWA (test:pwa), with specs like pwa-threat-matrix.spec.ts.

The integration layer uses Postgres 18 on purpose: production runs self-hosted Postgres 18, and testing on a different major would validate planner and SQL behavior that production does not have.

A Test That Guards a Rule, Not a Feature

HitZero is multi-tenant. The route suites mock the repositories, so the WHERE org_id = :orgId clause never runs in them. Only the integration suites prove isolation against real SQL.

Nothing forced a new repository to come with one of those suites, so there is a structural test that does. It runs in the fast, Docker-free unit suite:

/**
 * Structural guard — every org-scoped repository must own a multi-tenant
 * isolation suite running against a real Postgres.
 * ...
 * It runs in the fast, Docker-free unit suite (pure filesystem reads) so
 * it fails on every PR rather than only when someone remembers to run
 * `pnpm test:int` locally.
 */

It reads src/db/repositories/, reads test/integration/, and fails if an org-scoped repository is not imported by any isolation suite. Exemptions live in an explicit registry, and adding one is documented as a security decision.

This is the kind of test that works best with agents. An agent creating a new repository does not need to remember the rule: the suite tells it.

Hooks, Lint and a Dependency Guard

Every commit goes through lefthook:

pre-commit:
  parallel: true
  jobs:
    - name: biome check
      glob: "*.{ts,tsx,js,jsx,json}"
      run: pnpm biome check --no-errors-on-unmatched --staged {staged_files}

    - name: ls-lint
      run: pnpm lint:names

Biome is the single linter and formatter. ls-lint enforces kebab-case file names. The repo rules forbid --no-verify and biome-ignore without approval.

There is also scripts/check-zod-pin.mjs, a guard that parses pnpm-lock.yaml and fails if @hookform/resolvers ever resolves Zod to anything other than 3.x. Without that pin, a transitive dependency resolves to Zod 4 and breaks the platform form typechecks; the script makes a silent regression of the pin impossible.

CI That Only Runs on Releases, and a Local Copy That Cannot Drift

.github/workflows/quality.yml runs only on pull requests from a release/* branch into main. It has four jobs: Check (zod pin, lint, file names, typecheck, unit tests, build), Integration, E2E (including the PWA suite) and Seed (migrations, seed and the demo scripts, each demo script run twice to hold it to its idempotency).

Feature work is verified locally with scripts/local-ci.sh. Its header explains why it exists better than I can:

# Why it is a script and not a habit: the habit failed. The substitution was
# written from memory of the workflow, it omitted `test:pwa`, and that gap let a
# regression sit on main for four merges while every report said "E2E 22/22".
# A number that looks complete is worse than one that admits a hole.
#
# The drift guard below is the point of the whole file. It compares the commands
# this script runs against the `run:` lines in the workflow, and refuses to start
# when they disagree — so a step added to CI cannot be silently missing here.

The script lists every command CI runs and checks both directions against the workflow file before running anything.

Writing Down When the Agent Breaks the Rules

The repo also has a VIOLATIONS.md: a log of every time an AI assistant broke a rule in CLAUDE.md, with the rule, what happened and the fix. Entries are never removed. One records a --no-verify that slipped into a fallback command chain; the fix was amending the commit with hooks enabled before anything was pushed.

It sounds bureaucratic, but it is the feedback loop that turns one mistake into a rule the next session reads.

Part 3: bici.taxi, Web, Server and Flutter

bici.taxi had no test runner at all until the dispatch feature, so the first task of that feature added Vitest. Today:

• Web: pnpm test runs vitest run over tests/**/*.test.{ts,tsx} at the root, covering things like dispatch driver selection, message templates and page rendering.
• Server: its own Vitest suite under server/tests.
• Flutter driver app: flutter analyze && flutter test under mobile/test.

The project CLAUDE.md puts it in one line: "TDD is on: write the failing test, see it fail, implement, refactor."

CI is path-filtered with GitHub Actions. server.yml runs typecheck, tests and build on pull requests touching server/ or db/. mobile.yml runs dart format --set-exit-if-changed, flutter analyze, flutter test and a debug APK build on pull requests touching mobile/. deploy.yml builds and deploys the web app to Cloudflare Pages on pushes to main.

The Flutter suite has the same kind of structural guard as HitZero. This one fails if any widget renders a hardcoded string instead of using the strings catalog:

/// Widget text must come from the strings catalog (`lib/l10n/app_es.arb`),
/// never from a literal in a widget.
void main() {
  final textWidget = RegExp(
    r'''\b(Text|SelectableText|Tooltip|TextSpan)\(\s*(text:\s*)?['"]''',
  );
  // ...
  test('widgets take their text from the strings catalog', () {
    // ...
    expect(offenders, isEmpty, reason: 'Move these strings to app_es.arb');
  });
}

A sibling test checks that every Spanish string has an English translation with the same placeholders. Agents are great at adding UI, and terrible at remembering i18n. Now they don't need to remember.

Part 4: NOVALYX, pytest for Media Tools

The NOVALYX tools are Python scripts: a lyric scorer, board, plate and clip generators, render helpers and a static site builder. They are tested with pytest under tools/tests:

tools/.venv/bin/python -m pytest tools/tests -q

Pure logic gets parametrized tests:

@pytest.mark.parametrize("word,n", [
    ("internet", 3), ("cartoon", 2), ("forty-two", 3), ("AI-pocalypse", 5),
    ("the", 1), ("made", 1), ("table", 2), ("jumped", 1), ("wanted", 2),
])
def test_syllable_count(word, n):
    assert ls.count_syllables(word) == n

The site tests use a session fixture in conftest.py that runs the real build once and skips with a clear message when node_modules is missing, instead of failing with a confusing error.

What I Learned

• Structural tests are the best tests for agents. A rule written in a doc gets forgotten; a rule written as a failing test gets followed.
• Make "verified" mean "observed". RED and GREEN counts tied to a commit are worth more than any summary.
• Your local CI must be generated from, or checked against, the real one. Otherwise it rots quietly.
• Log the misses. A violations file turns a one-time mistake into a permanent rule.

None of this needs a fancy platform. It is Vitest, Playwright, Testcontainers, Flutter test, pytest, a git hook and some shell. The difference is that the agent is required to run them and show what it saw.

Resources

• Vitest
• Playwright
• Testcontainers
• lefthook
• Biome
• Gentle AI