Property Testing for AI-Generated Code: Cover Inputs the Model Did Not Anticipate with Invariants

Letting the same model write both a function and three example tests often yields a self-consistent but wrong answer: the implementation treats the empty string as a missing value, and the tests never generate an empty string; pagination code skips the last page, and the tests only cover evenly-divided samples. Property testing does not enumerate expected outputs; instead it repeatedly generates inputs that satisfy domain constraints and checks invariants that must hold for any valid input—so it is especially good at exposing blind spots shared by the implementation and the examples.

It is also not "randomly run it many times." A usable gate must report the generation space, rejection ratio, run count, random seed, the minimal counterexample after shrinking, and the replay command. How to let the model help produce initial samples is covered in AI Test Generation, but invariants must come from the spec, the callers, and domain rules—they cannot be reverse-derived from the implementation under test.

Write the Failure Criteria Before the Generator

When reviewing a task, break the natural-language requirement into four classes of properties:

1. Preservation: sorting does not change the multiset of elements; serialize-then-parse preserves data.
2. Algebraic: normalization is idempotent; set merges satisfy commutativity or associativity.
3. Boundary: page numbers, lengths, amounts, and times never exceed the agreed range.
4. Relational: an optimized implementation and a simple reference implementation give the same result for the same input.

"Does not crash" is usually too weak, and "the result equals the current implementation's output" has no error-detection power. For slug normalization, you can require that normalize(normalize(x)) === normalize(x), that the result contains only allowed characters, that the maximum length is no more than 80, and that two different protected names must not collide. If the requirement is unclear about which Unicode characters should be equivalent, property testing exposes the spec hole—at which point you should clarify rather than let the agent guess.

Generators should encode the valid domain, rather than generating arbitrary values and then heavily filter-ing. An order generator should directly produce non-negative integer cents, supported currencies, and valid tax rates; when you need to test illegal inputs, build a separate generator and declare the expected error. Composite types can start from small structures, limiting array length and recursion depth to avoid spending 99% of the time on unrepresentative giant objects. TypeScript Advanced Types can constrain compile-time shapes, but runtime boundaries still need generator validation.

Pin the Environment and Preserve Every Counterexample

The team pins fast-check to an exact version in dependencies and commits the lockfile. With Node 22.18.0 and pnpm 10.14.0 as the baseline, CI runs the tests from a clean checkout and writes the test framework and property library versions into the report:

mkdir -p .artifacts
pnpm install --frozen-lockfile
node --version
pnpm --version
pnpm exec vitest --version
pnpm exec vitest run tests/property --reporter=json \
  --outputFile=.artifacts/property-tests.json

The property config makes seed, numRuns, endOnFailure, and timeout explicit. A PR fast gate can run 500 times with a frozen seed, while a nightly gate then runs 10 derived seeds 5000 times each. Failure logs must contain the property name, seed, path, original input, and shrunk result. With fast-check, replay using the same seed and path, rather than copying a console string that may lose encoding details.

After finding a counterexample, add the minimal counterexample to the ordinary regression tests and keep the property that triggered it. Saving only the example would once again shrink the explored space; relying only on random reproduction may change the generation sequence due to a library upgrade. When a counterexample contains sensitive data, save a publicly shareable minimal structure and hash, and place the original input into a controlled artifact store.

Set a Quality Gate That Can Block Merges

The property testing report should check at least: all approved properties pass; the number of valid samples reaches the budget; the filter rejection rate is below 20%; there are no timeouts or uncaught async tasks; all historical counterexamples replay successfully. High-risk code such as amounts, permissions, and parsers must also pass reference-implementation differential testing. The gate binds headSha + lockfileHash + propertyConfigHash, so any change to source or generator after testing invalidates the evidence.

If a failure occurs, the agent first classifies the counterexample as an implementation bug, a spec ambiguity, a generator out-of-bounds, or a test-oracle error. Implementation bugs can be fixed and the full seed set re-run; spec ambiguity goes to a human; generator errors may only be modified after proving the input is outside the valid domain; you must not add a filter just because a value is hard to handle. After two consecutive failed auto-fixes, stop changes and submit the minimal counterexample, call stack, and candidate root cause.

Property testing must also prevent false greens. Async properties must return or await a Promise, external services should be replaced with deterministic models or recorded replays, and time and randomness sources should be explicitly injected. Concurrency logic cannot be proven correct by "sleeping 100ms"; instead, generate event orderings and check state-machine invariants. The pure functions and immutable data from Functional Programming in TypeScript can significantly lower this testing cost.

Validate the Properties Themselves with Defect Samples

Before enabling the gate, inject a set of known defects into the implementation: remove a boundary comparison, swap input order, break a round-trip, truncate Unicode, apply a side effect twice. If the properties fail to catch the corresponding defect, the invariants or generation domain are still insufficient. Then combine the property layer with the unit, integration, and end-to-end layers in Agent Testing Strategy: properties cover breadth, examples cover the business narrative, and integration tests cover real boundaries.

What high-quality property testing ultimately delivers is not "random tests passed," but a piece of auditable evidence: which inputs were explored, which invariants were proven to which budget, how failures replay precisely, and why baselines need rebuilding when library versions change. This forces AI-generated code to face inputs the model did not prearrange, rather than only passing exams it wrote for itself.