Building an AI test case generator that QA engineers actually trust

The naive version of QALabs took an afternoon: fetch a URL, dump the HTML into a model, ask for test cases. It produced fifty cases that read beautifully and tested nothing. This is the rewrite.

Why the naive version fails

A single page's HTML tells you what elements exist. It does not tell you:

• which of those elements are reachable from where,
• what state the app must be in first,
• which flows actually carry business risk.

Test cases written without that context describe the DOM, not the product.

The architecture

The pipeline that works has four stages, and only one of them is a model call:

1. Crawl — Playwright walks the app, records the reachable route graph, and captures the accessibility tree rather than raw HTML. The a11y tree is already a semantic summary: roles, names, states.
2. Reduce — collapse the graph into flows. A flow is a path from an entry point to a state change.
3. Generate — for each flow, ask the model for cases with an explicit risk rating and preconditions.
4. Verify — a second pass checks every generated case against the captured tree. A case referencing a selector that was never observed gets dropped.

That last stage is the one that buys trust. It is also the one every demo skips.

What I would do differently

Start with the accessibility tree, not the HTML. I lost two weeks to token budgets before I made that change.