Letting an AI agent drive Playwright — what actually held up

Most "AI + Playwright" content stops at "the model writes the script for you." That part is the easy 80%. The interesting part is what happens when you let the agent drive the browser itself — click, read the accessibility tree, decide the next action — instead of handing it a finished script to run.

Script generation vs. an agent loop

Two different jobs get called "AI testing":

1. Generation — describe a flow in English, get a Playwright script back. You review it, commit it, and it runs deterministically forever after.
2. Agentic execution — the model is in the loop while the test runs, observing the page and deciding what to click next.

Generation is the one worth shipping to CI. Agentic execution is the one worth using to discover what to generate.

Where the agent loop earns its keep

Pointed at a staging app with nothing but "find every way a user can reach checkout," an agent exploring live — reading ARIA roles, trying nav items, backtracking on dead ends — found four entry points the existing suite never covered, including one buried behind a feature flag the team had forgotten was still on for 10% of traffic. A fixed script only tests the paths someone already thought to write down.

Where it falls over

Agentic runs are slow and non-deterministic — the same prompt can take a different path through the app on two different runs, which is exactly what you don't want from a regression gate. And the failure mode is worse than a normal flaky test: the agent doesn't fail loudly, it rationalizes — clicks something adjacent to the broken control and calls the flow successful.

So the loop I landed on:

1. Agent explores and finds a flow.
2. Every action it took gets recorded as a literal Playwright trace.
3. That trace is replayed and turned into a plain, boring, deterministic .spec.ts — no model in the loop at test time.
4. The generated script goes through the same review a human-written one would.

The agent's job is reconnaissance, not the test itself. Anything that runs in CI on every PR should have zero opinions at runtime.

The failure that changed the design

Early on I let the agent-generated script commit straight to the suite. One of them asserted on page.getByText("Success") — which matched a toast from an unrelated background job that happened to fire during the run. It passed for three weeks before anyone noticed it wasn't testing what its name said. The fix wasn't a smarter model; it was refusing to trust an assertion the agent wrote about its own success without a second, unrelated pass checking it against the captured accessibility tree — the same verify stage QALabs already used for generated test cases, just applied one level lower.