AI test automation with Playwright: what an AI testing agent does well, and where it fails
We gave an AI agent a browser and a goal instead of a script. Where agentic testing made our Playwright suite better, where it failed, and the workflow we now use for AI test automation.
AI test automation is sold as "the model writes the script for you." That part is the easy 80%. The interesting part is what happens when you let an AI testing agent drive the browser itself (clicking, reading the accessibility tree, deciding the next action) instead of handing it a finished script to run.
We have done both on real applications. This is what held up, what did not, and how we now combine AI agents with Playwright in the automation we build for clients.
What is AI test automation?
Two different jobs get called "AI testing":
- Generation. Describe a flow in English, get a Playwright script back. You review it, commit it, and it runs deterministically forever after.
- Agentic execution. The model is in the loop while the test runs, observing the page and deciding what to click next. This is what people mean by agentic testing.
Generation is the one worth shipping to CI. Agentic execution is the one worth using to discover what to generate.
Mixing the two up is the most common reason an AI testing pilot disappoints. Teams expect a regression suite and get a demo, or expect exploration and get a brittle script.
Where an AI testing agent earns its keep
Given a staging app and nothing but "find every way a user can reach checkout," an agent exploring live (reading ARIA roles, trying nav items, backtracking on dead ends) found four entry points the existing suite never covered, including one buried behind a feature flag the team had forgotten was still on for 10% of traffic. A fixed script only tests the paths someone already thought to write down.
This is exploratory testing at a speed a person cannot match. The agent is good at:
- Mapping an unfamiliar application, which is the slow first week of any QA project.
- Finding journeys nobody documented, especially behind flags and rarely used menus.
- Drafting test cases and scripts for a QA engineer to review.
Where agentic testing falls over
Agentic runs are slow and non-deterministic: the same prompt can take a different path through the app on two different runs, which is exactly what you don't want from a regression gate. And the failure mode is worse than a normal flaky test: the agent doesn't fail loudly, it rationalizes, clicking something adjacent to the broken control and calling the flow successful.
So the loop we landed on:
- Agent explores and finds a flow.
- Every action it took gets recorded as a literal Playwright trace.
- That trace is replayed and turned into a plain, boring, deterministic
.spec.ts, with no model in the loop at test time. - The generated script goes through the same review a human-written one would.
The agent's job is reconnaissance, not the test itself. Anything that runs in CI on every PR should have zero opinions at runtime.
The failure that changed the design
Early on we let the agent-generated script commit straight to the suite. One
of them asserted on page.getByText("Success"), which matched a toast from
an unrelated background job that happened to fire during the run. It
passed for three weeks before anyone noticed it wasn't testing what its name
said. The fix wasn't a smarter model; it was refusing to trust an assertion
the agent wrote about its own success without a second, unrelated pass
checking it against the captured accessibility tree. It is the same verify
stage we use when
generating test cases with AI,
just applied one level lower.
Self-healing tests: useful, with one rule
When the interface changes, an AI agent can repair the tests that broke. That is the "self-healing" promise, and it is real for the boring cases: a renamed button, a moved field, a changed label.
The rule we hold to is that healing must never hide a real bug. If a test fails because the application is wrong, the test has to stay red. A healing step that quietly weakens an assertion until it passes is worse than no automation at all, because it reports quality that is not there.
Does AI replace QA engineers?
No, and the three-week false pass above is why. An AI agent is fast at exploring and drafting. It is poor at judging whether behaviour is correct for your business. In the way we work:
- The agent explores, writes tests and writes bug reports.
- A QA engineer reviews what it produced, decides what matters and signs off.
- Your pipeline runs the finished suite with no AI involved, so results are repeatable and cost nothing per run.
How our AutoQA Agent applies this
Our AutoQA Agent is this workflow packaged. On its first run against a demo store it had never seen, it explored the application, wrote 25 Playwright tests and found 4 real bugs in about 18 minutes. Every failure it reported was a real application bug, not a test mistake, and the suite it left behind re-runs in about 30 seconds with no AI.
The output is a normal Playwright project in a repository: page objects, tests, a test plan and bug reports with steps to reproduce. It runs in GitHub Actions, Jenkins or Azure DevOps like any other suite.
AI test automation alongside Cypress and Selenium
The agent loop is not tied to one framework. The exploring and reviewing steps are the same whether the finished scripts are Playwright, Cypress or Selenium. We default to Playwright for new work because its traces and accessibility snapshots give an agent the clearest view of a page, and recent releases keep adding to that. Our notes on Playwright 1.64 cover the latest of them.
Frequently asked questions
- What is AI test automation?
- AI test automation uses an AI model to help create or run software tests. It covers two different things: generating test scripts from a description, and agentic testing, where an AI agent drives the browser itself and decides what to do next.
- What is agentic testing?
- Agentic testing is when an AI agent explores an application in a real browser, reads the page and chooses its next action instead of following a fixed script. It is strong at discovering journeys and weak as a regression gate, because two runs can take different paths.
- Can AI write Playwright tests?
- Yes. An AI agent can explore an application and produce working Playwright tests with page objects. The tests still need review by a QA engineer, because an agent can write an assertion that passes for the wrong reason.
- Will AI replace QA engineers?
- No. AI is fast at exploring an application and drafting tests, but it cannot judge whether behaviour is right for the business. The reliable setup is an agent that drafts, a QA engineer who reviews and signs off, and a pipeline that runs the finished tests without AI.
- Are self-healing tests reliable?
- They are reliable for interface changes such as a renamed button or a moved field. They are only safe if healing refuses to hide real bugs, so a test that fails because the application is wrong must stay failing.
Want to see an AI testing agent on your own application? Our free test run puts the AutoQA Agent on your staging site and gives you the results in 48 hours, reviewed by our team.