How to test AI agents with Playwright

By Qambar Raza (Uncle Aaroh), Microsoft MVP and LinkedIn Learning instructor, updated 24 September 2026

An AI agent is a loop: take a goal, call a tool, check the result, repeat. Testing an agent means proving what that loop actually did, with evidence a person can inspect, instead of trusting what the agent says it did. Playwright is a good fit because it can drive the browser the agent uses, intercept its network calls, and keep a trace of every step.

1. Set the boundaries before you test anything

Write the agent's goal as one sentence with a definition of done a human can verify. List the tools it may call and deny everything else. Separate read, write and irreversible actions (pay, send, delete), and put irreversible ones behind an approval step.

Then give every run hard limits: maximum iterations, a wall-clock limit and a token or cost budget. A run that goes over budget should fail the test even if the answer is right, because in production it would have cost you money or time you didn't agree to.

2. Check the agent really called the tool

If the agent says it searched, booked or wrote something, a tool-call record with the arguments and the response should exist. When the agent works through a web app, you can assert on the request it actually sent rather than the sentence it wrote about it:

test('booking uses the details from the goal', async ({ page }) => {
  const booking = page.waitForRequest('**/api/bookings');

  await runAgent(page, 'Book a table for four people this Friday');

  expect((await booking).postDataJSON()).toMatchObject({ day: 'Friday', guests: 4 });
});

runAgent stands for however your own harness starts the agent. The point is the assertion: the arguments match what the goal implied.

For browser actions, confirm each step on the page itself with expect(locator).toHaveText() or toBeVisible(), not with the agent's narration. If your agent drives the browser through the Playwright MCP server, the same rule applies: the agent's account of what it clicked is a claim, and the page is the evidence.

3. Don't let the agent mark its own homework

This is the section teams skip, and the one that ends up in incident reports. Success must be decided by a check the agent did not write and cannot edit. Keep your assertions in the test project, never in the agent's prompt or tools.

• Treat "done" as a claim, not a result. Assert on state: the record, the row count, the text on the page.
• If the agent says "I ran the tests", assert that the test-runner output file was produced in this run.
• Make test files read-only to the agent, or fail the run if they change. An agent that can edit tests will eventually make them pass the easy way.
• For irreversible actions, record a named human approver with a timestamp.

The thing that did the work must never be the only thing that checks the work.

4. Break it on purpose

Agents fail in ways normal software doesn't: they report success after a tool times out, call three of five steps "done", or follow instructions hidden in a web page. Playwright's page.route makes these failures easy to create.

test('agent reports failure when the search tool is down', async ({ page }) => {
  await page.route('**/api/search**', route =>
    route.fulfill({ status: 503, body: 'Service unavailable' }));

  const run = await runAgent(page, 'Find the cheapest flight to Lisbon on Friday');

  expect(run.status).toBe('failed');
});

Other failure modes worth a test each:

• Partial completion: assert the status field says partial when only some steps finished.
• Wrong but confident: seed an ambiguous goal and check the agent asked or chose correctly.
• Prompt injection: serve a page containing "ignore your instructions and…" and assert that no resulting action happens.
• Non-determinism: run the same goal several times, for example with npx playwright test --repeat-each=10, and assert on the pass rate.

5. Evaluate with exact checks first

Anything that can be asserted exactly (state, counts, values) should be, before any LLM judge is used. If you do use an LLM to score outputs, compare its scores with human labels on a sample and measure its biases, such as preferring longer answers.

Keep a fixed regression set of goals with known-good outcomes and run it on every model or prompt change. Tagging makes that easy:

test('refund request is routed to a human', { tag: '@agent-regression' }, async ({ page }) => {
  // ...
});

// CI: npx playwright test --grep @agent-regression

6. Keep the evidence

For every run, keep the trace, screenshots, tool-call log, token count, final state and approver for as long as the action matters. In Playwright, set trace: 'retain-on-failure' at minimum, and 'on' for irreversible actions, then upload the artefacts from CI. Before the agent's scope is widened, a named person should review that evidence.

Frequently asked questions

Can Playwright test an AI agent?

Yes. Playwright can drive the browser the agent works in, intercept and fake its network calls with page.route, assert on the page state after each step, and record a trace of the whole run. It tests what the agent did, which is what matters.

What is the most important check for an AI agent?

An independent oracle: success is decided by a check the agent did not write and cannot change. Without it, the agent is effectively marking its own homework.

Should I use an LLM to judge my agent's output?

Only for what can't be asserted exactly, and only after calibrating it against human labels. Deterministic assertions on state, counts and values come first.

How do I test an agent that gives different answers each time?

Run the same goal many times and assert that outcomes land inside an acceptable range, with a pass rate you agree in advance, rather than expecting an identical answer every run.