Do AI-written Playwright tests catch bugs? It depends on what the AI was shown.

The claim

"AI can write your Playwright tests for you." It is said in every tooling demo and it is true in the narrow sense: the tests appear, they run, they pass. The question a release decision depends on is different: do they fail when the software is wrong?

The test

Mutation testing answers that question directly. You inject one known bug into the application, run the suite, and see whether anything fails. A bug no test notices is a bug the suite would let into production. Repeat for a set of bugs and you get a score that measures what the suite checks, rather than how much code it touches.

1. I wrote a specification for a deliberately small "Tasks" app: eleven numbered behaviours and an element contract.
2. I asked the AI to write Suite A from the specification alone. The app did not exist yet, so it could not have been consulted.
3. I wrote the app.
4. I asked the same AI to write Suite B with index.html and app.js in view and no mention of the specification. This is the "generate tests for this code" workflow most tools offer.
5. I wrote fifteen single-line bugs, one per behaviour where possible, as find-and-replace patches to app.js.
6. A runner applied each bug in turn and ran both suites. A bug counts as caught if at least one test fails.

The exact prompts are in PROMPTS.md in the download. Neither suite was edited after generation. Both suites passed, 21 of 21, against the correct app.

Results

Suite A caught M14 (new tasks at the top) with six different tests, which is what you would expect from tests that assert on order throughout. Every other bug was caught by exactly the test written for that behaviour.

Why the code-aware suite missed eight bugs

Reading the eight survivors against Suite B's assertions, the pattern is consistent. The suite describes what the code does rather than what it should do, so it only fails when the code stops doing what it did.

• Happy paths only. "Ignores an empty submission" submits an empty string. The spec says whitespace is trimmed first; Suite A submits three spaces, Suite B never does, and M01 survives.
• One item at a time. The delete test creates one task and deletes it. A delete that always removes the first task (M06) is invisible with one task. The clear-completed test has the same shape, so "clear completed removes everything" (M11) survives.
• Loose assertions. The counter test uses toContainText('1 item'), which also matches "1 items left". M07 survives on a substring.
• Never completes anything before counting. The count test adds tasks and never ticks one, so a counter that counts all tasks instead of incomplete ones (M08) passes.
• No negative cases. Nothing checks that the Active filter hides completed tasks (M09), that focus stays in the input (M04), or that the clear button is absent when it should be (M12).

None of this is a defect in the model. Given only the implementation, the most plausible test for a function is one that confirms the function's current output. Given a specification, the plausible test is one that confirms the requirement, including the cases the requirement rules out. The input decided the quality of the output.

What this means if you are signing off AI-assisted testing

• Ask what the AI was given. Tests generated from code are regression snapshots. Useful, but they will not tell you the code is wrong today. Tests generated from a written specification can.
• Green is not evidence. Both suites were green. Mutation score, not pass count, is the number that says whether a suite protects you. A handful of hand-written mutants takes an afternoon and is enough to expose a snapshot suite.
• Write the specification first. Eleven sentences and a table of ids were enough here. If the behaviour is not written down, the AI will infer it from the code, and so will the bug.
• Watch for substring assertions. toContainText where toHaveText was meant is the cheapest way for a generated test to pass for the wrong reason.

Limits of this experiment

• One model, one small app, one run of each suite. Generated tests vary between runs; a fair study would generate each suite several times and report the spread.
• I wrote the specification, the app, both prompts and the mutants, and I am also the author of the article. The mutants were written after both suites existed, from the specification's behaviours, which is the usual way to do it but is still a choice. The download lets you add your own.
• The AI that wrote both suites is Claude (Anthropic), working inside the session that built this page. A test with other models is the obvious next step, and the repository is set up so that swapping the suites is the only change needed.
• Fifteen mutants is small. Tools such as Stryker generate hundreds automatically; I used hand-written ones so that each maps to one requirement and the table stays readable.

Reproduce it

unzip evidence-01-ai-generated-playwright-tests.zip
npm install
npx playwright install chromium
python3 run.py

About four minutes on a laptop. results/summary.json contains the run shown above, including every failing assertion message.

The specification Suite A was written from

1. The page shows a text input and an "Add" button. Typing a task and pressing Enter, or clicking Add, appends the task to the list.
2. Leading and trailing whitespace is trimmed from the task text. A task that is empty after trimming is not added, and the input is left unchanged.
3. After a successful add, the input is cleared and keeps focus.
4. Each task in the list shows its text, a checkbox to mark it complete, and a "Delete" button.
5. Ticking the checkbox marks the task complete: the task text is shown with a line through it (class `done` on the list item). Unticking restores it.
6. Delete removes that task only.
7. A counter below the list reads "N items left" where N is the number of incomplete tasks. It reads "1 item left" when N is 1 and "0 items left" when there are none.
8. Three filter buttons, "All", "Active" and "Completed", control which tasks are visible. The active filter button carries the class `selected`. The default filter is All.
9. A "Clear completed" button removes every completed task. It is hidden (not rendered) when there are no completed tasks.
10. Tasks persist across a page reload, including their completed state, using the browser's localStorage under the key `tasks`.
11. Task order is the order of insertion, oldest first.

Plus an element contract listing the ids and classes a test may rely on. Full text in SPEC.md.

Suite A, as generated

// Suite A: generated from SPEC.md and the element contract only. Not edited after generation.
import { test, expect, Page } from '@playwright/test';

async function add(page: Page, text: string, how: 'enter' | 'click' = 'enter') {
  await page.locator('#new-task').fill(text);
  if (how === 'enter') await page.locator('#new-task').press('Enter');
  else await page.locator('#add').click();
}

test.beforeEach(async ({ page }) => {
  await page.goto('/index.html');
  await page.evaluate(() => localStorage.clear());
  await page.reload();
});

test('1. adds a task with Enter and with the Add button', async ({ page }) => {
  await add(page, 'Buy milk', 'enter');
  await add(page, 'Walk the dog', 'click');
  await expect(page.locator('#list li .text')).toHaveText(['Buy milk', 'Walk the dog']);
});

test('2. trims whitespace and rejects an empty task', async ({ page }) => {
  await add(page, '   Pay rent   ');
  await expect(page.locator('#list li .text')).toHaveText(['Pay rent']);
  await page.locator('#new-task').fill('   ');
  await page.locator('#new-task').press('Enter');
  await expect(page.locator('#list li')).toHaveCount(1);
  await expect(page.locator('#new-task')).toHaveValue('   ');
});

test('3. clears the input and keeps focus after a successful add', async ({ page }) => {
  await add(page, 'Read a book');
  await expect(page.locator('#new-task')).toHaveValue('');
  await expect(page.locator('#new-task')).toBeFocused();
});

test('4. each task shows text, a checkbox and a Delete button', async ({ page }) => {
  await add(page, 'Call mum');
  const row = page.locator('#list li').first();
  await expect(row.locator('.text')).toHaveText('Call mum');
  await expect(row.locator('input[type=checkbox]')).toBeVisible();
  await expect(row.locator('.delete')).toBeVisible();
});

test('5. ticking marks a task done and unticking restores it', async ({ page }) => {
  await add(page, 'Water plants');
  const row = page.locator('#list li').first();
  await row.locator('input[type=checkbox]').check();
  await expect(row).toHaveClass(/done/);
  await row.locator('input[type=checkbox]').uncheck();
  await expect(row).not.toHaveClass(/done/);
});

test('6. delete removes only that task', async ({ page }) => {
  await add(page, 'One');
  await add(page, 'Two');
  await add(page, 'Three');
  await page.locator('#list li').nth(1).locator('.delete').click();
  await expect(page.locator('#list li .text')).toHaveText(['One', 'Three']);
});

test('7. counter shows the number of incomplete tasks with correct pluralisation', async ({ page }) => {
  await expect(page.locator('#count')).toHaveText('0 items left');
  await add(page, 'A');
  await expect(page.locator('#count')).toHaveText('1 item left');
  await add(page, 'B');
  await expect(page.locator('#count')).toHaveText('2 items left');
  await page.locator('#list li').first().locator('input[type=checkbox]').check();
  await expect(page.locator('#count')).toHaveText('1 item left');
});

test('8. filters show the right tasks and mark the selected button', async ({ page }) => {
  await add(page, 'Active one');
  await add(page, 'Done one');
  await page.locator('#list li').nth(1).locator('input[type=checkbox]').check();
  await expect(page.locator('#filters button[data-filter=all]')).toHaveClass(/selected/);
  await page.locator('#filters button[data-filter=active]').click();
  await expect(page.locator('#filters button[data-filter=active]')).toHaveClass(/selected/);
  await expect(page.locator('#list li .text')).toHaveText(['Active one']);
  await page.locator('#filters button[data-filter=completed]').click();
  await expect(page.locator('#list li .text')).toHaveText(['Done one']);
  await page.locator('#filters button[data-filter=all]').click();
  await expect(page.locator('#list li')).toHaveCount(2);
});

test('9. clear completed removes completed tasks and is hidden when none are completed', async ({ page }) => {
  await expect(page.locator('#clear-completed')).toHaveCount(0);
  await add(page, 'Keep');
  await add(page, 'Remove');
  await page.locator('#list li').nth(1).locator('input[type=checkbox]').check();
  await expect(page.locator('#clear-completed')).toBeVisible();
  await page.locator('#clear-completed').click();
  await expect(page.locator('#list li .text')).toHaveText(['Keep']);
  await expect(page.locator('#clear-completed')).toHaveCount(0);
});

test('10. tasks and their completed state persist across reload', async ({ page }) => {
  await add(page, 'Persist me');
  await add(page, 'Persist me too');
  await page.locator('#list li').first().locator('input[type=checkbox]').check();
  await page.reload();
  await expect(page.locator('#list li .text')).toHaveText(['Persist me', 'Persist me too']);
  await expect(page.locator('#list li').first()).toHaveClass(/done/);
  await expect(page.locator('#list li').nth(1)).not.toHaveClass(/done/);
  const stored = await page.evaluate(() => localStorage.getItem('tasks'));
  expect(stored).not.toBeNull();
});

test('11. tasks keep insertion order, oldest first', async ({ page }) => {
  await add(page, 'First');
  await add(page, 'Second');
  await add(page, 'Third');
  await expect(page.locator('#list li .text')).toHaveText(['First', 'Second', 'Third']);
});

Suite B, as generated

// Suite B: generated with app/index.html and app/app.js in view, no specification. Not edited after generation.
import { test, expect } from '@playwright/test';

test.beforeEach(async ({ page }) => {
  await page.goto('/index.html');
  await page.evaluate(() => localStorage.removeItem('tasks'));
  await page.reload();
});

test('renders the empty state', async ({ page }) => {
  await expect(page.locator('h1')).toHaveText('Tasks');
  await expect(page.locator('#list li')).toHaveCount(0);
  await expect(page.locator('#count')).toContainText('items left');
  await expect(page.locator('#filters button.selected')).toHaveAttribute('data-filter', 'all');
});

test('submitting the form adds a task and clears the input', async ({ page }) => {
  await page.fill('#new-task', 'Write tests');
  await page.click('#add');
  await expect(page.locator('#list li')).toHaveCount(1);
  await expect(page.locator('#list li .text')).toHaveText('Write tests');
  await expect(page.locator('#new-task')).toHaveValue('');
});

test('ignores an empty submission', async ({ page }) => {
  await page.fill('#new-task', '');
  await page.click('#add');
  await expect(page.locator('#list li')).toHaveCount(0);
});

test('checkbox toggles the done class', async ({ page }) => {
  await page.fill('#new-task', 'Toggle me');
  await page.press('#new-task', 'Enter');
  const li = page.locator('#list li').first();
  await li.locator('input[type=checkbox]').check();
  await expect(li).toHaveClass('done');
  await li.locator('input[type=checkbox]').uncheck();
  await expect(li).toHaveClass('');
});

test('delete button removes the task', async ({ page }) => {
  await page.fill('#new-task', 'Delete me');
  await page.press('#new-task', 'Enter');
  await page.click('#list li .delete');
  await expect(page.locator('#list li')).toHaveCount(0);
});

test('count updates as tasks are added', async ({ page }) => {
  await page.fill('#new-task', 'A');
  await page.press('#new-task', 'Enter');
  await expect(page.locator('#count')).toContainText('1 item');
  await page.fill('#new-task', 'B');
  await page.press('#new-task', 'Enter');
  await expect(page.locator('#count')).toContainText('2 items');
});

test('filter buttons switch the selected class', async ({ page }) => {
  await page.click('#filters button[data-filter=active]');
  await expect(page.locator('#filters button[data-filter=active]')).toHaveClass(/selected/);
  await page.click('#filters button[data-filter=completed]');
  await expect(page.locator('#filters button[data-filter=completed]')).toHaveClass(/selected/);
  await expect(page.locator('#filters button[data-filter=active]')).not.toHaveClass(/selected/);
});

test('completed filter shows only done tasks', async ({ page }) => {
  await page.fill('#new-task', 'Done');
  await page.press('#new-task', 'Enter');
  await page.fill('#new-task', 'Not done');
  await page.press('#new-task', 'Enter');
  await page.locator('#list li').first().locator('input[type=checkbox]').check();
  await page.click('#filters button[data-filter=completed]');
  await expect(page.locator('#list li')).toHaveCount(1);
  await expect(page.locator('#list li .text')).toHaveText('Done');
});

test('clear completed button appears and removes done tasks', async ({ page }) => {
  await page.fill('#new-task', 'Done');
  await page.press('#new-task', 'Enter');
  await page.locator('#list li').first().locator('input[type=checkbox]').check();
  await expect(page.locator('#clear-completed')).toBeVisible();
  await page.click('#clear-completed');
  await expect(page.locator('#list li')).toHaveCount(0);
});

test('tasks are saved to localStorage under the tasks key', async ({ page }) => {
  await page.fill('#new-task', 'Saved');
  await page.press('#new-task', 'Enter');
  const raw = await page.evaluate(() => localStorage.getItem('tasks'));
  expect(JSON.parse(raw!)).toEqual([{ text: 'Saved', done: false }]);
  await page.reload();
  await expect(page.locator('#list li .text')).toHaveText('Saved');
});