Evidence Lab
Claims about AI in testing, actually tested.
Every investigation takes one claim, designs an experiment anyone can rerun, runs it, and publishes the code, the data and the verdict together. Nothing appears here before it has been run.
The format
| The claim | Stated as the people who make it would state it. |
|---|---|
| The test | A reproducible experiment with its inputs fixed in advance: prompts, versions, data. |
| The evidence | Code, logs, results and traces, downloadable. Failures and surprises included. |
| The verdict | What was demonstrated, separated from what it might mean more generally, with the limits stated. |
Investigations
01, 17 October 2026
Do AI-written Playwright tests catch bugs? It depends on what the AI was shown.
Two suites, same AI, same app. Written from the specification: 15 of 15 injected bugs caught. Written with the code in view: 7 of 15. Both were green on the working app.
Read the investigationIn the queue
Designed, not yet run. Nothing below is a result.
- Can an agent complete a task without exceeding its permissions? A bounded task, a fixed tool allow-list, adversarial page content, and a check on every tool call the agent made rather than on what it said it did.
- Does an LLM judge agree with Playwright about whether a task was done? Agent runs scored two ways: by exact end-state assertions, and by a model reading the transcript. How often do they disagree, and which way?
- Playwright MCP against a hostile page. Hidden instructions in page text and accessibility names: which agents follow them, and what in the trace gives it away?
Have a claim you want tested, or a tool you would like put through one of these? Suggest an investigation. Sponsored investigations are possible and are labelled; the method and the verdict are never for sale. How sponsorship works.