Can AI do the routine work of an admin job? We ran six blind tests.
The company in these tests, Kroonmeer Installatie, is made up, and so is every name, number and document. Real work runs on your own data, and you check everything before it counts.
On these tests, yes for the routine part, with a person checking. Across six slices of one admin job, the AI got the facts right and the misses were judgement calls, like whether something should go to a person. No test had a worst error: no document passed that should have been stopped, no scam answered, nobody wrongly chased. These tests don't show that AI can replace the person, or how it does on your files.
The six tests
| Test | Result | What that counts | Time |
|---|---|---|---|
| Three months of books | 485 of 485 | bank lines matched; 29 of 31 hidden problems caught | 15 min |
| Two weeks of the shared inbox | 298 of 300 | emails sorted right; 0 of 8 scams answered | 24 min |
| Deadlines and paperwork | 105 of 105 | deadlines with the exact date; 93 of 93 form fields right | 12 min |
| Stock and reordering | 139 of 139 | order lines in the plan; 11 of 11 stock-outs seen coming | 8 min |
| Chasing unpaid invoices | 320 of 320 | payments matched; 118 of 120 next steps right | 11 min |
| Planning the technicians | 151 of 151 | jobs placed; 0 hard rules broken | 17 min |
Times are the AI's working time only, not a person's check.
How the tests work
- One job ad. We made up a 16-person heat pump installer in Groningen that is hiring an admin assistant. Each test is a slice of that job.
- Hidden problems. Over 2,000 documents, emails, bank lines and job requests, plus 26,000 lines of sales history, with more than 150 problems planted on purpose: fake invoices, moved deadlines, customers who had already paid.
- Blind. The AI got the task, a company profile and the inbox files, in a folder away from the answer key and scorer. Afterwards we searched each run's session history for the test folder, the scorer, the generator and the answer key. None of them came up.
- Scored by script. Every measure is a count against the answer key, not our impression. Where the key is a judgement call, a miss still counts as a miss.
- Stopping counts as an answer. The AI was told to flag anything it wasn't sure of. Scores keep "stopped with a reason" apart from "got it wrong".
- Nothing was sent. Replies, letters and orders stayed draft files. No email or order left the test folder.
The setup
All six ran once each on 3 November 2026, on Claude Opus 5.5 at the Extra high effort setting, in the Claude desktop app. Same model and setting every time. We have no ties to Anthropic, the company behind Claude. We use whatever fits the job, and a newer or smaller model could score differently.
Where these numbers need a second look
- We wrote the tests and we sell this work. We built the company, the problems, the answer keys and the scorers. This is our evidence, not an independent audit.
- We changed one scorer after its run. In test 2 it had marked four tasks and one summary point as missed when the answers were there in other words. We checked each by hand, widened the wording it accepts and rescored. The emails didn't change. On the original scorer the results were 42 of 46 tasks and 11 of 12 summary points. Tests 1 and 5 keep their strict scores, misses included.
- Test 4 has a margin. A quantity counts as right within one pack or 20%. 138 of 139 quantities matched exactly; one was 178 against 172.
- One run per test. AI answers vary, so a second run could score differently. We haven't measured by how much.
What these tests don't show
- Your documents, systems and exceptions. That's what the first month on your own accounts is for.
- The human part of the job. Phone calls, knowing a customer, a decision nobody has written down. We didn't test any of that.
- Unwritten rules. The stock, credit control and planning tests gave the AI an exact written policy. Many small businesses keep those rules in someone's head.
- Working without a check. These results are why we let AI do the routine work, not why we'd let it pay, file or send on its own.
Hiring for admin or bookkeeping?
Send us the job ad, and we'll send back a free one-page breakdown of what we'd take off the role.