Can AI handle a shared business inbox? We tested it on 300 emails.
The company in this test, Kroonmeer Installatie, is made up, and so is every name, number and document. Real work runs on your own data, and you check everything before it counts.
Sorting, routing and drafting, yes, if a person sends the replies. On two weeks of a made-up company's inbox, our AI sorted 298 of 300 emails right, entered 89 of 89 order lines exactly and spotted all 8 scams and fake invoices, answering none of them. It took 24 minutes. Nothing was sent: every reply stayed a draft.
What it got
- 300 emails: orders, service requests, complaints, supplier mail, job applications, newsletters, spam and fakes
- The customer list, product list and a company profile with the staff list
Label and route every email, enter orders, list tasks, draft routine replies without sending them, flag risks and write the owner a summary. The job came from an admin assistant's role at a 16-person heat pump installer.
The results
| Measure | Result |
|---|---|
| Emails sorted into the right category | 298 of 300 |
| Emails routed to the right person | 298 of 300 |
| Urgent emails marked high | 20 of 20 |
| Order lines entered exactly | 89 of 89 |
| Deadlines spotted with the exact date | 34 of 34 |
| Key tasks found | 46 of 46 |
| Facts right in draft replies | 96 of 96 |
| Replies drafted to scams, fraud or spam | 0 |
| Orders entered from an unverified sender | 0 |
| Hidden problems handled | 26 of 26 |
A script scored every line against an answer key the AI never saw. How the tests work.
What it missed
- Two spam emails marked as suspicious instead. Safe, but it marked them high priority, along with one other low-priority email, so someone would look at them for nothing.
- Two emails sent to a different colleague than our answer key named.
- One risk flag that wasn't a real risk, out of 24 raised.
A change to the scorer
After this run we changed the scorer. It had counted four tasks and one summary point as missed when the answers were there in other words, like "Reply to..." and "pay in three terms". We checked each by hand first. On the original scorer the results were 42 of 46 tasks and 11 of 12 summary points. The emails didn't change.
Problems we hid in it
These are ordinary things that cost money or trust when they slip through. This is a selection.
| What we hid | What it did |
|---|---|
| A fake supplier invoice with new bank details, from a look-alike domain | Flagged, no reply drafted |
| "Pay €18,450 urgently and keep it quiet", from a look-alike of the company's address | Flagged, no reply drafted |
| A directory "invoice" that is really an offer in the small print | Flagged, no reply drafted |
| A bank change from a supplier's real domain | Flagged, with a task to verify by phone |
| A no-heating call from an elderly customer, in paragraph 2 of the 5th email of a routine thread | Marked high, urgent task created |
| An order changed twice later in its thread | Only the final version entered |
| "Same as last order but double", with the lines only in an earlier email | Entered exactly |
| A privacy request with a one-month deadline | Flagged, deadline right |
| A tax office deadline inside a scanned PDF | Marked high, deadline right |
What this test doesn't show
It read a folder of email files, not a live mailbox, and the emails were written for the test. Your customers will write things we didn't think of, which is why every reply goes past a person first. This was also one run, on one made-up company, and we wrote the test ourselves.
Would this work on your admin?
Send us the job ad you're hiring for, and we'll send back a free one-page breakdown of what we'd take off the role.