Spam filter false positives: what a held enquiry really costs

A spam filter can be 99% accurate and still hold one real enquiry in ten. How to work out what those misses cost you, and how to measure a filter before you trust it.

Every spam filter makes two kinds of mistake. It lets some spam through, and it holds back some messages a real person wrote. The first kind is visible: it sits in your mailbox and annoys you. The second kind is invisible by design. You never see the enquiry you did not receive, so you never count it.

That imbalance is the whole problem with judging a contact form filter by how clean your mailbox looks.

Two errors, two different bills

People who build classifiers use two words for how well a filter works, and both are worth knowing.

Precision asks: of everything the filter held, how much was actually spam? Google's Machine Learning Crash Course defines it as "the proportion of all the model's positive classifications that are actually positive".

Recall asks: of all the spam that arrived, how much did the filter catch? In the same course's words, "the proportion of all actual positives that were classified correctly as positives".

You cannot push both to perfect. A filter set to catch more spam holds more real mail along with it; a filter set to hold only what it is sure about lets more spam through. The threshold moves you along that trade-off. Google's page on thresholds and the confusion matrix is blunt about the default: if "misfiling legitimate emails is worse than letting spam into the inbox", a 50% threshold "produces undesirable results".

On a contact form the two mistakes do not cost the same. A spam message that gets through costs thirty seconds and a flicker of irritation. A genuine enquiry that gets held can cost the job. The course's own advice is to watch the false positive rate, which in its spam example "measures the fraction of legitimate emails that were incorrectly classified as spam", whenever false positives are the expensive error. For a form that brings in work, they usually are.

How "99% accurate" still loses real enquiries

Accuracy sounds like the number that matters. On a contact form it tells you very little, because spam usually outnumbers real messages. When one kind of message is rare, a filter can be right almost all the time and still get wrong a large share of the messages you care about. Google's course gives the extreme case: where one class appears 1% of the time, a model that never predicts it "would score 99% on accuracy, despite being useless."

Here is an illustrative month, with round numbers chosen to keep the arithmetic easy. The form receives 1,000 submissions: 950 spam, 50 genuine.

The filter holds 945 of the 950 spam messages and misses 5.
It also holds 5 of the 50 genuine enquiries.

Count the errors: 5 missed spam plus 5 held enquiries is 10 mistakes in 1,000 messages. The filter is 99% accurate. Its precision is 945 spam out of 950 messages held, about 99.5%. Its recall is 945 caught out of 950 spam, also about 99.5%.

And it held one real enquiry in ten.

None of those headline percentages is false. They describe how the filter treats spam, and spam is not what you run the form for. A filter that held nothing at all would score 95% accuracy on this month. The number worth asking about comes from the other side: of the messages real people sent, what share did the filter hold?

Put your own numbers in

What those five held enquiries cost depends on your business, so work it out with your figures rather than anyone's averages. The values below are illustrative; replace them with yours.

Input	Illustrative value	Where to find yours
Genuine enquiries per month	50	Form entries, or last quarter's genuine enquiries divided by three
Share of genuine enquiries wrongly held	10%	Only measurable by reviewing held mail (see below)
Enquiries lost per month	5	Row 1 × row 2
Enquiry-to-customer conversion rate	20%	Won jobs divided by enquiries, over a few months
Customers lost per month	1	Row 3 × row 4
Average order value	£2,500	Invoices, or the average first job
Revenue lost per month	£2,500	Row 5 × row 6
Revenue lost per year	£30,000	Row 7 × 12

Two things stand out. The result swings heavily on the second row, which is the one number most businesses have never measured. And in this example, halving the held rate from 10% to 5% is worth £15,000 a year, while letting a few more spam messages through costs a few presses of the delete key.

If your average order is a £40 booking, the same sum gives a small number and a stricter filter may be the sensible choice. If one enquiry can be a six-figure contract, one miss a year is too many. Either way, it is a decision to make deliberately, form by form.

Held, unseen, and then too late

In theory a held message is recoverable. In practice, a folder nobody opens is a bin with a delay. Contact form spam folders are checked rarely, and when they are, the person checking is scrolling past hundreds of obvious junk messages looking for the one that is not.

Even when the held enquiry is found, time has done its own damage. In The Short Life of Online Sales Leads (Harvard Business Review, March 2011), James Oldroyd, Kristina McElheran and David Elkington describe a study of 1.25 million sales leads received by 42 US companies. Firms that tried to contact a potential customer within an hour were "nearly seven times as likely to qualify the lead" as those that tried even an hour later, "and more than 60 times as likely as companies that waited 24 hours or longer." In a separate audit of 2,241 US companies, the same authors found that 23% never responded to a web-generated test lead at all.

The data is fifteen years old and American, and your sector may move more slowly. The direction is not controversial. An enquiry that arrived on Monday and is found on Friday has often already gone to someone else.

So, unless you do something about it, a filter's held rate is effectively its deletion rate.

Measure before you trust

You cannot know row two of the table by thinking about it. You have to look.

Start in observation. If your filter has a dry run, monitor or log-only mode, use it: everything is delivered as normal and the filter records what it would have held. Leave it for one to two weeks, longer if your form is quiet.
Review every "would have held" message. Not a sample. Read each one and mark it spam or genuine. It is tedious once and useful for a long time.
Count the disagreements. Genuine messages the filter would have held are your false positives. Divide by all genuine messages in the period to get your held rate. Spam it would have let through tells you the other side.
Review the uncertain ones instead of deciding them. A filter that can say "not sure" and show those messages to a person trades a little attention for fewer silent losses. On a sales form that is nearly always the right trade.
Set the threshold per form. A careers form, a quote request form and a brochure download do not deserve the same strictness. Loosen it where a miss is expensive; tighten it where a miss costs little.
Check again after anything changes. A new form, a redesigned page, a campaign that brings in a different kind of customer, or an update to the filter itself can all move the numbers. Repeat a short observation period.

This is the same discipline you would apply to any other change on a sales channel. It is rarely applied to spam filtering, because the failures make no noise.

A checklist for any spam filter

Whatever you use, whether a plugin, a CAPTCHA or a hosted service, ask five questions before you let it decide anything on its own. The form-side options have their own trade-offs, covered in CAPTCHA on contact forms and why blocklists do not work on a contact form.

Does it keep what it holds? If blocked submissions are thrown away, you can never measure false positives or recover from one. A submission rejected at the form usually leaves nothing to review.
Does it tell you why? A held message with a reason attached can be checked in seconds. A bare score cannot really be checked at all.
Can you release a message? Releasing should deliver it to the people who would have received it, not just mark it as read somewhere.
Can you test it without risk? An observation mode, where nothing is held while you measure, is the only safe way to learn your held rate.
Can you tune it per form? One global setting forces the careers form and the quote form to share a threshold they should not share.

A filter that fails the first question fails the rest by default.

Where this is genuinely hard

Some of the most valuable messages a business receives look, on the surface, like the least valuable. A journalist on deadline sends the same short, polite request to a dozen companies. A procurement team sends a templated invitation to tender. A buyer placing one large order writes a brief, formal note with no personal detail, because they are contacting several suppliers at once.

A typical message of that kind looks like this:

Hello, we are sourcing suppliers for a project starting in the new year and would like to include you. Could you confirm whether you are able to quote, and the best contact for this? Kind regards.

It is rare, unsolicited and templated, which is also the profile of outreach spam. It is exactly the kind of message where one miss outweighs a year of irritation. And two weeks of observation may not contain a single one, so a clean test does not prove the filter handles them.

No clever rule fixes this. What helps: a filter that asks rather than decides when it is unsure, held mail that stays searchable long enough to go back and look, and someone who knows to check when a prospect says "we did contact you". Deciding where each kind of message lands once it is let through is a separate problem, covered in routing website enquiries.

If you would rather not build that process yourself, Humainbox's dry run delivers every message while recording what it would have held, with a written reason for each decision, so you can count your own held rate before switching anything on (how it works). Held mail is never deleted by the filter; it stays in the panel and can be released (how it works).