The science behind Mollom: Spam vs. Ham

Mollom is not just your average spam-fighting service. It is based on a radically new approach that both improves its spam fighting precision over time and reduces the moderation effort needed to correct its mistakes. After analyzing your content, we not only return a 'spam' or 'ham' result, we also return 'unsure'. If Mollom cannot be 100% certain into which class to put your submitted content, we categorize it 'unsure' and a CAPTCHA challenge is shown on the content submission form to authenticate that the user is human. A more in-depth treatment on this protocol can be found on the "How Mollom Works" page.

Using plots generated from the actual Mollom database, we will now explain in some detail how and why this can work.


Spam vs. Ham, the old way

Spam fighting tools compute a score based on words and links present in the content under investigation. This 'spaminess' score indicates how likely it is that a post is spam or not. Conventional spam fighting tools return a 'ham' result when it seems _likely_ that a post is ham rather than spam, given its spaminess score. This decision line is shown in the graph above. Here, the green line denotes known ham messages, while the red line denotes known spam messages. So if a message is analyzed, and its spaminess score is to the right of the decision boundary, it is considered to be ham.

What is the problem with this approach? Not all content is correctly classified. This may appear to be only a tiny fraction on the plot, but when millions of messages are being processed all the time, we are talking about 1,000's of misclassified messages every day. Some posts that are actually spam land on the right side, the ham side, of the decision boundary where they don't belong. This spam is not recognized by the system and is allowed onto your site. On the other hand, some legitimate messages fall into the spam bucket and will be blocked from your website. Neither of these are desirable outcomes. To counteract this, a conventional spam blocking system dumps all the messages in the spam category into a moderation queue. The site moderator has to periodically go through all of it to pick out the few ham messages misplaced among the spam. It is like looking for a needle in a haystack and not something anyone looks forward to doing.


Spam vs. Ham, the Mollom way

Here is how Mollom solves these problems. Instead of two classes, we define three: 'spam', 'unsure' and 'ham'. Mollom returns 'spam' only if it is 100% sure that the post is spam and these posts are discarded. If Mollom is quite certain (more certain than using the old technique) that a post is ham, it is accepted. But what about the rest?

We define a gray zone, an area of uncertainty, and here is where the CAPTCHAs come in. When Mollom is unsure about a submission, the user is asked to respond to a CAPTCHA. If the response is correct, and thus the submitter is human, the content will be accepted. Otherwise the post will be rejected. But wait, people hate CAPTCHAs ... True, but as you can see on the graph above, only a tiny fraction of real human-submitted content falls into our 'unsure' zone and triggers a CAPTCHA (currently, only approximately 4% of human submissions). To the very largest extent, CAPTCHAs are not shown to humans at all, they are shown to the bots!

So, to sum up: (i) Mollom is more accurate because our ham boundary is shifted to the right on our graph (making it very strict), so significantly less spam can sneak in (we are now at 99.94% correctly classified ham messages), and (ii) the need for a moderation queue is gone, since the real human users perform the moderation themselves instead of site owners or moderators.

This is just one of the innovations upon which Mollom is built. In future blog posts, we will investigate more of the 'crap-fighting' tools in Mollom's arsenal.

CAPTCHA's

I think motivating the need for CAPTCHA's and showing experimental data that indicates in only a minority of cases the human user will have to fill in the CAPTCHA form is a good thing. It might make people more accepting of the fact that if they encounter it, it's because it is really necessary, and not just some standard part of the submission procedure. Given this, I think such data may actually motivate site owners to start using Mollom rather than simply raising an extra barrier in the form of a CAPTCHA in every case. Good work!

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.