Need someone to lead product or development at your software company? I lead product and engineering teams and I'm looking for my next opportunity. Check out my resume and get in touch.

Filtered Comments

Freshness Warning
This blog post is over 17 years old. It's possible that the information you read below isn't current and the links no longer work.

A number of people have suggested using Bayes filters or other content analysis in order to defeat comment spam. The problem with this approach is that comment spam is a different animal that email spam. At this point in time (although it may change), the purpose behind comment spam is to add the spammer’s URL to your site. By sprinkling their links over hundreds or thousands of pages, they hope to increase their search engine rankings.

The messages themselves are always short, innocuous comments. Some of the ones I’ve seen make an attempt to appear to be a valid comment by copying verbatim the contents of someone else’s comment.

Since there is nothing spammy about the text of the comment, content filtering won’t do any good.

Martin Sutherland
October 10, 2003 6:52 PM

It sounds like the comments spam I get is probably not representative of comments spam in general. About half of the stuff I get has some kind of marketing message in the body, and I thought that would be enough to use as a basis for Bayesian filtering. Even the other kind, which consists of a URL and an apparently innocuous null sentence, would contain a URL that has never before been seen on my site. As a message token, this would therefore be assigned a higher spam probability than the homepage of a frequent comment poster, whose messages have been categorized as ham. However, you're right that in these cases, there isn't an awful lot to go on. A Bayesian filter generally wouldn't be able to make a definitive determination. In his article "So Far, So Good" (http://paulgraham.com/sofar.html) , Paul Graham calls messages like this "spam from the future", because spammers have figured out that this is a way to defeat (or at least sow doubt amongst) Bayesian filters. You could take the Bayesian path one step beyond the message itself, and apply filtering to the text of the target URL, but that obviates the unique feature of tools like SpamBayes: the fact that you don't need to consult any external references to identify spam. If you're willing to consult an external source to confirm the spam content of a URL, why not consult a blacklist rather than the URL itself? The blacklist would probably be quicker. (Unless, of course, spammers started using customized URLs that were materially different for each person on their mailing/comments list. Ugh.) Bayesian filtering works so well for email, that I'd like to think there is a place for it in identifying comments spam. But it clearly isn't enough *in itself* to be a complete solution. Does it have enough merit to be worth considering as *part* of a complete solution, or do other techniques cover the same ground just as (or more) effectively?

KO
October 11, 2003 6:28 AM

Yes, people have been leaving generic comments like "Nice post" and so on with spam urls in the links on my blog (despite the fact that it hardly gets any traffic. There doesn't really seem to be any easy solution to this besides blacklisting. Isn't there a limit to the number of different urls spammers use? After all, they are promoting actual existing sites, and there are a finite number of links to those. Cannot the blacklist only contain the main URL for a spam site, and automatically find and blacklist all the alternate url's which also point to it?

kim
October 12, 2003 11:13 AM

I've been noticing that most of my spam has been coming from the 209.210.176.* block. They always resolve to pornfilter#.someISP or pfilter#.someISP. The URLs always contain some sort of porn. I agree with KO, I've only found blacklisting to work, but it's becoming a pain.

MojoMark
October 13, 2003 10:36 AM

FYI: Lots more comment spam around hitting blogs recently. Just saw lots of discussion at WindsOfChange.Net and some good stuff at ScriptyGodess.

Shirley Kaiser
October 14, 2003 12:40 PM

Hi, Adam, I came over to your site to see if you had any information on SimpleComments with Jay's new filter, as there are some little quirks going on with numbers. You probably already know about Jay's MT-Blacklist plugin, but just in case you can see what's going on at Jay's site about this at http://www.jayallen.org/journey/2003/10/mtblacklist_stop_spam_now. I agree with you that content-only filtering is off-the-mark. Filtering the URLs that the spammers are trying to leave at our sites can be quite helpful, however, as I've experienced firsthand. Since Jay's approach includes all the form fields, it covers wherever the spammers try to add their links. Of course filtering isn't going to stop 100% of the spam, but I've seen it trickle to almost nothing since I implemented the filters at my own site. Not having the spam show up at all is a huge deterrent, and the volume of spam that does get through is far less time-consuming to manage. Just like email spam, though, I suspect the comment spammers will work on ways around the filters. So we'll see where it all goes. Too bad they don't use their energy in far more positive ways. Then again, the sites they're leaving for us are questionable, too. Hopefully the search engines and directories will address this comment spam, too, as they do try to ward off tactics to fool them. Additionally, I totally agree with you that removing spam immediately is critical. I've linked to your site about what you've written on this. Thanks for all the terrific content at your site, Adam.

Chris Vance
October 16, 2003 2:41 PM

I just found a link to MT-Blacklist, but Shirley linked to it first. I initially found the link from Luke Hetteman's blog (see http://www.hutteman.com/weblog/2003/10/14-132.html). In an earlier entry (http://www.hutteman.com/weblog/2003/09/22-123.html), a commenter in refered Luke to your "Comment Spam" article.

James Seng
October 17, 2003 5:09 AM

Actually, Bayesian proposed only needs about 10-15 significant words to work properly. And if you add IPs, URLs and Hosts (even those "Nice Post" spam will have URLs), it get pretty significant. And that's what I did with my MT-Bayesian.

mark
February 26, 2006 11:43 PM

I'm confused about just what a comment spammer is? If someone addresses the topic at hand, they should be able to link your article to their site. Especially if the site's relate (blog site to a blog site etc.).

This discussion has been closed.

Recently Written

Domain expertise in Product Management (Nov 16)
When you're hiring software product managers, hire for product management skills. Looking for domain experts will reduce the pool of people you can hire and might just be worse for your product.
Strategy Means Saying No (Oct 27)
An oft-overlooked aspect of strategy is to define what you are not doing. There are lots of adjacent problems you can attack. Strategy means defining which ones you will ignore.
Understanding vision, strategy, and execution (Oct 24)
Vision is what you're trying to do. Strategy is broad strokes on how you'll get there. Execution is the tasks you complete to complete the strategy.
How to advance your Product Market Fit KPI (Oct 21)
Finding the gaps in your product that will unlock the next round of growth.
Developer Relations as Developer Success (Oct 19)
Outreach, marketing, and developer evangelism are a part of Developer Relations. But the companies that are most successful with developers spend most of their time on something else.
Developer Experience Principle 6: Easy to Maintain (Oct 17)
Keeping your product Easy to Maintain will improve the lives of your team and your customers. It will help keep your docs up to date. Your SDKs and APIs will be released in sync. Your tooling and overall experience will shine.
Developer Experience Principle 5: Easy to Trust (Oct 9)
A developer building part of their business on your product needs to believe that you're going to do the right thing for them and their customers.
Developer Experience Principle 4: Easy to Get Help (Oct 8)
The faster you can unblock a stuck developer, the better their experience will be.

Older...

What I'm Reading

Contact

Adam Kalsey

+1 916 600 2497

Resume

Public Key

© 1999-2020 Adam Kalsey.