Filtered Comments

It sounds like the comments spam I get is probably not representative of comments spam in general. About half of the stuff I get has some kind of marketing message in the body, and I thought that would be enough to use as a basis for Bayesian filtering.

Even the other kind, which consists of a URL and an apparently innocuous null sentence, would contain a URL that has never before been seen on my site. As a message token, this would therefore be assigned a higher spam probability than the homepage of a frequent comment poster, whose messages have been categorized as ham.

However, you’re right that in these cases, there isn’t an awful lot to go on. A Bayesian filter generally wouldn’t be able to make a definitive determination. In his article “So Far, So Good” (http://paulgraham.com/sofar.html) , Paul Graham calls messages like this “spam from the future”, because spammers have figured out that this is a way to defeat (or at least sow doubt amongst) Bayesian filters.

You could take the Bayesian path one step beyond the message itself, and apply filtering to the text of the target URL, but that obviates the unique feature of tools like SpamBayes: the fact that you don’t need to consult any external references to identify spam. If you’re willing to consult an external source to confirm the spam content of a URL, why not consult a blacklist rather than the URL itself? The blacklist would probably be quicker. (Unless, of course, spammers started using customized URLs that were materially different for each person on their mailing/comments list. Ugh.)

Bayesian filtering works so well for email, that I’d like to think there is a place for it in identifying comments spam. But it clearly isn’t enough in itself to be a complete solution. Does it have enough merit to be worth considering as part of a complete solution, or do other techniques cover the same ground just as (or more) effectively?

Yes, people have been leaving generic comments like “Nice post” and so on with spam urls in the links on my blog (despite the fact that it hardly gets any traffic.

There doesn’t really seem to be any easy solution to this besides blacklisting. Isn’t there a limit to the number of different urls spammers use? After all, they are promoting actual existing sites, and there are a finite number of links to those. Cannot the blacklist only contain the main URL for a spam site, and automatically find and blacklist all the alternate url’s which also point to it?

I’ve been noticing that most of my spam has been coming from the 209.210.176.* block. They always resolve to pornfilter#.someISP or pfilter#.someISP. The URLs always contain some sort of porn. I agree with KO, I’ve only found blacklisting to work, but it’s becoming a pain.

FYI: Lots more comment spam around hitting blogs recently. Just saw lots of discussion at WindsOfChange.Net and some good stuff at ScriptyGodess.

Hi, Adam,

I came over to your site to see if you had any information on SimpleComments with Jay’s new filter, as there are some little quirks going on with numbers. You probably already know about Jay’s MT-Blacklist plugin, but just in case you can see what’s going on at Jay’s site about this at http://www.jayallen.org/journey/2003/10/mtblackliststopspam_now.

I agree with you that content-only filtering is off-the-mark. Filtering the URLs that the spammers are trying to leave at our sites can be quite helpful, however, as I’ve experienced firsthand.

Since Jay’s approach includes all the form fields, it covers wherever the spammers try to add their links.

Of course filtering isn’t going to stop 100% of the spam, but I’ve seen it trickle to almost nothing since I implemented the filters at my own site. Not having the spam show up at all is a huge deterrent, and the volume of spam that does get through is far less time-consuming to manage.

Just like email spam, though, I suspect the comment spammers will work on ways around the filters. So we’ll see where it all goes. Too bad they don’t use their energy in far more positive ways. Then again, the sites they’re leaving for us are questionable, too.

Hopefully the search engines and directories will address this comment spam, too, as they do try to ward off tactics to fool them.

Additionally, I totally agree with you that removing spam immediately is critical. I’ve linked to your site about what you’ve written on this.

Thanks for all the terrific content at your site, Adam.

I just found a link to MT-Blacklist, but Shirley linked to it first. I initially found the link from Luke Hetteman’s blog (see http://www.hutteman.com/weblog/2003/10/14-132.html).

In an earlier entry (http://www.hutteman.com/weblog/2003/09/22-123.html), a commenter in refered Luke to your “Comment Spam” article.

Actually, Bayesian proposed only needs about 10-15 significant words to work properly. And if you add IPs, URLs and Hosts (even those “Nice Post” spam will have URLs), it get pretty significant.

And that’s what I did with my MT-Bayesian.

I’m confused about just what a comment spammer is? If someone addresses the topic at hand, they should be able to link your article to their site. Especially if the site’s relate (blog site to a blog site etc.).