Referral Abuse

It appears that each time an RSS file from my site is loaded by one of these applications, a referer is deposited in the log file. Each time I load a page in Internet Explorer, I don’t leave a referer for www.microsoft.com/ie in the log files of the site whose page I loaded, so why should any of the RSS readers be different?

RSS readers misusing the referer field? (kottke.org)

Amen. I’ve always found it irritating that news aggregators insert their URL into the referrer field. Some aggregators have taken things a step further by allowing the user to use any arbitrary URL as the referrer. So I get 48 "referrals" each day from www.hardhathosting.com even though there’s not a single link from their site to mine. It’s not that I mind knowing where my readers are coming from—that’s kind of nice, and I’m glad the person behind Hardhat Hosting finds me interesting enough to grab the feeds for this this blog and Simplelinks once an hour. It just makes it difficult to distinguish a real referrer from someone else. If Hardhat Hosting were to put a real link on their site to mine and people followed it, I probably wound’t notice. I’d just think the referral logs were crying wolf.

As a temporary measure, you can use a custom URL for the referrals instead of your site’s home page or the home page for the aggregator. L.M. Orchard does this with Amphetadesk, setting the referrer to a thanks page on 0xDECAFBAD and I do this with Aggie. Sites I visit see a link to http://kalsey.com/blog/thanks/ in their referral logs.

It would be nice if there was some sort of browser header the aggregator could send to identify itself instead of using the referrer field. Oh, that’s right, there is. It’s called User-Agent.

The user agent field is designed for browsers, robots, and other user agents to identify themselves to the Web server. You can even add additional information, like a contact URL or email address. I’d like to see aggregators start using it.