If you own a web site or a blog, you no doubt know all about the term
"referer spam". When you click a hyperlink on one web site, your
browser passes to the next site the address of the page where you
clicked the link. This is logged by the server hosting the next web
site.
The referer information can be faked very easily. Some unscrupulous
web site owners will arrange to have several computers access a
particular web site with a referer that lists their own web site
address. There are a number of ways to accomplish this (see below),
but the result is that the web server logs of the targeted site will
contain hundreds or possibly thousands of entries with the fake
referer information. This is known as "referer spamming".
What's the point?
Why go to the effort of leaving a web site address in someone else's
log files?
Most web servers have the ability to log an extensive amount of
information about web sites visitors. Many webmasters and bloggers
use web-based software to parse those log files automatically. The
result is one or more pages breaking the information down into very
detailed statistics. These statistics include the referer information
and often those referers are displayed as hyperlinks.
Bloggers quite often will display a link to the most frequent or most
recent web site found in their referer logs using scripts. Some will
even put those links right on the front page of their site in a
sidebar area.
Unscrupulous web site owners are spamming the log files in order to
have their web sites listed on those referer links. This creates an
artificial boost in that site's popularity among those search engines
that measure the number of links to a site. It also generates traffic
when curious visitors of a victim site clicks the links displayed in
the referer listing.
Simply put, these people are running advertisements on your web site
and using it to boost their search engine rankings. They do this
without your knowledge, without your permission, and without
compensating you in any way for the use of your network.
How is it done?
Spammers decide which sites to spam by checking sites automatically
such as blo.gs, weblogs.com, and popdex.com for blogs that have
updated recently. They also may do a simple Google search for the
phrase "recent referers".
Once the spammer has chosen which sites to spam, there are several
ways to go about it. At one point, there was even a company that
offered to spam the logs of over 55,000 sites for a fee.
One clever method used by porn sites is to include an image tag in
their page's HTML that calls your web site's home page. No image is
loaded, but visitors to the porn site generate a hit on your site
that is logged by the server.
Another is to use a desktop application similar to a spambot email
harvester. Instead of scooping up email addresses, the purpose of
this sort of application is to load all of the pages on your site
while leaving a custom referer in your logs. One blogger wrote such a
program, but never released it fearing abuse (and the wrath of his
fellow bloggers no doubt).
Another method that people speculate about is browser hijacking.
Internet Explorer can be hijacked to change the start page, search
settings, DNS error handling, search hooks, and to have BHOs and
strange toolbars installed. Certainly the referer setting can also be
altered by a BHO. To my knowledge, this has never been done. Let's
hope I'm not giving anyone ideas...
Legitimate companies do it too
It is worth noting that it is not just porn sites that spam referer
logs. A few software developers include an advertisement for their
product that is sent out each time a customer uses it to visit a
site. Some RSS news aggregators are known to do this.
At least one well known security company also does this. Outpost
firewall by Agnitum spams referer logs with "Field blocked by Outpost
firewall (http://www.agnitum.com)" when users turn on referer
blocking. This is why you will not see Outpost recommended on my
software page. Norton Internet Security also alters the referer in
some nonstandard fashion. I don't know if it also spams the log or if
it changes the referer to gibberish. Either way, it is not blank and
it is not the real referer.
My message to software developers is to leave the referer alone.
Either block it entirely and leave it blank or don't touch it at all.
By meddling with it, you are causing problems on web sites that use
scripts that depend on the referers. Also, you are spamming and
legitimate software companies should know better.
How to stop this?
You have this happening on your own web site. The spammers are all
over your log files. How do you stop it?
God bless the people at Apache for creating such a simple and
powerful web server. If you have a web site, chances are it is hosted
on Apache. Apache makes it very simple to block anyone from your site
you wish to block, based on just about any parameter you choose.
I have no idea how to block someone on a Windows IIS server. If
anyone knows, hopefully they will clue me in.
[Update: A reader has written in to say that ISAPI_Rewrite can make
these techniques work on some IIS servers. You will need to have your
host install it for you if you are on shared hosting. ]
Log into your site's FTP server. Make sure it is set to display
hidden files on the server. Check your FTP client's documentation for
help with that. If there is a file named .htaccess, download it and
open it in a text editor. If there isn't one, create one on your hard
drive and upload it when you are finished.
Put this in the .htaccess file, changing "spammersite" to the name of
sites found spamming your logs. The first site listed should not have
"NC", and the last one should not have "OR".
If your site starts generating errors after you upload this file,
remove the # from the first line. If you use Microsoft Frontpage on
your site, do not do this. Changing the .htaccess file could
interfere with Frontpage.
This will redirect any request with a spammer site in the refer to
any other site you wish. If you prefer, you can rewrite that last
line as "RewriteRule .* - [F,L]" to give them a "Forbidden" error.
Either way, it keeps them out of your referer logs and there is no
way to defeat it.
If you want a nice, large list to start your blacklist, you may use
mine. I was fed up with spammers in my own logs, so I tracked them
all down and listed them. It was to the point that six of the top ten
referers in my stats were spammers. Some of them had made tens of
thousands of entries in my logs.
If you prefer, you can also use wild cards to filter out domain names
that are likely to belong to porn sites. That should catch most of
these spammers, as porn sites seem to be the biggest offenders.
Blogger Joe Maller has a blacklist that should work perfectly for
this.
Since people are far more likely to have their browsers hijacked
while surfing for porn, there are a large number of porn sites that
do link to SpywareInfo. I can't use Joe's list since I don't want to
block those people.
Despicable behavior, but easy to block
The people who would spam someone else's web site to promote their
own are scum. Thankfully, using the method above, it is stopped
easily.
If you notice a web site that displays referers that is being
spammed, please contact the webmaster of that site and point them
here so they can learn how to block it.
