In case you haven’t heard of Ahrefs, it is an amazing tool that helps you spy on competing websites and spot opportunities to steal traffic from them.
You do have to put in the effort to create valuable content that would outrank the competition, but essentially Ahrefs makes it about 100 times easier and takes most of the guesswork out of it.
So the question is – if Ahrefs makes it easy for YOUR competitors to steal YOUR traffic, should you block them from indexing your site?
The answer is… YES!
Perhaps the question should be… why wouldn’t you block them?
Recently one of my blogs that I tracked using Ahrefs has had a sudden an unexplained drop in traffic. Interesting enough another website appears to have poached at least one of my articles. Perhaps it is a coincidence? Not sure.
Sure, the service allows you to “track your own website” but this comes at the risk of sharing all of the hard work you have put into your blog with your direct competition. With your site included in Ahref’s database, all your competitors have to do is click a few buttons, apply a couple of filters and find some of your low hanging fruit to steal.
And just like that, your traffic is gone.
By allowing Ahrefs to index your site, all you are doing is making it super easy for your competitors to outrank you and steal your traffic.
If I am wrong, I’d love to know one benefit of allowing them to index my site without risking traffic loss. Please feel free to comment below if anyone has an answer lol (crickets chirping)
So personally, I recommend to everyone that owns a blog to block Ahrefs. To make matters worse, they are also located outside of the USA. So, if you are a company located inside the USA, what are you going to do about it if their tools really put a dent in your business?
Now, I’m sure they have already figured out a way around anyone that blocks them using robots.txt, but why make it easy for them?
Keep reading and we’ll show you how to add the necessary information to your robots.txt file to prevent Ahrefs from indexing your blogs or web sites.
How to Block Ahrefsbot Using Robots.txt
Add the following code to your robots.txt file of the website you wish to prevent Ahrefs from indexing:
More Information on Ahrefsbot
Ahrefsbot Main IP Ranges
According to their web site, the current IP ranges are:
User-Agent String: Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)
One Final Catch…
They mention what I think is a loophole on their web site that “it need some time to pick up changes in your robots.txt file…”
Also mentions if your robots.txt file contains errors and AhrefsBot won’t recognize commands it will continue crawling by default. Interesting! Why not block by default? Hmm…
If all else fails you may need to email them.
They have a form where you can check the status of your robots.txt file after you update it. Here is that link:
The existence of tools like Ahrefs makes me wonder about the future of Google. It seems like somewhat of a Pandora’s box that may really screw things up for everyone. Everyone except the rich media companies… the content mills…etc.
Perhaps it is yet another sign of a bubble about to burst?
Or maybe Google will consider adding some type of SEO tool penalty to anyone using them too frequently. After all… it’s not as if it’s hard for Google to figure that out. 😉