Blocking Competitive Analysis Tools in robots.txt.... Worth it?
-
I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all.
My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?
-
Hi David,
Well, if you use a tool that can change the name of the bot that comes after a site, then you can't do anything about it. However, I do know that SEMrush is causing problems for some hosts and the likes of Heart Internet, they block access to the SEMrush bot on all of their servers.
To be honest with you, it is quite difficult to block everything and time consuming and unless you have some secret formula to keywords that are magically drawing in more traffic, then I wouldn't worry about it too much.
Take Accuranker. I use this for tracking and part of setting up, is it will find new phrases that I perhaps haven't been using or looking at before. It doesn't get these from either SEMrush or Ahrefs. All of this information is available from many sources.
I would honestly spend the time making your pages as good as they can be. Look at your cometitors. Are they winning? How are they doing it? Fight back against them rather trying to hide yourself from view.
-Andy
-
Issa is correct and to be honest, he's pointed out what I see as a pretty major problem - not being able to see these metrics (particularly ahrefs) really isn't ideal for analysis and/or troubleshooting.
Also, it may very well be that the competitor that keeps targeting the same terms as you is doing so via other means. There are countless tools available to all of us that allow for analysis of strong and relevant terms so your site may have little to do with their decisions.
Beyond this, one glance at your page titles will often be enough to figure out what you're targeting
-
Hi David,
This is completely your choice. Your website's SEO and overall ranking wont be affected by block SEMRush for example.
The only drawback I can think of is that you wont be able to use this tool for your own domain research, which can be helpful if you're looking for ranking stats, or click share from search engines. I would hate to not be able to research my client's websites regularly, as this is the kind of intelligence I use to a) understand Google's and Users behaviour to find my website. b) discover technical and SEO issues with my websites.
So although you can go a head and do it, I would say you should always be ready to unblock them when needed. Unfortunately SEM Rush does not update their database so regularly, so you wouldn't be able to predict when you should unblock their robot for information.
I hope I could help
Issa
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Competitive Analysis Leaving Me Confused
I'm doing work for https://www.charlestonblackcabcompany.com While I know whey we are only at #2 for Charleston Party Bus as Google treats it as a branded keyword search and I don't know of any ways around that there are other competitors that don't make sense to me. These guys are #1 for Charleston Limo Service and we are further down the list despite having much better numbers across the board for months now with out seeing much gain. https://mza.bundledseo.com/researchtools/ose/links?site=http%3A%2F%2Fwww.charlestonlimo360.com%2F&filter=&source=external&target=page&group=1&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site= I'm talking over 6 months and still these guys and some others whose competitive analysis shows us beating them across the board still has us further down the rankings for over 6 months. It seems like the competitive analysis doesn't matter at all.
Competitive Research | | mwaters19790 -
Currently Number 2 - Worth Shooting for 1?
Hello SEO Community, So I was able to rise to #2 in a highly competitive search term after a guest blog post that I wrote really took off (wish I could recreate that for my other terms, haha) Anyway, I steadily rose from 30ish to number 2 and have obviously seen more traffic, which is awesome and I'm happy. But still, in terms of the amount of traffic that google estimates the keyword gets, and the amount I'm getting at #2, it seems like #1 is getting the VAST majority (I know this is how most markets work). Judging from my link profile compared to #1, it would be extremely difficult to dethrone them. So what do y'all think? Is it worth spending time and resources shooting for the glory and traffic of #1 and potentially never getting there - or focusing my efforts on my middling keywords and getting those on the first page as well?
Competitive Research | | CleanEdisonInc0 -
SEMRUSH--- Tool Review
Please review this tool on your personal experiences with it. I used it on my domain and it reported Search Engine traffic as 10% of it's real (google analytics) traffic. I haven't really mastered GA or SEMRush but I am getting a bad taste. Please indicate your experiences. Tell me what you use the tool for. What are it's competitors etc...
Competitive Research | | JML11790 -
Looking for a tool or service which does the following
Hello, I wanted to ask if anybody knows a tool or website which does the following: You enter your competitors URLs the service will find websites which links to those URLs the service will sort these by PageRank Then you can contact these website owners and introduce your website as well and hope they'll link to you or write a blog post about you. Regards, Oliver
Competitive Research | | delpino0 -
How do I find my client's real competition?
A recently acquired client keeps insisting that his major competitors are online fashion magazines, fashion news sites and insider bloggers but the site he owns is basically a fashion encyclopedia. It contains facts and catalogues and even a dictionary of fashion concepts. According to me, these latter are his benchmarks and not his competition His major competitors, according to me. are people competing on the first SERP pages for terms like "fashion history", "fashion icons", "fashion biographies", etc. Am I right? How do I convince him that these are his real competition. This is really going to determine the direction of my keyword research.
Competitive Research | | amit20760 -
Website Neighborhood Tool
Hello,
Competitive Research | | Ric
Do you know a tool where you can find other websites with the same Adsense code or Analytics Code? Thanks, Eric0 -
My competition is using 1px images and links without anchor text. Good or bad idea?
I've been scoping out some of my competition and I noticed the number 1 ranking site is doing a lot of "interesting" stuff. I saw a bunch of 1px blank images that they must be using for the alt tag keywords. I also noticed several links to internal pages with different keywords without anchor text so they don't actually appear on the page. Is this considered black hat? Should I be doing something similar? It feels like the work I'm doing to try to rank #1 with a clean/professional looking site are wasted when the #1 sites are doing things like this.
Competitive Research | | technota0 -
Is there a tool that will tell me how many visitors per day a website receives?
Hey guys, I am new to SEOMoz and have setup my first campaign, so far very impressed with what is possible. However I am trying to find out how much traffic a website is getting, as I know that people can be high up on certain keywords but are not actually generating any traffic from them. Can anyone help? Also I have a potential client who is ranking really high on alexa because of its american traffic, yet when I asked them they get very few bookings from americans. How can I view the UK ranking of the site? Thanks Andy
Competitive Research | | iprosoftware0