Blocking Competitive Analysis Tools in robots.txt.... Worth it?
-
I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all.
My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?
-
Hi David,
Well, if you use a tool that can change the name of the bot that comes after a site, then you can't do anything about it. However, I do know that SEMrush is causing problems for some hosts and the likes of Heart Internet, they block access to the SEMrush bot on all of their servers.
To be honest with you, it is quite difficult to block everything and time consuming and unless you have some secret formula to keywords that are magically drawing in more traffic, then I wouldn't worry about it too much.
Take Accuranker. I use this for tracking and part of setting up, is it will find new phrases that I perhaps haven't been using or looking at before. It doesn't get these from either SEMrush or Ahrefs. All of this information is available from many sources.
I would honestly spend the time making your pages as good as they can be. Look at your cometitors. Are they winning? How are they doing it? Fight back against them rather trying to hide yourself from view.
-Andy
-
Issa is correct and to be honest, he's pointed out what I see as a pretty major problem - not being able to see these metrics (particularly ahrefs) really isn't ideal for analysis and/or troubleshooting.
Also, it may very well be that the competitor that keeps targeting the same terms as you is doing so via other means. There are countless tools available to all of us that allow for analysis of strong and relevant terms so your site may have little to do with their decisions.
Beyond this, one glance at your page titles will often be enough to figure out what you're targeting
-
Hi David,
This is completely your choice. Your website's SEO and overall ranking wont be affected by block SEMRush for example.
The only drawback I can think of is that you wont be able to use this tool for your own domain research, which can be helpful if you're looking for ranking stats, or click share from search engines. I would hate to not be able to research my client's websites regularly, as this is the kind of intelligence I use to a) understand Google's and Users behaviour to find my website. b) discover technical and SEO issues with my websites.
So although you can go a head and do it, I would say you should always be ready to unblock them when needed. Unfortunately SEM Rush does not update their database so regularly, so you wouldn't be able to predict when you should unblock their robot for information.
I hope I could help
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Looking for Competitive Research Tool Recommendations
Hi Moz Community! I'm doing research for a client and trying to gain insights around their competitors website data. Specifically trying to see what landing pages on the competitors site receive the most traffic. Does anyone currently use a tool that shows estimated traffic here? As an FYI, outside of MOZ Pro (which I love), I use SpyFu and SimilarWeb. Right now my client's landing page traffic is mainly homepage traffic and we want to make reccos on where to grow outside the homepage to increase visibility in other specific areas where there is search demand. We have a good idea but I'm hoping the competitor web data will back up those recommendations. Any recommendations here would be super valuable! Thank you in advance for your time. Abbi
Competitive Research | | abbiplunkett0 -
Tools for constant SERP monitoring?
Does anyone know of a tool where you enter in a [search term] and it constantly (or periodically) monitors the SERPs and reports back on the top results? IE- If you wanted to monitor for the term [trump] it would fetch that search result every hour or so and report those via an online interface or downloadable .csv
Competitive Research | | Cox-Media-Group0 -
Difficult to pinpoint competitive differences
Hi moz, Im currently working with a client that has a number of affiliates that work under similar URLs but operate independently. For example I work with client.com and we are concerned about SEO and ranking competition from affilate1.client.com, affiliate2.client.com etc. As it stands right now, the domain client.com has a higher DA, the homepage has a higher PA, and the equity passing links, both internal and external are much higher as well as number of linking domains, yet for whatever reason for many of the keywords we attempt to optimize for, the smaller affiliates often rank higher. The only things ive been able to pinpoint is that affiliate1.client.com has a spam score of 2 (rather than 4, for client.com) and the homepage has more total links. External anchor text is pretty branded across the web so i can't imagine it playing into keyword associations. Lastly the sites are constructed and appear very similar so basic on page keyword best practices have been taken care of. We worry that because affiliate.client.com has been at the SEO game longer, in addition to Google only linking two sites from a domain per search, we will never be able to rank for the keywords we want to rank for. How else can I go about a competitive analysis if I've taken into account everything said above? Any help will be very appreciated.
Competitive Research | | Resolute0 -
Where do we find a Keyword Discovery Tool in SEOmoz?
I've been looking for a way to compare keywords amongst my competition websites.
Competitive Research | | homesonthesound0 -
Www.seoprofiler.com - anyone knows the service? worth the investment?
A little help just to check if the service worth the money spend.
Competitive Research | | Naghirniac0 -
Competitive Link Finder blank screen
Hi. I am bran-new to the site so if I am asking a daft question I apoloise 😉 When I go to the Competitive Link Builder, enter some URLS and press GO I am taking to a blank screen with no info. I've tried on different browsers & same issues. Any advice? Thanks.Christoffa.
Competitive Research | | Christoffa0 -
Is there a tool that will tell me how many visitors per day a website receives?
Hey guys, I am new to SEOMoz and have setup my first campaign, so far very impressed with what is possible. However I am trying to find out how much traffic a website is getting, as I know that people can be high up on certain keywords but are not actually generating any traffic from them. Can anyone help? Also I have a potential client who is ranking really high on alexa because of its american traffic, yet when I asked them they get very few bookings from americans. How can I view the UK ranking of the site? Thanks Andy
Competitive Research | | iprosoftware0 -
What is the right way to due a competitive Link analysis?
Trying to use the saturation report suggestion with no luck for google.com,and looking for away to benchmark either Pagerank or page authority to figure out how many links are required to overr take competitors from diversified sources. What is the process to do this to get the best results for the client.
Competitive Research | | bmcfarlane0