How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Solve a Lost GA Connection - can't get it to work
Hi, I followed all steps on https://mza.seotoolninja.com/help/moz-pro/getting-started/troubleshoot-google-analytics?_ga=2.15852085.2007735045.1676157945-122376721.1676157945 below Solve a Lost GA Connection multiple times but I keep getting the message "Our connection to your Google Analytics account was lost. Don't worry, you won't lose any data. Please follow the steps in our guide to reconnect.
Moz Pro | | silvansoeters
" I am on the free trial and would obviously like to connect to GA to see how everything is working. Any help would be greatly appreciated. Thanks.0 -
Mobile Search Visibility Has Fallen Significantly in Recent Weeks
I checked my MOZ account last week to find that my site's Mobile Search Visibility has dropped pretty significantly 4 weeks after an algorithm update. It has ticked back up some but after weeks and weeks of steady position, it fell. I thought the latest update had affected the visibility but I'm not sure that is the case. One of our competitors ironically jumped UP in Mobile Visibility about as equally as my site has fallen. I've attached a screenshot of the MSV graph. My site is plotted in BLUE and our competition is plotted in PURPLE. Does anyone have a explanation or potential reason? 6hsAuD.jpg
Moz Pro | | SteveZero120 -
How can we get follow links from Pinterest and Instagram
While using Open Site Explorer from Moz I came across follow links from Instagram and Pinterest. Now my question is how can we get follow links from Pinterest and Instagram
Moz Pro | | rakesh10 -
My localized domains are not getting any love or juice it seems?
Not sure what best practices are, but I have my main domain, a .com and my localized domains, .com.sg and .co.id. As you can see from the screenshot from opensite explorer, i am not getting any linkjuice from my .com domain to my other local domains - what am i doing wrong here? j3LWnOJ
Moz Pro | | venkatraman0 -
Why do I see a duplicate content errors when rel="canonical" tag is present
I was reviewing my first Moz crawler report and noticed the crawler returned a bunch of duplicate page content errors. The recommendations to correct this issue are to either put a 301 redirect on the duplicate URL or use the rel="canonical" tag so Google knows which URL I view as the most important and the one that should appear in the search results. However, after poking around the source code I noticed all of the pages that are returning duplicate content in the eyes of the Moz crawler already have the rel="canonical" tag. Does the Moz crawler simply not catch whether that tag is being used? If I have that tag in place, is there anything else I need to do in order to get that error to stop showing up in the Moz crawler report?
Moz Pro | | shinolamoz0 -
Can I calculate "Keyword Difficulty" metric using Mozscape API data?
We already have a web application that pulls certain metrics about websites using the Mozscape API, but we are wanting to extend the usefulness of this application to enable users of the app to pull "Keyword Difficulty" metrics in bulk, instead of one at a time (or 5 at a time). I wouldn't mind the 5 at a time limitation if we could just automate the API calls and let the tool pull data for 50 or so keywords without user-interaction. I know that it's a "formula", but I don't know what SEOMoz uses for it's formula. Has anyone figured out a way to calculate this, based on the Mozscape API data? Has anyone ever tried to reverse engineer this metric?
Moz Pro | | brchap0 -
My trial ends on 2nd June. Can I cancel it on 1st or we need to initiate it before that , to not get charged ?
I need to cancel my trial. It ends on 2nd june as per seomoz. When can i cancel it exactly to avoid charges.
Moz Pro | | VarunBansal0 -
What's name of SEOmoz and Open Site Explorer robots?!
I would like to exclude in robots.txt SEOmoz and Open Site Explorer bots to don't let them index my sites… what's their names?
Moz Pro | | cezarylech0