How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
DA AND PA estimation based on Backlink or search engine traffic??
I'm want to know that estimation of blogs DA and PA based on search engine traffic(google,yahoo,bing) or it totally depend on link connected to that perticular domain.
Moz Pro | | imDelu0 -
Page Title "Untitled Document" with mozbar
Hi, I'm working on a big news website (in tourism) and, when checking my homepage with mozbar, I often get: "My Title • Untitled Document" - 69 characters With no apparent reason, If I refresh the page or simply visit it later, it could randomly switch to the "normal" title: "My Title" - 44 characters It looks like completely random to me, like something more is loaded in the first case. I've checked my source code of course, and <title>tag seems to be ok with no other <title> even when I get the "untitled document".</p> <p>I have no idea about that. I don't know if that can be a useful info, but I was trying to understand why the majority of my main keywords searches are only showing other pages (I can rarely find my homepage) even though my home looks correctly indexes and shows as first result for every site: research.<br /><br />Hope everything's is clear....any idea or suggestions?<br /><br />Thank you in advance.</p> <p> </p></title>
Moz Pro | | Daniele_Carollo0 -
Get into Google : New Sites
I have a brand new website. It was created 10 days ago. How long would it take for it to show up in search results? I understand that since the site is new, there are no sites sending it backlinks. Also, i have optimized the page for my keyword "xyz" and it received an A grade. The site does not figure even in the top 50 results. Please help me out. It is a one page web application that needs to drive traffic to survive.
Moz Pro | | dl_s0 -
Crawl results on keyword rankings differ from actual search rankings
For the second week in a row, the weekly keyword rankings in my MOZ account show "not in top 50" for most keywords. However, when I check some of my rankings I still broadly have the rankings that I used to have. It seems that the MOZ crawl is experiencing some difficulties. Does anyone have suggestions how to find out what's going on? If I can't rely on the crawl outcomes, then the tool is not very useful anymore!
Moz Pro | | matteis0 -
Campaign Domain "change" (www. vs no www.)
Hello, I'm tracking a domain (with www.thespin.be) since months. Last week we changed things related to domain & server and now SEOmoz has 0 pages crawled. I think that's because now my main domain is without www. (thespin.be). How can I modify my campaign to reflect this "change" (that's not really a new domain..) to avoid losing all my campaign history ? Also, on Google Webmaster Tools it's like I have 2 different sites (one with www. and another without), although I've defined my favorite domain (without www.) months ago.. Google won't update its index. And I have this problem on several other sites.. Thank you very much for giving me a feedback. Regards Jean-Louis
Moz Pro | | JeanlouisSEO1 -
Is it normal to see the "Loading a sweet graph" message for more than 10 minutesd?
Hi! I'm on the Rankings History page, and for more than ten minutes I have seen this message where I should see data on organic visits: Organic Google traffic for visitors from United States Loading your graph data. Thank you for your patience… Loading a sweet graph... Is this normal? Could there be something wrong with my Google Analytics connection? Thanks! P.S. I looked up "loading a sweet graph" in this forum before posting, but didn't find anything.
Moz Pro | | ScottShrum0 -
Quote "There is no SEO software that can help optimize a site"
Hi all I am a developer for a small agency in the UK. I am using SEOmoz for a couple of our clients and I am also using it to learn from. So far I am finding it a great tool and this forum is also incredibly helpful. Now I also visit another forum called HighRankings.com and this months newsletter is a list of points that are defined as "SEO Mistakes". In it, point number 2 reads: "Repeat after me: There is no specific number of times a keyword phrase should be used in my content. There is no magic number of words that my pages should have written on them. And there is no best number of words or phrases that belong in a Title tag. And most of all: There is no SEO software that can optimize my website (despite the claims of their creators). Use your common sense to optimize your site! Learn how your target audience searches for products, services and information such as yours, and write about it accordingly on your website. Then write to make an emotional connection with your visitors so that they'll convert into happy customers." Now as a user of SEOmoz I wondered what others thought of this comment? Many thanks for reading. Kris
Moz Pro | | yousayjump0 -
Anyone know how/where to get keyword yield ratio?
I'm trying to incorporate keyword yield ratio (the ratio of keywords to pages yielding search traffic) in my seo report. I want to be able to compare on a monthly basis the percentage ratio of keywords to a landing page yielding search traffic. So for example, in July we might have 5 keywords that earned a click to landing page A. But in September, the number of keywords grew to 8 keywords that earned a click to landing A. I would like to be able to report monthly on the % ratio growth of number of keywords to landing pages that yield search traffic. Aim is to grow # of keywords per page... Is there a way I can get this data through seomoz tools or google analytics? Thanks, Gemma
Moz Pro | | RBA0