Moz Crawl Test error
-
Moz crawl test show blank report for my website test - guitarcontrol.com. Why??? Please suggest.
-
Hi Zoe,
I just tried to view your Robots.txt file and was unable to access it. So I requested a crawl. The error I got from Moz is "612 : Page banned by error response for robots.txt."
I would suggest you look at the file permissions of your robots.txt file. It should be readable by the public; Apache permissions would be (644). This means it is not available to view or it could be that it is in the wrong directory or non-existent altogether. It should be in your root and accessible via a browser here: http://www.guitarcontrol.com/robots.txt
I would also note that some people feel that listing your sensitive directories in the robots.txt file is a security risk, and perhaps they are right. However, there are other ways to secure a directory and keep them out of the Indexes, and if the information being protected is that important, I would imagine that all security precautions would be taken.
The robots.txt file is important for crawlers so it should always be used and available to be read. If security is the problem here, I would point to some high profile sites that don't seem to have a problem with showing the public their robots.txt file.
https://www.google.com/robots.txt
https://www.facebook.com/robots.txtI hope this helps,
Don
<colgroup><col width="907"></colgroup>
| 612 : Page banned by error response for robots.txt. |
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Looking for some help to train corporation on MOZ?
Short term. Basically helping a company train internal SEO people from round the globe. Involves time and fee to create your training plus several 4 hour sessions and possible follow up consulting. Pays well. Email me at [email protected]. Thanks Al Blanco
Moz Pro | | InfusionAdmin0 -
How to force SeoMoz to re-crawl my website?
Hi, I have done a lot of changes on my website to comply with SeoMoz advices, now I would like to see if I have better feedback from the tool, how can I force it to re-crawl a specific campaign? (waiting another week is too long :-))
Moz Pro | | oumma0 -
Crawl Diagnostics shows two title and meta tag errors but they are false positives.
I got one hit each on "Missing Meta Description Tag" and "Title Missing or Empty" but in the source of my page they are clearly there: <title>Protein Powder | Compare and Get the Best Prices</title> <meta name="keywords" content="protein powder, whey protein, protein supplement, whey protein isolate, hydrolyzed whey" /> I understand there are conventions which may or may not be followed by Drupal (I read an earlier question where ordering and W3C conventions were suggested) but i'm not sure how to fix them given Drupal will just overwrite any hand editing the next time something is built and importantly, I can't get the crawl to work on cue - it works on the automatic once a week crawl in the main campaign summary but every time I've specifically used the Crawl Test tool it gives me a "There was an error submitting your request to the crawler. Please try again later" so I can't really test any changes. Given Google seems to be recognising the title tag - ie showing it in the results - Do I put this down as seomoz just not working? Kind Regards, Brian
Moz Pro | | btrr690 -
After fixing errors can I re-crawl for diagnostics?
As I am fixing errors will the campaign automatically update to show where I have fixed issues?
Moz Pro | | eidna220 -
Sub-domain not crawled
One of our sites was recently re-designed. The home page is a landing page (www.labadieauto.com) and I moved the blog to this domain (labadieauto.com/blog/) and put a link is the bottom left of the home page. Since the change the SEOMOZ campaign overview is showing only 1 page crawled. This is not setup as a sub-domain so why isn't it showing in the crawl? Help!
Moz Pro | | LabadieAuto0 -
Campaign Not Crawling
I set up my first 5 campaigns and one is not crawling beyond one-page. It's been over 48 hours. This site has nearly 3.500 pages the others much less, however, this shouldn't make any difference. I searched for the problem and couldn't find it so I hope this question isn't redundant. Comments and advice would be appreciated.
Moz Pro | | JavaManOne0 -
Exporting ranking stats; errors
previously, when I went to export the weekly rank stats, the csv opened into excel and everything was fine. This week, one campaign data will not export other than purely in comma deliniated and will not translate to colums in ecel. A second campaign does export directly to excel but the data showing (on 27th June) is for the 22 february (report created march 1). so all my spread sheets for position analysis are up the creek and I cant even get last weeks positional report on my key domain. any thoughts??
Moz Pro | | Citybase0