Cannot Crawl ... 612 : Page banned by error response for robots.txt.
-
I tried to crawl www.cartronix.com and I get this error:
612 : Page banned by error response for robots.txt.
I have a robots.txt file and it does not appear to be blocking anything
Also, Search Console is showing "allowed" in the robots.txt test...
I've crawled many of our other sites that are similarly set up without issue.
What could the problem be?
-
Thank you everyone... I'm learning! And you are helping!
-
Great - just checked the robots.txt with web-sniffer & shows a 200 status now so crawl shouldn't be an issue.
Dirk
-
I think I figured it out... For some reason, robots.txt was set at 600...I changed it to 644... I will run crawl again... Thanks.
-
Thank you for the responses. Can you give me any direction on how to correct this? I am lost
-
Your robots.txt renders in a browser - but from technical perspective it generates a 403: Forbidden (check http://www.cartronix.com/robots.txt with web-sniffer.net)
Moz will not crawl if your robots.txt is returning a 403 (see answer from Chiaryn Miranda / Moz on https://mza.bundledseo.com/community/q/without-robots-txt-no-crawling
Quote: "The only commands from the http responses that we consider to block our crawler from accessing a site would be a 403: Forbidden error or a 5xx error."
Dirk
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz can't crawl my new website?
We had a new website go live at the end of April - I keep requesting crawl tests but I get this in the excel copy... URL Title Tag
Moz Bar | | RayflexGroup
http://www.pvc-strip.co.uk 602 : Page redirects to a URL outside the scope of this campaign. I always list the website as https://... but the crawl always returns the http:// version. Not sure what I can do to make sure the website can be crawled?0 -
804 : HTTPS (SSL) error encountered when requesting page.
Hi there, I am attempting a crawl test and am getting "804 : HTTPS (SSL) error encountered when requesting page." with no further information to debug the issue. Site seems to be fine as far as SSL configuration goes. How can we get more debugging information? Thanks,
Moz Bar | | RickEH
Rick0 -
On page grader not working with Squarespace?
The on page grader tool does not seem to be able to access Squarespace websites. I've tried several website urls that are all hosted on square space with the same result "Sorry, but that URL is inaccessible." Is this a known incompatibility or a bug? examples: lightningjar.agency (a site I manage) flotrend.com (a site that a friend manages) ollymoss.com (a site chosen at random from SquareSpace.com featured websites list)
Moz Bar | | kevindpeckham1 -
Signed up for moz reports - have received Moz error report - need someone who is capable to take report and perform cleanup edits within Joomla site?
Looking for someone in the US - please contact me at [email protected] If available and interested in task. Thanks Mary
Moz Bar | | PortlandWebDesign0 -
Crawl Test cannot be seen on my PC. Using Windows 8.
I received and downloaded my Crawl Test. When I try to open it, my pc says "This app can't run on your PC. To find a version for your PC, check with the software publisher". I'm running Windows 8. Can I view my Crawl Test with my PC? Is there a work-around for this issue? Update I can apparently open my Crawl Test and view it as an Excel Spreadsheet. But when I download it and choose Save As, it saves it as a MS-DOS Application. This is my very first Crawl Test and I am not sure if I am doing everything right.
Moz Bar | | jameskoby010 -
Correcting a 4xx on my crawl report
How can I correct a 4xx error on my crawl report. This page no longer exists. What can I do?
Moz Bar | | henne0 -
Duplicate page content
Hi guys the feedback form my campaign suggests I have to much duplicate page content. I’ve had a look at the CSV file but it doesn’t seem to be abundantly clear as to which pages on my site have the duplicate content. Can anyone tell which columns I need to refer to on the sheet, to ascertain this information. Also if the content is only slightly different, will Google still consider it to be duplicate? I look forward to hearing from you
Moz Bar | | Hardley1110