Cannot Crawl ... 612 : Page banned by error response for robots.txt.
-
I tried to crawl www.cartronix.com and I get this error:
612 : Page banned by error response for robots.txt.
I have a robots.txt file and it does not appear to be blocking anything
Also, Search Console is showing "allowed" in the robots.txt test...
I've crawled many of our other sites that are similarly set up without issue.
What could the problem be?
-
Thank you everyone... I'm learning! And you are helping!
-
Great - just checked the robots.txt with web-sniffer & shows a 200 status now so crawl shouldn't be an issue.
Dirk
-
I think I figured it out... For some reason, robots.txt was set at 600...I changed it to 644... I will run crawl again... Thanks.
-
Thank you for the responses. Can you give me any direction on how to correct this? I am lost
-
Your robots.txt renders in a browser - but from technical perspective it generates a 403: Forbidden (check http://www.cartronix.com/robots.txt with web-sniffer.net)
Moz will not crawl if your robots.txt is returning a 403 (see answer from Chiaryn Miranda / Moz on https://mza.bundledseo.com/community/q/without-robots-txt-no-crawling
Quote: "The only commands from the http responses that we consider to block our crawler from accessing a site would be a 403: Forbidden error or a 5xx error."
Dirk
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why page load time is different in google webmaster vs what is displayed in moz?
When I analyze the site through Moz tool and compare the results with google webmaster, I am not able to figure what why Moz does not report the slow pages. Fro example this page has an avrage LCP of 3.0 sec https://www.collegehippo.com/graduate-school/programs/gre-score-business-analytics-data-analytics When I see the report in moz, it does not point to any such issue. Should I be worried about what google reports and try to fix the page?
Moz Bar | | etattva0 -
Why do I have so many site crawl issues with my site?
Hi there, Our website is www.mormonhub.com. We have many other websites that we own as well. For some reason, our website has tens of thousands of Site Crawl Issues. We have tried looking into what's causing the problem, but can't figure it out. We do use Wordpress for our website. Any help would be greatly appreciated. Thanks! wFL8I
Moz Bar | | ThridHour0 -
Canonical in Moz crawl report
I'm wondering if the moz bot is seeing my rel="canonical" on my pages. There are 2 notices that are bothering me: Overly Dynamic URL Rel Canonical Overly Dynamic URL - This notice is being generated by urls with query strings. On the main page I have the rel="canonical" tag in the header. So every page with the query string has the canonical tag that points to the page that should be indexed. So my question...Why the notice? Isn't this being handled properly with the canonical tag? I know I can use my robots.txt or the tool in Google search console but is it really necessary when I have the canonical on every page? Here is one of the links that has the "Overly Dynamic URL" notice, as you can see the the canonical in the header points to the page without the query string: https://www.vistex.com/services/training/traditional-classroom/registration-form/?values=true&course-title=DMP101 – Data Maintenance Pricing – Business Processes&date=March 14, 2016 Rel Canonical - Every page in my report has this notice "Using rel=canonical suggests to search engines which URL should be seen as canonical". I'm using the rel="canonical" tag on all of my pages by default. Is the report suggesting that I don't do this? Or is it suggesting that I should? Again...why the notice?
Moz Bar | | Brando160 -
On-Page Grader supporting HTTPS?
Hi there, I am getting the error "Sorry, but this URL is inaccessible" on the Moz On-Page Grader. I have tested with multiple URL's but any that link to my site give the error. I have noticed my site's URL's redirect to https so my question is..is the on-page grader in-compatible with https? Cheers!
Moz Bar | | ToteIreland
Ross0 -
On-page Grader API?
Hi guys, I have a massive list of keywords i want to optimise against a specific URL. I'm checking them manually with the MOZ on-page grader to determine if the keyword is optimised for the URL (Grade A score). I was wondering if there is a way to pull the scores automatically for each URL with the MOZ api? Or does anyone know of any tools which can automate this process? Cheers, Chris
Moz Bar | | jayoliverwright1 -
On-Page Grader is not counting H2 and H3 Keywords
I used on-page grader in one of my webpages and it showed the limit of 15 keywords used. The problem is that it counted only when the keyword appears in the Title, URL, Meta Description, H1, Body and IMG ALT. If you take into account the H2 and H3 tags, the count will go to 22. Is there a problem with the On-Page Grader Tool or it's not a problem to go above 15 keywords if they are in H2 and H3? Here's the link for the page: http://www.tororadar.com.br/acoes/bbse3-bb-seguridade/cotacoes-bovespa Keyword: BBSE3 Thanks in advance.
Moz Bar | | Guilhermealvesc0 -
We Launched a new site and Rogerbot is still reporting on links/errors from the old site, is there a way to clear those out?
We are mostly a Branding agency, and have not put a lot of effort into SEO for ourselves... SEO tends to take a backseat to design most of the time, making it a little difficult for me at times when it comes to SEO. We recently launched a new site, http://Roninadv.com/ and the developer and I have done quite a bit of work to make it work well for Google. I was really looking forward to a new crawl report from Roger, but alas, It's like Roger crawled the old site? The new site has been up since last Monday. Is there a way to clear out the old errors? Do I just need to give roger more time?
Moz Bar | | PaulRonin0 -
Comparing details of current on-page grade reports with previous reports
Hi Whilst i've set up reporting for on-page grader when it comes through on report it just shows any changes in grade and rank, what i also want to look at is changes in the detail/components of current report compared to previous reports, is this possible historically (over the previous reporting periods) ? There doesn't seem to be any kind of archive that i can find to previously carried out on-page grade report ? Cheers Dan
Moz Bar | | Dan-Lawrence0