Crawl error robots.txt
-
Hello, when trying to access the site crawl to be able to analyze our page, the following error appears:
**Moz was unable to crawl your site on Nov 15, 2017. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
Can help us?
Thanks!
-
@Linda-Vassily yes
-
The page is: https://frizzant.com/ And don't have noindex
-
Thanks Lind and Tawny! i 'll check it
-
Hey there!
This is a tricky one — the answer to these questions is almost always specific to the site and the Campaign. For this Campaign, it looks like your robots.txt file returned a 403 forbidden response to our crawler: https://www.screencast.com/t/f42TiSKp
Do you use any kind of DDOS protection software? That can give our tools trouble and cause us to be unable to access the robots.txt file for your site.
I'd recommend checking with your web developer to make sure that your robots.txt file is accessible to our user-agent, rogerbot, and returning a 200 OK status for that user-agent. If you're still having trouble, it'll be easier to assist you if you contact us through [email protected], where we can take a closer look at your account and Campaign directly.
-
I just popped that into ScreamingFrog and I don't see a noindex on that page, but I do see it on some other pages. (Though that shouldn't stop other pages from being crawled.)
Maybe it was just a glitch that happened to occur at the time of the crawl. You could try doing another crawl and see if you get the same error.
-
The page is: http://www.yogaenmandiram.com/ And don't have noindex
-
Hmm. How about on the page itself? Is there a noindex?
-
Yes, our robots.txt it's very simple:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php -
That just says that you are blocking the Moz crawler. Take a look at your robots.txt file and see if you have any exclusions in there that might cause that page not to be crawled. (Try going to yoursite.com/robots.txt or you can learn more about this topic here.)
-
Sorry, the image don't appear
Try now -
It looks like the error you are referring to did not come through in your question. Could you try editing it?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl test
I used to use the crawl test tool to crawl websites and it presented the information in a really useful hierarchy of pages. The new on-demand crawl test doesn't seem to do this. Is there another tool I should be using to get the data?
Product Support | | Karen_Dauncey0 -
Haven't received an update on site crawl issues more than a week
Hello, my account has be scheduled to have the next updated report on 1st March. However, up till now, the latest data I have for our site crawl issues is made on 21st Feb. May I know if there is any issue related to this? Any way that i can draw the data for this week?
Product Support | | Robylin10 -
What is the difference between the "Crawl Issues" report and the "Crawl Test" report?
I've downloaded the CSV of the Crawl Diagnositcs report (which downloads as the "Crawl Issues" report) and the CSV from the Crawl Test Report, and pulled out the pages for a specific subdomain. The Crawl Test report gave me about 150 pages, where the Crawl Issues report gave 500 pages. Why would there be that difference in results? I've checked for duplicate URLs and there are none within the Crawl Issues report.
Product Support | | SBowen-Jive0 -
Moz Campaign Error
A lot of links seem to be broken or glitching with the Moz Campaigns dashboard. For whatever reason moz decided to archive my campaigns without permission. It then will not let me activate them again, forcing me to recreate them. When I go to the campaigns page, it will not show me my active campaigns, including ones that I have just made and it is impossible to get to them unless I save the campaign id and manually navigate to it in the browser. This has been tested on every web browser, so it is not some kind of user error or browser thing on my end. How can I get to my active campaigns since Moz will not show them to me?
Product Support | | Atomicx1 -
Why has my Google Analytics dropped completely from the results on this weeks crawl?
The Account code hasn't changed in either Moz or on site? Also I added 53 new keywords to my campaign and they haven't all been ranked, it's like Moz has done half a job?! S.O.S.
Product Support | | danwebman0 -
MOZ Crawl help
Our MOZ report says it crawled 1800 pages so it reports a lot of errors based on those pages. We don't have that many pages on our site. What is MOZ crawling? I updated the profile to make sure it crawls the filtered page section of Google Analytics.
Product Support | | JessiK0 -
Moz crawls
Hi all! I'm running two separate campaign crawls at the moment that I'm having a few issues with. The first is http://www.muchbetteradventures.com/. I'm tracking this at root to try an pick any problems with a previous version of the site, which is on the v1.muchbetteradventures subdomain. However, I'm only seeing 100 or so pages as being crawled in Moz, compared to thousands in Google. There's no access blocked alerts in Moz either. The second is http://www.sothebys.com/. I started this crawl at root a few days ago and no pages at all have been processed. Very odd! Any advice would be much appreciated.
Product Support | | neooptic0 -
Traffic Data Report Error: Anyone else?
I included a screenshot of the report. I had a 111 organic visits for last Mondays past week report, this Monday it is telling me I had 0. It's also telling me I have -10 non-branded keywords. I know the organic search visits isn't 0. Also, (this part I'm not sure about) but I don't see how I could have a negative amount of non-branded keywords. I would think it would just stop at 0. Has anyone had something like this happen? Or am I just missing something? Thanks for any help or comments. zK3FbG4.png
Product Support | | KempRugeLawGroup0