Different Errors Running 2 Crawls on Effectively the Same Setup
-
Our developers are moving away from utilising robots.txt files due to security risks, so e have been in the process of removing them from sites. However we, and our clients still want to run Moz crawl reports as they can highlight useful information.
The two sites in question sit on the same server with the same settings (in fact running on the same Magento install). We do not have a robots.txt files present (they 404), and as per Chiaryn's response here https://mza.seotoolninja.com/community/q/without-robots-txt-no-crawling this should work fine?
However for www.iconiclights.co.uk we got: 902 : Network errors prevented crawler from contacting server for page.
While for www.valuelights.co.uk we got: 612 : Page banned by error response for robots.txt.
These crawls were both run recently, and there was no robots.txt present. Not to mention, they are on the same setup/server etc as mentioned. Now, we have just tested this, by uploading a blank robots.txt file to see if it changed anything - but we get exactly the same errors.
I have had a look, but can't find anything that really matches this on here - help would really be appreciated!
Thanks!
-
Hey there! Tawny from the Customer Support team here!
This sounds like a juicy issue, and one I'd love to dive in and help you with! Unfortunately, without being able to take a look at your campaigns and account directly, it's tough to provide specific support for these issues.
That said, if you write in to [email protected] and give us the details of what you're seeing - basically exactly what's in this question - we should be able to help investigate for you.
-
Having no Robots.txt, or a blank one, is perfectly fine (though honestly its no more a security risk than your Sitemap.xml). But your current issue is that both of your sites are returning 403 status codes at crawlers while people are still able to land on your pages. This has nothing to do with the Robots.txt file being changed or removed; just an odd coincidence. This most likely is an issue in htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Pro OnDemand Crawl fail on on WordPress site
Hello, I just can't seem to understand why OnDemand Crawl fails on further attempts only 4 pages out of 68 I am using WordPress, Divi Theme and on LiteSpeed server. Robots.txt allows rogerbot just can seem to find the issue
Moz Bar | | ChrisSanClaire0 -
Moz Crawler Causing Server Timeouts... Crawling thousands of non-existant pages with query parameters
Moz crawler is crawling all pages like this: http://www.xxxx.com/?product_count=100&product_order=desc&product_orderby=date http://www.xxxx.com/?product_count=100&product_order=desc&paged=1 http://www.xxx.com/?product_count=100&product_order=desc&product_view=grid Last month it crawled 80,000 pages on a site with less than 100 pages. Is there a way to select only certain pages to be crawled? Right now it is still crawling this site, since Monday morning and it's Tuesday mid-day. Every Monday it is causing time-outs from high band width on our server. Just getting ready to delete this client from the account unless there is a solution someone can give us. Thanks.
Moz Bar | | adirondack0 -
How to defined the most important keyword for YouTube Video Effects
I make a site https://www.videoeffects.org/. This site is about YouTube Video Effects. I want to know how to defind the most important keyword, such as: youtube, youtube video, youtube effects.
Moz Bar | | micheal90000 -
Possible bug in Crawl Issues report?
Hi all - My crawl issues report shows 3 pages with missing titles. These are just google verification files and the robot.txt file - shouldn't these be excluded? Pages with Title Missing or Emptyas of May 11
Moz Bar | | A-Drive
URL Page Authority Linking Root Domains
https://www.mysite.com/googlea87e28121c071983.html
1 0
https://www.mysite.com/robots.txt
1 0
https://www.mysite.com/google9b9dc57478f61677.html0 -
I'm getting an error when I try to preview my custom report
I'm getting the following error when I attempt to preview my custom report. I have also attached a screenshot. The moz.com page isn’t working moz.com sent an invalid response. ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITIONI need to send my client his report for a meeting tomorrow morning, and I am unable to run it. The campaign is propacusa.comf3VhmfU
Moz Bar | | chill9860 -
Getting 'Sorry, but that URL is inaccessible' error msg when trying to run On-Page Grader
I just signed up for MOZ Pro for the first time today. Tried to run the 'on-page grader' tool on some of my pages but I'm getting a 'Sorry, but that URL is inaccessible' error msg. I have verified against the robot.txt file that the pages are NOT blocking any crawlers. Can anybody help?
Moz Bar | | spinoki0 -
Way has the number of pages crawled plummeted?
Why has the number of pages crawled for our campaign plummeted in Moz Analytics – down to 729 from over 10k? Don't see any issues in Google Analytics with crawling our site.
Moz Bar | | EyeglassesGuy0 -
Where to find one off crawl report
Hello, I don't know if I am being a bit daft but I don't seem to be able to find the area where I can request a one off crawl report anymore (rather than setting up a campaign). Can someone let me know where this is now? Thanks!
Moz Bar | | RikkiD220