How do can the crawler not access my robots.txt file but have 0 crawler issues?
-
So I'm getting this errorOur crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.https://www.evernote.com/l/ADOmJ5AG3A1OPZZ2wr_ETiU2dDrejywnZ8kHowever, Moz is saying I have 0 Crawler issues. Have I hit an edge case? What can I do to rectify this situation? I'm looking at my robots.txt file here: http://www.dateideas.net/robots.txt however, I don't see anything that woudl specifically get in the way.I'm trying to build a helpful resource from this domain, and getting zero organic traffic, and I have a sinking suspicion this might be the main culprit.I appreciate your help!Thanks!
-
Hey there! Tawny from Moz's Help Team here.
It looks like your site's robots.txt file is returning some errors to our tools. When I try to visit the robots.txt file from the root domain, which is where our crawler starts, I get a warning that the DNS address can't be found: https://www.screencast.com/t/ezfsiyVso4B9 That same file is returning a 503 error to our crawler: https://www.screencast.com/t/ROlNo8AQz
That robots.txt file doesn't redirect anywhere, so you may want to consider putting in a redirect there to your robots.txt file at http://www.dateideas.net/robots.txt.
The reason you're seeing 0 issues reported is that we weren't able to reach your robots.txt file, so we stopped crawling and didn't have any issues to report.
I would speak to your web developer or whoever manages your site for you about making sure that your robots.txt file is fully accessible to our crawlers and can be reached in a browser.
I hope this helps! If you've still got questions, feel free to shoot us a note at [email protected] and we'll do our best to sort everything out with you!
-
This just an advice that worked for me in the past for the same issue and of course is not in the user guide. Simple a delete the website or project from Moz Pro and create a new one for the same domain, I dont know why but all the errors and Issues disappeared.
-
Well technically if it can't crawl your site, it won't be able to find any issues...
I think this may be an error with the Moz crawler rather than any crawlers - checking with site:dateideas.net pulls up some results, although they seem to be tag and archive pages at the moment, which I'd be looking to noindex to avoid duplicate results in the future... You can always check Google's crawling by making sure you've registered your site with Google Search Console, and I'd suggest also making sure you've submitted your sitemap.
I have had issues in the past with the Moz bot struggling to crawl on specific hosts, so it might be that... The new crawl seems to work fine for me at the moment, but is it possible your site was down at the moment Moz was crawling it? You can always request a new crawl and monitor your site uptime to see if there's an issue...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Can I Batch Upload URLs to get PA for many pages?
Howdy folks, I'm using advanced search operators to generate lists of long tail queries related to my niche and I'd like to take the batch of URLs I've gathered and upload them in a batch so I can see what the PA is for each URL. This would help me determine which long tail query is receiving the most love and links and help inform my content strategy moving forward. But I can't seem to find a way to do this. I went to check out the Moz API but it's a little confusing. It says there's a free version, but then it looks like it's actually not free, then I try to use it and it says I've gone over my limit even though I haven't used it yet. Anyone that can help me with this, I'd really appreciate it. If you're familiar with SEMRush, they have a batch analysis tool that works well, but I ideally want to upload these URLs to Moz because it's better for this kind of research. Thanks!
Moz Bar | | brettmandoes2 -
On Page Grader - URL not accessible
We have tried to use the On Page Grader today and it is coming back with URL not accessible for all pages on our website. We previously used the On Page Grader on Friday 10th Nov for a couple of product pages with no issues. Since then, the only changes we have made on the websites is updating some downloadable documents. We have done this several times before and it has never affected Moz. We have not changed the page URLs, and therefore do not know why it is now not working. The pages are working fine on the website with no issues. A link to one of the pages is below. http://www.processinstruments.co.uk/products/dissolved-oxygen-monitor/ Any help would be greatly appreciated.
Moz Bar | | PiMike0 -
Why is 410 (Gone) being classed as a high priority issue in crawl diagnostics?
Are high priority issues have suddenly soared by over 100 because Moz is classing 410s as high priority.
Moz Bar | | Melissabraz
Google doesn't class these as so serious, so we were wondering if anyone knows why Mos does?0 -
I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
The website is www.bigbluem.com and is a wordpress site. I'm getting the following error: 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag But what is weird is the domain it lists below that is http://None/BigBlueM.com Any advice?
Moz Bar | | TumbleweedPDX1 -
Moz crawl issues: All pages keep resolving to our "cookies not enabled" page
Upon running the Moz Pro site crawler, I noticed that I received quite a bit of duplicate titles along with 302 redirects (which is our site creating a temporary 302 to our "cookies not enabled" page). How would I get around the crawler being redirected to this page? I've never ran across this issue before, despite using the crawler with sites that use the same framework as the one thats affected. Any ideas?
Moz Bar | | responsivelabs0 -
How many links we can share in a day through bilty.com?
How many links we can share in a day through bitly.com tool? Any kind of limitation in bitly.com of the share purpose? An open site explorer how many links fetch through bilty links in a day.
Moz Bar | | surabhi60 -
How Can I intreptret The Crawl Report Resulst?
Hello, I am new to Moz and I have received 2 crawl reports. The first one was ok. I made a few changes to my site plugins, and my next crawl report came up with 41 4XX errors. Basically, a lot of my posts. I went back to my plugins and saw the following plugins: 404 redirect plugin & Utlimate Tiny MCE I reactivated both. I am presuming that these must have caused the issues or maybe my site was hacked. I re ran a crawl this morning, but I don't know what the different headings mean or how to understand the report. Can anyone advise? My site is new and just started to go up the rankings...so quite disappointed with this set back. regards Chriss
Moz Bar | | chrisspell0 -
Moz bar issues? Does anyone else have them?
At times it's unbelievably slow and other times it doesn't show any link data. It's not just on my computer, it's on other office computers too. I'm using Google Chrome and have tried removing the mozbar and reinstalling from the chrome bar.
Moz Bar | | iresources2