Moz unable to crawl my Zenfolio website
-
Hey guys,
I am attempting to optimize a website for my wife's business but Moz is unable to crawl it. Zenfolio is the web hosting service (she is a photographer).
The error message is:
**Moz was unable to crawl your site on Apr 1, 2019. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
I did read the troubleshooting guide but nothing worked.
My robots.txt file disallows a few bots, but not roger bot. Anyone have any idea what is going on? Or do I need to request server logs from Zenfolio?
Thanks
-
It appears that Zenfolio really doesn't want Moz crawling their site for some reason. They want me to jump through hoops to get it to work and, being an ultra small business, cant justify the effort to get it working.
Guess I could always migrate to another web host.
Thanks for the replies.
-
Hi OP.
Hopefully you got in touch with Dave and the Moz team, and you have got your answer!
I haven't got a direct answer but from experience of Moz and other crawler softwares, my money would be on a temporary issue with your server being unresponsive.
As per the "Moz Can't Crawl your site guide, as it states, if it can't read your robots.txt file, then "Roger" won't crawl your site.
This is my simplest explanation I have
Hopefully it's resolved soon!
P.S
I looked at Zenfolio, if it's a self-hosted CMS system, it might be worth checking they haven't got any special rules in place to block Moz and other popular crawlers
-
Hey! Dave here from the Help Team
Can you write into us at [email protected] with the site that is having crawl issues? We can take a look and help you out.
thanks!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz DA Issue
Hey Community, This is my site which is the Best Drifitng in Australia. I am getting a problem and my site's DA is unable to show using. I want to see it as soon as possible, although I am using a premium tool. Thanks
Getting Started | | jndjue40 -
Why Moz DA of my website not increasing?
Hi, I have two queries which are related to Moz DA and Moz Spam Score. 1. I have created my website 6 months ago and also do proper SEO of my site. Now according to Ahrefs my DR is 47 and also shows me some organic traffic, Organic Keyword of my site. But when I've checked my site in Moz.com it shows me the DA is only 1 and also there is no organic keywords and no backlinks. Why my DA is not increasing and stuck in 1 position? When will DA increase and please guide me how I can Improve DA of my website? URL: https://www.fasterapk.com 2. My second query is, what is the Moz Spam Score? Is it important for a website and which factors Moz used to create a Spam Score of any website? Experts guide needed. Thanks, overview?site=https%3A%2F%2Ffasterapk.com⌖=domain
Getting Started | | dekho870 -
Why has my website stopped working?
Signed up to Moz and my site is now down as it's "gathering data". Is this normal?
Getting Started | | BurrowsPT0 -
What Moz tool is best to find reasons google has not spidered by site
I just joined Moz and am trying to use the tools however, when I attempt to do so every link comes to a that only allows me access to post questions here. If anyone can tell me what tool is best to find reasons google has not indexed my site, I would greatly appreciate the help. Also if anyone knows why I am keep getting routed to this forum when I try to use any of the tools, I would also appreciate help with this. So far Moz is very frustrating.
Getting Started | | Johndeeray19640 -
Back on Moz
I quick my seomoz subscription a long time ago. I got an invite to the new moz analytics. I don't see anything new at first glance. What's changed?
Getting Started | | joseph11790 -
What are the solutions for Crawl Diagnostics?
Hi Mozers, I am pretty new to SEO and wanted to know what are the solutions for the various errors reported in the crawl diagnostics and if this question has been asked, please guide me in the right directions. Following are queries specific to my site just need help with these 2 only: 1. Error 404: (About 60 errors) : These are for all the PA 1 links and are no longer in the server, what do i do with these? 2. Duplicate Page Content and Title ( About 5000) : Most of these are automatic URL;s that are generated when someone fills any info on our website. What do I do with these URL;s. they are for example: _www.abc.fr/signup.php?_id=001 and then www.abc.fr/signup.php?id=002 and so on. What do I need to do and how? Plzz. Any help would be highly appreciated. I have read a lot on the forums about duplicate content but dont know how to implement this in my case, please advise. Thanks in advance. CY
Getting Started | | Abhi81870 -
How do you "Moz Crawl" a website? Newbie...
Hi everyone; I've used Screaming Frog in the past and it's simple: you enter the URL to the box and click "start" and.. voila: as the button says, the crawling starts. I've had the Pro version of Moz for a while now and haven't really 'done' anything with it. I'd like to crawl a website and thought it would be as easy as it's always been with Screaming Frog... but, for some reason, I can't find the 'way' to do it. I find it really frustrating especially cause I feel like an idiot going around in circles thinking I'm missing something really obvious... Until I realised the only solution was to ask here! So... how in the world do you crawl a website using Moz tools? (Pro version) Thanks!
Getting Started | | patrihernandez1
