Moz unable to crawl my Zenfolio website
-
Hey guys,
I am attempting to optimize a website for my wife's business but Moz is unable to crawl it. Zenfolio is the web hosting service (she is a photographer).
The error message is:
**Moz was unable to crawl your site on Apr 1, 2019. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
I did read the troubleshooting guide but nothing worked.
My robots.txt file disallows a few bots, but not roger bot. Anyone have any idea what is going on? Or do I need to request server logs from Zenfolio?
Thanks
-
It appears that Zenfolio really doesn't want Moz crawling their site for some reason. They want me to jump through hoops to get it to work and, being an ultra small business, cant justify the effort to get it working.
Guess I could always migrate to another web host.
Thanks for the replies.
-
Hi OP.
Hopefully you got in touch with Dave and the Moz team, and you have got your answer!
I haven't got a direct answer but from experience of Moz and other crawler softwares, my money would be on a temporary issue with your server being unresponsive.
As per the "Moz Can't Crawl your site guide, as it states, if it can't read your robots.txt file, then "Roger" won't crawl your site.
This is my simplest explanation I have
Hopefully it's resolved soon!
P.S
I looked at Zenfolio, if it's a self-hosted CMS system, it might be worth checking they haven't got any special rules in place to block Moz and other popular crawlers
-
Hey! Dave here from the Help Team
Can you write into us at [email protected] with the site that is having crawl issues? We can take a look and help you out.
thanks!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz DA Issue
Hey Community, This is my site which is the Best Drifitng in Australia. I am getting a problem and my site's DA is unable to show using. I want to see it as soon as possible, although I am using a premium tool. Thanks
Getting Started | | jndjue40 -
8 hotel aggregate website - initial set up?
Before I get ahead of myself, I thought I'd ask the community for a quick assist: We have 8 hotels set up under a single domain aggregate website: staypineapple.com
Getting Started | | PHC-TOM
staypineapple.com/the-maxwell-hotel-seattle-wa
staypineapple.com/hotel-five-seattle-wa
staypineapple.com/the-alise-chicago-il
and so on. Should I technically set up the 8 hotels as separate tracked campaigns, complete with new branded search terms, broad search terms and such? Is this best practice? Thank you in advance for any assistance!0 -
Moz could not crawl my httpS website
Hi, we have a website with HTTPS, moz could not crawl it and we get "902 : Network errors prevented crawler from contacting server for page" while in logs we see moz robot access but fail after some seconds, what could be the problem, while moz can access site when it is without httpS | 902 : Network errors prevented crawler from contacting server for page. |
Getting Started | | Hamedkhorasani10 -
Getting started with moz
Hi i use to be a user of SEO moz before the change to just Moz, however i am struggling to navigate around and work on campaigns. I need to know what steps to take methodically from setting up a campaign, keyword research, competitors and monitoring SERPs results. in addition improving the page grade reports and analysing keyword difficulty. Whilst Moz assist you to set up some of this initially when creating the campaign i feel it doesn't seem to take you through a logical methodical step process of configuring in depth the steps of ensuring the correct settings are relevant to the campaign i.e. an outline of the steps and i need to take that follow on from each other and getting your campaign completed. for example: create campaign keyword research onpage optimisation competitor research link finder analyser run reports that related to the info provided above I feel lost in the features of moz whilst i can see they are highly beneficial putting them to use in a chronological order to ensure the the correct setup and make use of these tools. i.e. where to start and where to end currently i feel i can only find where to start and what i should do after that to make use of Moz fully is somewhat missing. Thanks in advance, any links and direct appreciated, i would alsolike to possible speak to a Moz team member regarding my account setup if possible.
Getting Started | | mari-rose0 -
I am managing an existing ecommerce website and just subscribed to the MOZ tools - what is the best rout to learning how bets leverage all the tools to optimize my site?
I am managing an existing ecommerce website and just subscribed to the MOZ tools - what is the best rout to learning how bets leverage all the tools to optimize my site?
Getting Started | | DiveNSail0 -
Multiple websites -- one for each brand
I am a MOZ newb, and didn't find answer to the following: We are an enterprise with four distinct brands, each acting as a (reasonably) independent subsidiary. I want to track their performance and report on keyword rankings for the enterprise. Is Moz Analytics capable of this? In Google Analytics, for instance, we have an account for each brand, and multiple properties within each account, and so on. I use 3rd-party dashboard tools to combine multiple accounts into a single report. Can I do something similar here. Thanks in advance.
Getting Started | | afhiWebMaster0 -
Why Moz Pro Campaign overview display not properly?
when i browse my campaign, i see like this on my computer http://i.imgur.com/7G9IRT8.jpg then when i see source code of http://pro.cdn.seomoz.org/stylesheets/campaigns/overview.css?1378244858 <html><head><title>404 Not Foundtitle>head><body bgcolor="<a class="attribute-value">white</a>"><center><h1>404 Not Foundh1>center><hr><center>nginxcenter>body>html> what happen??
Getting Started | | dimazm0 -
Moz Analytics - Reports
In the guide video (http://moz.com/help/guides/getting-started) a report button is show. However, I cannot see this button in Moz Analytics. I'm assuming this is due to the tool being in beta?
Getting Started | | David_ODonnell0