Moz not able to crawl our site - any advice?
-
When I try and crawl our site through Moz it gives this message:
Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.
Anyone got any advice on where I could go to get this sorted?
-
Hello, have the same issue and MOZ getting "Bummer no data found" for domain https://wherebuycoin.com/ . The SEMRush and Ahrefs see this site, but MOZ not, even after 2 years. What can be wrong? Where to read why MOZ not crawling my site?
Thanks!
-
Check your website's robots.txt file to ensure that it's not blocking Moz's user-agent or specific directories/pages that you want to be crawled. Update the robots.txt file if necessary to allow Moz to crawl your site.
-
Hey,
We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?
Looks like they are well aware of it though.
Thanks!
-
Hi,
I'm sorry to hear about this! Would you be able to write into [email protected] as this will allows us to investigate into this via the most convenient method.
Looking forward to hearing from you,
Eli
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I check PA in moz
i see where I can check the DA for a site, but how can I check the PA of a page?
Getting Started | | Konvertica0 -
My page can not be crawled
Hi all,Hope you could you help me here.I just seen this message but I don't know how to fix it?"Your page redirects or links to a page that is outside of the scope of your campaign settings. Your campaign is limited to pages with _____ in the URL path, which prevents us from crawling through the redirect or the links on your page. To enable a full crawl of your site, you may need to create a new campaign with a broader scope, adjust your redirects, or add links to other pages that include ig.com/de. Typically errors like this should be investigated and fixed by the site webmaster."Any ideas about how should I fix it?Thanks
Getting Started | | lauracelada23100 -
How to have MOZ site crawl pre-launch
Hi, Our new website is about to launch. We would love to have moz.com SITE CRAWL our site before launch. For issues like "missing meta description" and everything else that moz.com checks. We would love to do it before we launch. The new site is currently on a different domain than our live site. example.com <-- this is our live site. new-site.com <-- this is our "staging" server with the new site. We have a long running campaign for example.com Do we need to create a new campain for new-site.com ? Or is there some other simpler way? When we launch we will switch the site from new-site.com to example.com .. example.com will be the address for the new site.. Any ideas or suggestions? best practices? edit Forgot to say thank you for your help and input 🙂
Getting Started | | tandvarden0 -
Why is Moz.com saying that none are linking to www.oneworldcetner.eu
It still says 0 after 6 weeks if i use other tooks like http://openlinkprofiler.org/r/oneworldcenter.eu#.VhbF7ivzJ8E I do see the backlinks that i was counting on was there
Getting Started | | onewordcenter0 -
I don't believe moz is seeing everything that is on my webpage
I used the page key word grader and got an "F" Moz said that my keyword employee handbook was not in my title nor was it found in the body of my page. But when I look at the page and double check everything it is there all over the place. I am not blaming moz this is a wiz site and while I am a beginner and very well could be wrong could anyone just take a look and tell me if I am nuts or what. The web page is http://www.cestoday.com/#!employee-handbook/co0h I now have the font so big I will have to fix that. Thank you
Getting Started | | redsman9440 -
Daily crawl reports, are they wasting my time?
I am relatively new here, I have 5 campaigns. I get new crawl complete reports almost every day for all of them. Wow great, except when I check the reports nothing has changed. Even if I have gone in and changed things or fixed errors, the same ones are still there and takes 4-7 days for that work to show up. Everytime I get one of these reports I am opening them up going through and not seeing the changes I implemented the previous days before. I'll spend 20-30 minutes going over these and checking details. So the question is, Are these reports wasting my time? Are they actually new reports or am I just getting spammed repeat notices everyday?
Getting Started | | RandyFriesen0 -
Why I can't add more campaign to my moz analytics?
As we know that being a standard pro member in Moz community, we have at least 5 campaigns to add and track & as well in new moz analytics. But when I just made http://www.norwoodgreen.in website campaign an archive to make active or add one of the old himachalpackagetour.com campaign which I was unable to do WHY WHY WHY? Let me know . .? Also do guide or train me How to use Moz analytics as I was not enough familiar with it as I am habitual with in Pro tools. Best,
Getting Started | | Futura
Teginder0 -
Is there a way to force re-crawl?
1- Can we force the system to recrawl a website without having to wait for the scheduled date? 2- Please forgive me as I am new to this, but is "Moz Analytics" supposed to replace the "Pro"? Thank you!
Getting Started | | BlackTreeIT0