When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?
-
Hi guys,
When I try to run a site crawl on Moz it returns an error saying that it has failed due to an error with the robots.txt file.
However, my site can be crawled by SEM Rush with no mention of problems with roots.txt file issues.
My developer has looked into it and insists their is no problem with my robots.txt and I've tried the Moz crawl at least 6 times over an 8 week period.
Has anyone ever seen such a large discrepancy between Moz and SEM Rush or have any ideas why Moz has this issue with my site??
TIA everyone
-
Hi there!
Thanks so much for the great question! I'm so sorry you're having trouble getting your site crawled. Without knowing the exact site you're working with I can't say for sure what's going on but I can offer some suggestions that may help. If you're seeing the crawl is successful with other tools, there may be a bot-specific setting or directive that is banning our crawler, rogerbot, from your site. This may be coming directly from your server or it may be listed in your robots.txt file.
I have a troubleshooting guide here that may help- it outlines some other common issues that can keep rogerbot from being able to move forward with the crawl.
If you're still having trouble, please feel free to send an email on over to [email protected] with the site you're having trouble with. That way we can take a look and see what's going on!
-
I also had this issue today on my site. Haven’t tried Semrush but got the same error that you are describing when I used Moz
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Is there anyway to cancel moz local?
Is there anyway to cancel moz local? I can not figure it out. I've tried for hours now.
Getting Started | | SouthBayWebs0 -
Domain Authority hasn't recovered since August
I really need some major advice on this one. Back in September, I asked a question on here as follows: "A client wanted to change their domain name, which we have now done. The site content itself is exactly the same. We put 301 redirect links in so that Google searchers would redirect from the old site to the new one. However Moz then said that it couldn't crawl the old domain because of the redirects and advised creating a brand new campaign for the new domain. We have done this but now Moz says that the domain authority of the new site is 2 (it was 14 on the old domain)." My original question and the answers I got are here: https://mza.bundledseo.com/community/q/new-domain-wipes-out-domain-authority). Generally the responses I got were that we should give Moz time to crawl the new domain and process all the "new" pages. It is now February, ie 6 months after the domain rename, and on Moz the site still has a DA of 2. It seems like 6 months is enough time to wait. We checked all the recommended guides and believe we have done it all correctly. I really don't know what to do now. Can anyone help or have a quick look and work out why this is so bad? Specifics are:
Getting Started | | mfrgolfgti
old domain: https://ryemeadcleaning.co.uk
new domain: https://ryemeadgroup.co.uk0 -
Can't track my site, keep getting "Ooops. Our crawlers are unable to access that URL"
Hello, So i keep getting this message and I went to hurl.it and I get 200 response. But it appears its not my actual homepage bc it says the body is empty and in the title it says "COMING SOON" which is not what my actual homepage says. Does anyone know what this means?? Thank you in advance! Rena
Getting Started | | Palila-Studio0 -
How can I find out what is the list of keywords I currently use in my website?
How can I find out what is the list of keywords I currently use in my website? In other words I want to know my current state of keywords
Getting Started | | Rosalia.Perez0 -
New to Moz Pro? Join our latest free webinar this Friday!
Hello everyone! We'll be holding a webinar on tomorrow to help new members learn about what all Pro has to offer, show some off our most popular tools, and get you comfortable with the dashboard. Register here: https://www3.gotomeeting.com/register/355765902 Date: Friday, September 26th (this Friday!) Time: 10:00 AM - 11:00 AM PDT Hope to see you all there!
Getting Started | | jennita1 -
Analytics stats not showing up in MOZ Campaign report
I connected a MOZ campaign to a client's Analytics. It seems it's connected properly with the same Analytics ID # and everything, but the stats still show 0 in MOZ, even though there are thousands of visits in Analytics for the same date range. Pls help! Derek
Getting Started | | DerekM42420 -
Can not add another campaign
Im using the trial for moz however I can not add another campain. how can I add more campaign?
Getting Started | | deeplysense0 -
Fastest way to set up Moz to produce a list of easy-to-rank for keyword opps
I've used Hubspot's keyword tool previous to MOZ, and I know that they import data generated from MOZ into their tool... but I find it a much more convoluted process using Moz's more robust system. ALL I want is to dump in a long list of keyword possibilities, then see where we can easily rank for these terms, especially against our local competitors. How do I set this up???? It's not as self-explanitory as Hubspot! Thanks!!!
Getting Started | | uptongirl340