Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi Harini,
Jo from the Moz help team here.
I've had a look at your site and it looks like there is something server side that is blocking our bot.
When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1
https://screencast.com/t/J3hhDTCM
I'm also seeing this message in this third party tool.
"The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)"
All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through.
I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2
You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2
You'll also want to make sure you are not blocking AWS (Amazon Web Services).
Best of luck!
Jo
-
Thank you Andy . But, the problem is MOZ crawler was unable to crawl the website even though the line " Allow: / " was present in the robots.txt.
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?color=
Disallow: /?manufacturer=
Disallow: /?filter_material-fabric=
Disallow: /?filter_color=
Disallow: /?query_type_color=
Disallow: /?filter_size=
Disallow: /?taxonomy=
Disallow: /?view_mode=
Disallow: /?query_type_material-fabric=
Disallow: /?orderby=
Disallow: /?source_id=
Disallow: /?source_tax=
Disallow: /?shop-2__trashed?
Allow: /wp-admin/admin-ajax.phpSitemap: https://www.khadination.shop/sitemap.xml
this was the previous version of robots.txt that were been used ....
-
Hi - As Andy has said, you're not allowing Moz to crawl the site.
Read up on Rogerbot here: https://mza.seotoolninja.com/help/guides/moz-procedures/what-is-rogerbot
-
Hi there,
You forgot the most important thing. You're disallowing a lot of things but not allowing access in the first place.
Allow: /
add this on line 2 of your robots.txt file.
Good luck
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
In Moz's Keyword Explorer/Explore By Site, Can I Exclude a Keyword From the Results?
Hypothetical question - Let's say I sell tax preparation software and I want to see the keywords that "turbotax.com" ranks highest for. Based on that list I might want to optimize for similar keywords on my site. But if I use Keyword Explorer, select "Root Domain", and enter "turbotax.com" as the root domain, the first 200 or so keywords in the list have the word "turbo". None of those interest me. Can I exclude certain keywords (like "turbo") from the list? Thanks.
Feature Requests | | CurtisB0 -
Why Moz Update has 48 days this time?
Hi There! We have been Monitoring our DA in Moz.com, from past 6 Months and we have observed that every Moz update happens between 25 to 30 days. Why is the next update has been scheduled for 48 days of time? Is there any specific reason for this? Awaiting your response Thanks Malik Zakaria
Feature Requests | | mzakaria0 -
Will MOZ be updating there tools with regards to the new Meta Description length?
Now that Google has increased the number of characters allowed for meta descriptions/snippets when will MOZ be updating the tools to cater for the new lengths? I'm sure a lot of my Meta descriptions that are being flagged as too long with disappear once updated. Cheers Lee B
Feature Requests | | lbagley1 -
How to reschedule a missed free moz pro walk through with the expert ?
I scheduled a moz pro walk through few days back, but by mistake I chose wrong timings for the same. How Can I rescheduled my free moz pro walk through now ?
Feature Requests | | Anupmehta0 -
Does Moz Pro provide a back link suggestion area for a particular site?
Is there a tool with Moz Pro that offers link suggestions for specific sites?
Feature Requests | | DenisZilberberg0 -
Way to track public's use of keywords over time in MOZ?
I apologize for the basic question but is there a way to track within MOZ the public's use of my keywords over time? My traffic understandably goes down certain times of the year -- holidays, vacation season, back to school, etc. What I want to see is whether my share of the traffic available is staying steady, slipping or even increasing. Sometimes I'll see a big spike in traffic that I suspect is related more to a general increase in traffic in my keywords than anything else. Thanks for any help.
Feature Requests | | NCCompLawyer0 -
Competitor changed URL. What data do I lose if I update Moz settings?
One of my competitor's has changed their domain URL. I want to update this in the Moz campaign settings but I get a warning that all historical data will be lost if I do. Is that really true? I replaced one of my competitors with another for a week and didn't seem to lose historical data. Incidentally, could this be a feature request to allow more competitors to be tracked? eqEhEHz.jpg
Feature Requests | | justin-brock0 -
Can I add more then 3 competitors do my SEO moz pro account?
I have a client who will need more then the three competitors tracked(possibly up to 10) is there any way to do this with one campaign or will i need to start multiple campaigns to achieve this?
Feature Requests | | BBI_Brandboost8