Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi Harini,
Jo from the Moz help team here.
I've had a look at your site and it looks like there is something server side that is blocking our bot.
When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1
https://screencast.com/t/J3hhDTCM
I'm also seeing this message in this third party tool.
"The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)"
All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through.
I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2
You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2
You'll also want to make sure you are not blocking AWS (Amazon Web Services).
Best of luck!
Jo
-
Thank you Andy . But, the problem is MOZ crawler was unable to crawl the website even though the line " Allow: / " was present in the robots.txt.
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?color=
Disallow: /?manufacturer=
Disallow: /?filter_material-fabric=
Disallow: /?filter_color=
Disallow: /?query_type_color=
Disallow: /?filter_size=
Disallow: /?taxonomy=
Disallow: /?view_mode=
Disallow: /?query_type_material-fabric=
Disallow: /?orderby=
Disallow: /?source_id=
Disallow: /?source_tax=
Disallow: /?shop-2__trashed?
Allow: /wp-admin/admin-ajax.phpSitemap: https://www.khadination.shop/sitemap.xml
this was the previous version of robots.txt that were been used ....
-
Hi - As Andy has said, you're not allowing Moz to crawl the site.
Read up on Rogerbot here: https://mza.seotoolninja.com/help/guides/moz-procedures/what-is-rogerbot
-
Hi there,
You forgot the most important thing. You're disallowing a lot of things but not allowing access in the first place.
Allow: /
add this on line 2 of your robots.txt file.
Good luck
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Mobile friendly Moz forum
Hi team! I was just wondering if there’s any plans for the forums to move to a more mobile friendly format? I love reading the latest questions and advice however I’m typically on the go. Cheers, Casey
Feature Requests | | Casey_Bryan2 -
Can Moz add an alert to email us when a competitor's site gets a new backlink?
This would be a very useful feature, and other sites are doing this, including Ahrefs.
Feature Requests | | rabbit5190 -
Why Moz Update has 48 days this time?
Hi There! We have been Monitoring our DA in Moz.com, from past 6 Months and we have observed that every Moz update happens between 25 to 30 days. Why is the next update has been scheduled for 48 days of time? Is there any specific reason for this? Awaiting your response Thanks Malik Zakaria
Feature Requests | | mzakaria0 -
Moz crawler is not able to crawl my website
Hello All, I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. " We changed the robots.txt file and checked it . but still the issue is not resolved. URL : https://www.khadination.shop/robots.txt Do let me know what went wrong and wjhat needs to be done. Any suggestion is appreciated. Thank you.
Feature Requests | | Harini.M0 -
Moz's New Video Ads on YouTube
I've enjoyed Moz's new video ads and want to share them with colleagues, however am unable to find the vids anywhere other than when they're delivered before a video plays on YouTube. Are they available on demand? If anyone is unfamiliar with the videos I speak of, they are new ads for Moz products like Keyword Explorer, Mozbar, and MozCon. Thanks in advance!
Feature Requests | | Choice0 -
Including "Volume" on the Tracked Keywords Overview Page in Moz Pro
I've searched Q&A but I did not see where this question has been asked before, so hopefully I'm not repeating a question that has already been asked (and answered). For the past couple of years I have been using Conductor Searchlight as our SEO toolset, however recently I have made the decision to switch to Moz Pro. My question is regarding the Tracked Keywords Overview Page. In the current display, the Overview Page shows Keyword, Location, Universal Results, Rank, URL and Optimize. Has there every been any consideration to also include Volume? One of the nice features of CS Keyword Overview page was that I was able to view the rank of all my keywords, and the volume of each, at the same time. We currently track 275 keywords. Some keywords are obviously more important than others. The CS Overview Page allowed me to quickly scan through the list of keywords, see the ones that were within striking distance of page one (by rank), and since I could also see the volume at the same time, it also allowed me to prioritize which keyword to focus on. If two keywords were both ranked "11", but one had triple of volume of the other, it makes sense to focus on the one with the greater volume. From Moz's Tracked Keywords Overview Page I can drill down and get the "Volume" information that I'm looking for (and lots of other really good information), but I have to do it for each individual keyword. Moz Pro is a great SEO tool and the information available is amazing. The switch from CS has been an easy one, but CS did have some features that I liked, and being able to see both rank and volume on the Overview Page was one them. Any information, suggestions, or work-arounds regarding my question is appreciated. Regards Scott
Feature Requests | | WScottDuncan0 -
Do MOZ have an affiliate platform?
Do MOZ have an affiliate platform? If so, it's hiding from me, would somebody care to point me the right way? Shall I assume this is the page / they MOZ don't have a direct affiliate program? https://mza.seotoolninja.com/partners Many Thanks
Feature Requests | | Whittie0