Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi Harini,
Jo from the Moz help team here.
I've had a look at your site and it looks like there is something server side that is blocking our bot.
When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1
https://screencast.com/t/J3hhDTCM
I'm also seeing this message in this third party tool.
"The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)"
All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through.
I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2
You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2
You'll also want to make sure you are not blocking AWS (Amazon Web Services).
Best of luck!
Jo
-
Thank you Andy . But, the problem is MOZ crawler was unable to crawl the website even though the line " Allow: / " was present in the robots.txt.
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?color=
Disallow: /?manufacturer=
Disallow: /?filter_material-fabric=
Disallow: /?filter_color=
Disallow: /?query_type_color=
Disallow: /?filter_size=
Disallow: /?taxonomy=
Disallow: /?view_mode=
Disallow: /?query_type_material-fabric=
Disallow: /?orderby=
Disallow: /?source_id=
Disallow: /?source_tax=
Disallow: /?shop-2__trashed?
Allow: /wp-admin/admin-ajax.phpSitemap: https://www.khadination.shop/sitemap.xml
this was the previous version of robots.txt that were been used ....
-
Hi - As Andy has said, you're not allowing Moz to crawl the site.
Read up on Rogerbot here: https://mza.seotoolninja.com/help/guides/moz-procedures/what-is-rogerbot
-
Hi there,
You forgot the most important thing. You're disallowing a lot of things but not allowing access in the first place.
Allow: /
add this on line 2 of your robots.txt file.
Good luck
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I connect my moz crawl to datastudio to list 404s in a table
Hello I currently export 404s and other 'critical' issues to a google sheet and publish them in a monthly report in data studio Is there a way to automate this process so that my monthly report is automatically populated with critical issues
Feature Requests | | Andrew-SEO0 -
Is there a moz tool to optimize original advertorial content?
We are launching a new module that will involve paid advertorial along with unpaid articles. Does Moz have a tool that will help us with optimizing these articles for the English speaking Hong Kong market? We are not interested in optimizing for the US or other markets as people there would not be searching for the content we are producing so the optimization would not be very relevant. Example: https://hongkong.asiaxpat.com/other/263101/guide-to-buying-and-riding-a-motorcycle-in-hong-kong/
Feature Requests | | HKPaul0 -
Will MOZ be updating there tools with regards to the new Meta Description length?
Now that Google has increased the number of characters allowed for meta descriptions/snippets when will MOZ be updating the tools to cater for the new lengths? I'm sure a lot of my Meta descriptions that are being flagged as too long with disappear once updated. Cheers Lee B
Feature Requests | | lbagley1 -
MOZ Campaigns Down?
MOZ Campaigns has been displaying a blank page in the last 2 hours. Is anyone else experiencing this outage? I have attempted to access MOZ via an iPad and an iPhone using Safari and Chrome. My geographic location is France.
Feature Requests | | Kingalan10 -
Moz Crawler failing with https redirect?
Is there a way to get the Moz Crawl Test to work with HTTPS? I just got back this error: 902 : Network errors prevented crawler from contacting server for page. Site is set up with a standard 301 to redirect http to https - or at least I certainly hope it is! Rex Swain's HTTP Header Checker took shows a standard 301. Anyone else experiencing this error? btw - this is both a specific question and an opportunity for open discussion... Thanks!
Feature Requests | | seo_plus0 -
Discrepancy in "np" within in my Moz Report?
| Keyword |
Feature Requests | | jessential
| np - /google-structured-data-update/ | In my "New Rankings & Insights | Hill Web Creations" report, while "Google structured data update" moved up significantly, at the same time, "np - /google-structured-data-update/" moved down significantly. Any insights that would offer a deeper understanding of the discrepancy and how to pull both up? If this corresponds with Google Analytics, the "np" Acquisitions > Campaigns > Organic Keywords > "Organic Search Traffic" Report, how is Moz rendering is New Rankings & Insights?0 -
Posted by Link on Moz - Broken
I wasn't sure which category to place this in as Support doesnt feature Q&A or the Moz site in general so I dropped it under other research tools to which the Q&A kind of is 🙂 Now I am not sure if it is just me and that you have rectified the issue, but when ever I click on the "posted by" links to the user. I am getting a page not found error. The links in question can be seen in my two grabs and effect all posted by links on the Q&A section of Moz. A simply trailing slash after .com "/" will do the trick 🙂 https://mza.seotoolninja.comusers/view/636129 - BROKEN
Feature Requests | | TimHolmes
https://mza.seotoolninja.com/users/view/636129 - FIXED Aj5S2Pj,YfSQEmZ#0 Aj5S2Pj,YfSQEmZ#10 -
Crawl diagnostic errors due to query string
I'm seeing a large amount of duplicate page titles, duplicate content, missing meta descriptions, etc. in my Crawl Diagnostics Report due to URLs' query strings. These pages already have canonical tags, but I know canonical tags aren't considered in MOZ's crawl diagnostic reports and therefore won't reduce the number of reported errors. Is there any way to configure MOZ to not consider query string variants as unique URLs? It's difficult to find a legitimate error among hundreds of these non-errors.
Feature Requests | | jmorehouse0