Moz could not crawl my httpS website
-
Hi,
we have a website with HTTPS, moz could not crawl it and we get "902 : Network errors prevented crawler from contacting server for page" while in logs we see moz robot access but fail after some seconds, what could be the problem, while moz can access site when it is without httpS
| 902 : Network errors prevented crawler from contacting server for page. |
-
I check and seems that your site is OK from mine side - GCE, Amazon EC2 and in Eastern Europe. There are few issues - SSL3, weak DH and files as: https://www.pouyasazan.org/IT-news/25876/css/css/css/css/css/css/css/css/css/css/css/css/css/tooltipster.css (breaking relative and absolute path), also no 404 page because everything return "200 OK" (soft 404). But they isn't critical for your site working and it works. Server works and i just hammering it with many requests.
Please contact with Moz on mail here: [email protected] or fill form there: https://mza.seotoolninja.com/help/contact/pro to investigate your situation.
-
Hi,
website is httpS://www.pouyasazan.org
-
Hi,
I have read those and investigated, those are not reasons, because first server logs shows following:
<code>Line 31915: 2015-12-23 09:23:27 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /robots.txt - 80 - 54.167.7.207 HTTP/1.1 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected]](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected])) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 422 293 32 Line 31915: 2015-12-23 09:23:27 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /robots.txt - 80 - 54.167.7.207 HTTP/1.1 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected]](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected])) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 422 293 32 Line 31933: 2015-12-23 09:23:30 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET / - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected]](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected])) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 302 0 0 362 302 33 Line 31933: 2015-12-23 09:23:30 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET / - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected]](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected])) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 302 0 0 362 302 33 Line 31953: 2015-12-23 09:23:46 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /sitemap.xml - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected]](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected])) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 16407 313 74 Line 31953: 2015-12-23 09:23:46 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /sitemap.xml - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected]](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected])) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 16407 313 74 second crawl test shows following: http://www.pouyasazan.org HTTP Status Code 302 httpS://www.pouyasazan.org HTTP Status Code 0 this shows that your crawler could connect third, when i goto http://analyti/search/on-page/4311859.1138194 i can optimiget results for keywork analys that shows your crawler could access my site</code>
-
Did you block AWS IP range or Mozbot? Or did you applied some firewall policy?
PM me your URL so i can take a look on server and network level with different tools.
-
Hello Hamed!
Here's a little more information from our Crawl Errors page:
_ 902 Unable to contact server_
The crawler resolved an IP address from the host name but failed to connect at port 80 for that address. This error may occur when a site blocks Moz's IP address ranges. Please make sure you're not blocking AWS.
This question has come up a few times here in Q&A, so I'd recommend going to the following post for further help in getting this all diagnosed:
https://mza.seotoolninja.com/community/q/odd-crawl-test-issuesI hope this helps. Thank you!
Kevin
Help Team
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawling issue
Hi, I have to set up a campaign for a webshop. This webshop is a subdomain itself. First question: The two subfolders I need to track are /nl_BE and /fr_BE. What is the best way to handle this? Shall I set up two different campaigns for each subfolder, or shall I just make one campaign and add tags to keywords? **Second question: **it seems like Moz can't crawl enough pages. There are no disallows in the robots.txt. Should I try putting the following at the top into my robots.txt? User-agent: rogerbot
Getting Started | | Mat_C
Disallow: Or is it because I want to crawl only a subdomain that it doesn't work? Thanks0 -
Multiple campaigns for one website with multiple services?
Hi. Brand new to Moz Pro today. Question about whether to set up multiple campaigns for one website that promotes multiple services. I own a law firm and we have one main location, one website, 5 attorneys, and about 8-10 practice areas we promote. Each practice area has its own page and related content in the form of guides and blogs. Website is southernmarylandlaw.com. I only have this one website to track and promote, so I initially thought I only have one campaign. But I do want to track these separate practice areas separately. Each will have separate competitors, for instance. Each will need a separate set of keywords. People tend to search for a particular type of attorney, not just for the generic "lawyers." Should I use my 10 campaigns to set up a different campaign for each practice area we promote? Am I doing something wrong by tracking my one website 10 different times in my account? If I do not do this, then how do I possibly track the various practice areas and their separate keywords and competitors? Thanks for all your help and suggestions!
Getting Started | | tuckervc0 -
My website does not allow all crawler to crawl, Now my question is that whether i need to give permission to moz crawler if yes then whaat is moz bot name?
My website does not permit all crawler to crawl website. Whether ii need to give permission to moz bot to crawl website or not? If yes what is the moz bot name?
Getting Started | | irteam0 -
Website errors?
Where can I see my domain website errors. Things like how may pages are missing meta description, duplicate title tags or broken links. I use to see it when I signed in. Now I can't find it.
Getting Started | | gsam1 -
MOZ Monthly Custom Report Question
How do I set the Moz to produce monthly report on 1st of every month. I dont see any option there. we have 50+ Clients in Moz now and we need to generate report on 1st day of all month to send them report. Its 3rd Oct today and I don't see my previous month report being generated. Can you guys help me
Getting Started | | emarketexperts0 -
Why does Moz Analytics only let you look at last month / last week?
I cant think of any other reporting product that only lets you look at the last period, am I missing a setting somewhere. I need to take a look at Feb!
Getting Started | | locusm1 -
Moz Analytics - Custom Report Question
Hello Moz! I haven't been here since the days of SEOMoz, but it's great to be back and i'm really pleased to see lot of shiny new tools at my disposal, but I do have a question in regards to functionality. If I have, for example, 15 clients in my dashboard and would like to run the same report for all of them. Assuming that I just finished creating a custom report for my first client, how can I duplicate this report without having to re-create this report 15 times? Additionally, while I am here. I have noticed that when I add a campaign, the last screen never finishes loading and the window seems to freeze. A quick refresh fixes this, but I thought i'd flag it 🙂
Getting Started | | Paul_Tovey0 -
Whenever I try to access campaigns in moz pro I get an error page
I recently signed-up for a new pro account. As I was adding my first subdomain everything was fine until I was asked to link to GA, when I clicked yes I got this error message: 403 Forbidden Now every time I click on set-up campaign I get taken to a page with nothing but the 403 Forbidden text.
Getting Started | | Toptal0