Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Since implementing GDPR, has anyone seen website traffic plummet?
On December 1 and 2, I implemented a cookie banner on 5 of my Wix sites (https://www.sanitationsolutionsinc.com is one example) in order to be in compliance with GDPR, CCPA, and LGPD. Since then my traffic according to Wix Analytics and Google Analytics has plummeted. Anyone else have the same issue? How did you fix it?
Reporting & Analytics | | Jason_Taylor0 -
Website Essentially Delisted and Top SERPS Removed
So what a morning.... Did a search for our website and looked to see how we were doing with a couple of search terms and it turns out our website is GONE. I mean not a slap down or fluctuations in positioning but we are literal gone. I see that the contact page is pulls up on page 8 but effectively we dont exist. This is so odd and strange that i feel that this is a dream. The fact that we dont pull up on any search terms anymore is of a concern that this is something more than a penalty. What on Earth could have caused this? This is fluke? Can i call Google? Seems that index page has been removed from the search results Website www.catdi.com Catdi Printing Search Terms Houston Printing Houston Direct Mail EDDM Printing EDDM Help! C
Reporting & Analytics | | CatdiPrinting0 -
Bounce rate and Chrome
My client had a large spike in their bounce rate at the end of June, 2017. After spending far too long looking around Google Analytics I discovered that the bounce rate increased 5x when version 59.0.3071.104 of Google Chrome appeared on the scene. Version 59.0.3071.102 and older had a bounce rate of about 13%. Version 59.0.3071.104 increased to 65%. Each subsequent version is about the same.FireFox, Safari and the rest are showing no significant changes.Any idea what is happening here?Many thanks.
Reporting & Analytics | | climber170 -
How do I track specific referral traffics journey through a website?
Hello, A client has asked us to track the journey each separate referral traffic visitor takes through out the website. I have had a look through analytics and am not sure how to ensure I can do this for all referral traffic visitors? Can anyone help? Thank you.
Reporting & Analytics | | mblsolutions0 -
2 days in the past week Google has crawled 10x the average pages crawled per day. What does this mean?
For the past 3 months my site www.dlawlesshardware.com has had an average of about 400 pages crawled per day by google. We have just over 6,000 indexed pages. However, twice in the last week, Google crawled an enormous percentage of my site. After averaging 400 pages crawled for the last 3 months, the last 4 days of crawl stats say the following. 2/1 - 4,373 pages crawled 2/2 - 367 pages crawled 2/3 - 4,777 pages crawled 2/4 - 437 pages crawled What is the deal with these enormous spike in pages crawled per day? Of course, there are also corresponding spikes in kilobytes downloaded per day. Essentially, Google averages crawling about 6% of my site a day. But twice in the last week, Google decided to crawl just under 80% of my site. Has this happened to anyone else? Any ideas? I have literally no idea what this means and I haven't found anyone else with the same problem. Only people complaining about massive DROPS in pages crawled per day. Here is a screenshot from Webmaster Tools: http://imgur.com/kpnQ8EP The drop in time spent downloading a page corresponded exactly to an improvement in our CSS. So that probably doesn't need to be considered, although I'm up for any theories from anyone about anything.
Reporting & Analytics | | dellcos0 -
Bounce rate consistently dropping
We started using SEO MOZ campaigns and crawl diagnostics for our site in November 2011. At that time, our bounce rate was 57.58%. From there, it has consistently dropped. Dec. 57% Jan. 24.86% Feb. 14.61% March 17.48 % April 12.35% We started out with 1,000's of errors showing up in the crawl diagnostics, down to under 50. could this be associated with the bounce rate drop?
Reporting & Analytics | | Stevej240 -
Cover my tracks / two websites on the same Analytics
Hi, I've got 2 websites and I like one of them to make some links to the other one. These two sites are on the same Google Analytics account. They have been set up 3 month ago and they have some articles on them. I've heard that Google can see that those websites are related in my Analytics and use this information to give less power to the link I could make between them. What do think about it and what do you recommend me ? Shoud I closed the Analytics account ? Or down one of the website and rebuilt it whith a different domain / host / analytics ? The thing is if I delete one of the website i'll be force to republish the articles, because there are links on it, and they're part of a deal I've already made. Thank you
Reporting & Analytics | | Spleen0 -
Goal Conversion Rate increase AND Abandonment Increase
I wonder if someone can help me understand some data I'm seeing in Google Analytics. SetUp Goal with 3 step Funnel Date Comparison is 1st-8th September & 1st-8th June The Data (pics attached) Looking at the Goal Conversion Rate I can see: September - 11.58% vs June - 10.42% an increase which Google shows as (+11.18%) For same time period Abandonment Rate is showing September - 71.3% vs June - 61.5% - an increase which Google shows as (+16.5%) The Questions How can I bee seeing both an increase in conversion rate AND and increase in Abandonment rate on the funnel? And why doesnt "(Conversion Rate + Abandonment Rate) = 100%" ? Many thanks for any help! yqwFP.png cj5dn.png
Reporting & Analytics | | TobiasM0