Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need your Opinion on Bounce Rate Analysis
I'm currently doing a bounce rate analysis for our resource pages. These are information article pages - mix of plain texts and those containing either images, infographics, videos or even podcasts. By the way, I did search for bounce rate topics here, but I felt like i still need to post this. Unless I've overlooked a similar post, my apologies. It's a first for me to do an in-depth BR analysis, so I need to clarify few things. What is a good or bad range bounce rate? Is there even a range comparison? Like when can you say a bounce rate is high for an information type page? I've read some stuff online but they're confusing. What other Analytics factors should I consider looking at together with bounce rate? For pages (which purposely educate visitors) with high bounce rate, can you guys suggest tips to improve it? I would appreciate and value any advise. Thanks a lot!
Reporting & Analytics | | ktrich1 -
Main Website Redirects to Mobile Website, Mobile Website counts this as direct traffic, is there a way to tell what the source/medium is?
Hello, The situation is that someone is arriving on my main website https://www.example.com and being redirected to http://m.example.com. When this happens my analytics says that the traffic is all direct coming to my mobile site. However, I know people clicking on my google cpc, and some google organic users are hitting the main website and being redirected. Before we didn't have as good of a redirect on our main website so I could tell organic and cpc traffic coming in, now my main website has a huge drop in these categories because they are redirecting to mobile but I can't tell on my mobile how much traffic from each is going to the mobile site. Is there a way to fix this? Is it because my main website is https:// and mobile is a http:// (as I know that sometimes makes traffic direct) or is it a bigger problem that can't be resolved? Thanks
Reporting & Analytics | | oxfordseminars0 -
Open internal links in a new tab increase bonus rate?
Hello! This week I used a simple method to reduce my blog Google Analytics bounce rate. My blog all the posts are guides, in order to follow them, user need to download a zip file (same zip file). Otherwise they can't. Therefore I added a separate blog post to download all the necessary files. As a result of that I can reduce my bounce rate from 62-70% to 45-50% level. Now I'm thinking to open that zip file download page in a new tab. If I open my blog zip file download page, in a new tab. It will again increase my bounce rate? I reduced my bounce rate using that zip file download page. Thanks!
Reporting & Analytics | | Godad0 -
I have two campaigns that are only crawling one page, why is this?
I have a total of three campaigns running right now, and two of them are only crawling one page. I set the campaigns up the same, what is the problem?
Reporting & Analytics | | SiteVamp0 -
Is there a tool to automatically gather website SEO data?
I am looking for a tool that will crawl a website and create a spreadsheet listing out all key data such as title, meta description, etc. Anyone know of an available tool to do that?
Reporting & Analytics | | jfeitlinger0 -
How effective is OSE in crawling press release links?
How effective is OSE in crawling press release links? We have released a few press releases recently (over the last couple of months) and OSE doesn't seem to have found them.
Reporting & Analytics | | deuce1s1 -
I made 18 websites and the traffic keeps going down over 3 months
I made 18 websites, and have used a analytics web app called piwik. You can google it, but basically it is like google analytics. I have done nothing for the websites no links, no updates. I did do the onpage optimization extremely well. At first I had daily traffic over all the websites at about 200, then like a month went by and it was at 100, then another month has gone by it is hovering around 30 visits -- This is total traffic across all the websites. In addition my websites were ranking much better and alot of them were coming up together in the results in a single google query, now this is no longer true, only one or maybe two come for the same google query and they come up lower in the serp ranking ie. before it was 1st place now 3rd for example, so traffic has decreased respectively. Anybody can tell me what I can do, to regain the positions and traffic I had before.
Reporting & Analytics | | mickey110 -
How do you add Facebook Insights to Multiple Websites?
If I have 4 top-level domain websites all for the same corporate company but with different focuses (ie/ engineering, technology etc.) and I want to add Facebook insights, should I add the same meta tag to all websites or should I create separate meta tags? If I create seperate ones, the issue is that we have one main facebook page that has multiple admins that I would like to give access to but it looks like you can only associate one website to one facebook page. So from what I can tell, if I were to create separate ones, I would have to setup fake facebook pages or fake profiles in order to get different facebook insights meta tags? I'm hoping there is a better way... or maybe I should just put the same meta tag code on all 4 websites? Thanks for your help!
Reporting & Analytics | | randstadsocial0