Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need your Opinion on Bounce Rate Analysis
I'm currently doing a bounce rate analysis for our resource pages. These are information article pages - mix of plain texts and those containing either images, infographics, videos or even podcasts. By the way, I did search for bounce rate topics here, but I felt like i still need to post this. Unless I've overlooked a similar post, my apologies. It's a first for me to do an in-depth BR analysis, so I need to clarify few things. What is a good or bad range bounce rate? Is there even a range comparison? Like when can you say a bounce rate is high for an information type page? I've read some stuff online but they're confusing. What other Analytics factors should I consider looking at together with bounce rate? For pages (which purposely educate visitors) with high bounce rate, can you guys suggest tips to improve it? I would appreciate and value any advise. Thanks a lot!
Reporting & Analytics | | ktrich1 -
Bounce Rate Extremely Low
Bounce rate for one of my client's website has drastically reduced. Its below 5%. Have ensured that GA code is placed only once. Any help or tips in identifying the root cause is much appreciated. Shall be happy to share the URL if required.
Reporting & Analytics | | seoninj0 -
Sudden drop in traffic since website redesign – can't spot double firing sessions pre-update
Hi all, We launched an updated website around mid-April and have seen a long-term drop in unique page views and sessions since then. Between 25-35%. Page views returned to normal within a month, however. In terms of acquisition, we've not lost referrals or direct traffic, just organic sessions. I haven't seen any significant Google updates except for 'Fred' but that was in March (pre-website update). We did have to do some 301s and fix up an HTML and XML sitemap, etc, post launch – but that's been in place for a while now so shouldn't still have such an effect? We are also struggling with load speed at the moment, working on this at the moment. Didn't think this had a such a significant impact on ranking however. We've lost mostly non-UK traffic, especially USA. We're actually climbing rankings for a lot of new keywords I'm targeting, so I can't see how our SEO has been hampered. Frustratingly, I wasn't using Search Console or Moz Pro before this update, so can't compare much in the way of keyword traffic (I know I feel very silly now). I don't know if it's a reporting issue (I wish it was). I mean we had both GTM and GA tags on the website (I didn't know this until I looked into it) but these were present post website-launch. And I haven't found enough double firing sessions or page views to suggest we used to be getting double figures anyway. Before this, we were ranking brilliantly and getting more organic traffic than ever. All the best,
Reporting & Analytics | | landport0 -
Google Webmaster Tools, about multiple entries for your website
Hi I have a doubt about Google Webmaster Tools or Central as it is call today. I remember that google recommended to have one profile of your website for each domain structure. Let me try to be more clear one profile for http://www.yoursite.com, an other for http://yoursite.com, an other for https://www.yoursite.com, etc. Then in each of them we uploaded our sitemaps and cross our fingers. Now from my experience always the complete url have better index status from the sitemap. Now my question is, today as Google requested all our websites run under https, so conserving the other profiles is affecting how google index our pages? shall we have to delete the old profiles or is better to maintain them? Thanks. Pablo
Reporting & Analytics | | FWC_SEO0 -
How does Google sort multiple websites in one GWT account?
Today I noticed one of our sub-domains listed in our Google Webmaster Tools Account moved from the 6th position to the 2nd position. Is there a reason for this (perhaps urls/sites listed at the top require the most attention)?
Reporting & Analytics | | Prospector-Plastics0 -
Why is a section of our website dropping in&out of Google SERPs?
In July 2011 we started a news section that has it's own 'subfolder' /news/ (http://www.chorder.com/news/new_gear/, http://www.chorder.com/news/gear_deals/ etc.) The whole news section is dropping in&out of Google SERP's since late October, as show in attached graph. All news texts are real deal, written by our own staff, linked from homepage. Any idea why this happens and how to prevent it? cmqky.png
Reporting & Analytics | | imventurer0 -
Why is this website with worse metrics performing better on serps?
Hi! I would like to ask if anyone has any ideas on why the second website in this analysis is getting position 2 on google and website 1 is getting position 5: http://www.opensiteexplorer.org/comparisons?site=bonobetfair.info&comparisons[0]=www.bonobetfair.com&=Compare The metrics indicate that the first website is clearly superior in (almost) all metrics. Why could this be?
Reporting & Analytics | | inmonova0 -
Site crawler hasn't crawled my site in 6 days!
On 4.23 i requested a site crawl. My site only has about 550 pages. So how can we get faster crawls?
Reporting & Analytics | | joemas990