2 days in the past week Google has crawled 10x the average pages crawled per day. What does this mean?
-
For the past 3 months my site www.dlawlesshardware.com has had an average of about 400 pages crawled per day by google. We have just over 6,000 indexed pages.
However, twice in the last week, Google crawled an enormous percentage of my site. After averaging 400 pages crawled for the last 3 months, the last 4 days of crawl stats say the following.
2/1 - 4,373 pages crawled
2/2 - 367 pages crawled
2/3 - 4,777 pages crawled
2/4 - 437 pages crawled
What is the deal with these enormous spike in pages crawled per day? Of course, there are also corresponding spikes in kilobytes downloaded per day.
Essentially, Google averages crawling about 6% of my site a day. But twice in the last week, Google decided to crawl just under 80% of my site.
Has this happened to anyone else? Any ideas? I have literally no idea what this means and I haven't found anyone else with the same problem. Only people complaining about massive DROPS in pages crawled per day.
Here is a screenshot from Webmaster Tools: http://imgur.com/kpnQ8EP
The drop in time spent downloading a page corresponded exactly to an improvement in our CSS. So that probably doesn't need to be considered, although I'm up for any theories from anyone about anything.
-
2 weeks later, 2 more large spikes of 10X average...nothing bad is happening...oh well...
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Identifying Bots in Google Analytics
Hi there, While you can now filter out bots and spiders in Google Analytics, I'm interested in how you identify a bots and spiders in the first place. For example, it used to be thought that Googlebot wouldn't appear in GA as it 'couldn't process Javascript' but now Google has announced new developments for its crawler with regards to interpreting javascript and CSS, this argument isn't as cut and dry. I'm not suggesting Googlebot appears in Google Analytics, but I am saying that you can't make the case that it won't appear only because it can't interpret JavaScript. So, I'm interested to see what metrics you use to identify a bot? For me, the mix of Users > Browser, Users > Operating System Version is still quite handy, but is it possible to identify individual bots and spiders within Google Analytics? And would Googlebot appear?
Reporting & Analytics | | ecommercebc0 -
Installing Google Analytics tracking code on Landing Pages
Hello, I have my landing pages located on a separated subdomain (LP.example.com). I've created a new property in Google Analytics for that subdomain. My question is: where do I have to place the GA tracking code ? in the header ? before I close the tag or before I close the tag ? I have another issue: I've placed the GA Tracking code on some pages in the way below:
Reporting & Analytics | | JonsonSwartz
in the body tag of that Landing page HTML Code I added this line:0 -
How Accurate Is Google Analytics Audience Location
Cioa from Latitude 53.92705600 Longitude -1.38481600 Clients love to know which part of the country there visitors originate adn cant get enough of this type of data:
Reporting & Analytics | | Nightwing
http://i216.photobucket.com/albums/cc53/zymurgy_bucket/country-level_zps41f273a3.jpg But just how innacurate is this data when you consder is getting the data from isp locations put another way if i was sat in darkets deepest hole of Paisley Scotland and my ISP was in London would i be tagged as visitor originating in London? Ta muchly,David0 -
Setting up goals - google analytics
Hi Im new user google analytics I would like to set up goal for website. I asked to basic questions: What are my bussiness objectives? Answer:make the visitor to click on a advertisment (affiliate links and affiliate banners) to be redirected to company website to finish purchase. How to set up goals? How can I track ads (affiliate banners and affiliate links) on my website?
Reporting & Analytics | | info_tipovanie-stavkovanie.com0 -
Google Analytics - In-Page Analytics
I had a strange thought waking up this morning, and was curious to hear other people's opinions on it. In Google Analytics, under Content > In-Page Analytics, Google shows what links on your site pages get clicked and how many times plus other metrics. Do you think they use that data for ranking back links so-to-speak? What I mean is, say I had a back link to my site on example.com, and example.com had google analytics installed. Google can see through google analytics whether my link has been clicked on. Say that my link gets no clicks, do you think that Google would use that metric against my site deeming it "not popular" or "not a good resource", even if example.com was a very popular site? And it could work the other way. Say my link got thousands of clicks on example.com, do you think that Google might use that to promote my site? I couldn't find any other discussion on this anywhere, so am not sure if people have already thought about this.
Reporting & Analytics | | THB0 -
Analytics/Google Keyword comparison
Hi I'm trying to establish a methodology to best show the gap between potential and realised organic keyword traffic. To obtain potential keyword traffic I'm using the Google Adwords keyword tool to derive local monthly search volumes for exact keyword matches. However, I'm confused as to which is the best way of getting a comparable metric from Google Analytics (GA). I was using custom reports and the 'organic searches' metric. However, this provides different values to a standard report selecting non-paid search in the default advanced segments. What is the best report/metric in GA to use for both organic and paid search volumes that would be comparable to the Google Adwords keyword tool. Also, I'm having problems getting my kids to eat their greens, any advice! 😉 Thanks Neil
Reporting & Analytics | | mccormackmorrison0 -
Why is Webmaster Tools not Linking Correctly Google Analytics?
I have a global site with .com as the US site and then /country for all other markets i.e. .com/uk .com/de etc. Each site has it's own WMT profile. Each site has it's own GA profile. Since Google added Search Engine Optimization to the GA interface, I can't track these sites separately under this new feature. It seems to me that it can only be associated with a single WMT profile at a time or rather that WMT can only be associated with a single GA profile at a time. The numbers for the UK and US site are identical, but when I try and link the UK WMT account with GA it removes it from the US profile and vice versa.
Reporting & Analytics | | Red_Mud_Rookie0 -
Regular Expressions in Google Analytics
I want to use the Google Analytics landing page reports to look at the bounce rate of top level pages excluding the homepage. So pages with urls: www.example.com/example Does anyone know a regular expression that will allow me to do this? Just to clarify I do not want to look at the bounce rate of the homepage or any pages deeper than www.example.com/example e.g: www.example.com/example/example www.example.com/example/example/example etc Thanks in advance
Reporting & Analytics | | CPLDistribution0