Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Since implementing GDPR, has anyone seen website traffic plummet?
On December 1 and 2, I implemented a cookie banner on 5 of my Wix sites (https://www.sanitationsolutionsinc.com is one example) in order to be in compliance with GDPR, CCPA, and LGPD. Since then my traffic according to Wix Analytics and Google Analytics has plummeted. Anyone else have the same issue? How did you fix it?
Reporting & Analytics | | Jason_Taylor0 -
New website server code errors
I launched a new website at www.cheaptubes.com and have recovered my search engine rankings as well after penguin & panda devestation. I'm continuing to improve the site but moz analytics is saying I have 288 medium issues and i see the warning "45% of site pages served 302 redirects during the last crawl". I'm not sure how to fix this. I'm on WP using Yoast SEO so all the 301's I did are 301's not 302's. I do have SSL, could it be Http vs Https?
Reporting & Analytics | | cheaptubes0 -
How can I see what Google sees when it crawls my page?
In other words, how can see the text and what not it sees from start to finish on each page. I know there was a site, but I can't remember it.
Reporting & Analytics | | tiffany11030 -
Has anybody else had unusual /feed crawl errors in GWT on normal url's?
I'm getting crawl error notifications in Google Webmaster tools for pages that do not exist on my sites?! Basically normal URL's with /feed on the end.. http://jobs-transport.co.uk/submit/feed/ http://jobs-transport.co.uk/login/feed Has any body else experienced this problem? I have no idea why this is happening. Simon
Reporting & Analytics | | simmo2350 -
Irrelevant page with high bounce rate
I have a page on my site, www.waikoloavacationrentals.com/kolea-rentals/floor-plans, that gets me roughly 17% of my traffic. That being said it is not really relevant traffic because it comes from the search term "floor plans", which really has nothing to do with Hawaii vacation rentals, which is what I do. My question is does Google know how to figure that out when they are looking at my stats or is there a way to let google know that that page probably should not show up for that search phrase? On the positive, they are nice floor plans and if someone is searching for ideas for floor plans and see one of them in google images then it probably could help them, but it really is not relevant to my business. It has a 80% bounce rate, but does have an average time on page of 1.5 minutes, which is a fair amount for what is there.
Reporting & Analytics | | RobDalton0 -
Identifying conversion rate for product
Hi, I need to identify the conversion rate for a product, lets call it a spanner. I have 100s of spanner product urls and I ensured that the url protocols must include the product name e.g /red_spanner so its easy for me to work out the conversion rate in analytics for all my spanner pages as I just add 'spanner' to the landing page filter, hit the ecommerce tab and bingo. What I cant figure out is how to work out the conversion rate for all spanner sales which includes alot of sales which didnt originate on spanner pages e.g. home page > search result > checkout. Theres 1000s of variations of this e.g. email > home > search > product page > checkout. How can I work out the total conversion rate for spanners which needs to include: people landing on a spanner page and people who didnt arrive at a spanner page and did a search with eventually got them to the spanner pages. Hopefully its not as complicated as I think! Thanks in Advance!
Reporting & Analytics | | AndyMacLean0 -
Conversion Rate Question: Should I Measure Visits or Unique Visits?
When you measure conversion rates, is the equation: conversion rate = visits/conversions or conversion rate = unique visits/conversions I ask because it can actually make a pretty big difference in the conversion rate. For example, if you visit my ecommerce website 100 times before buying something (and assuming you're my only visitor), then my conversion rate is 100% _if I'm determining conversion rates by unique visits/conversions. _However, it's only 1% _if I'm determining conversion rates by visits/conversions. _Wow! Now this is clearly an extreme example, but it should serve to illustrate the point that in more reasonable cases, the way the data is measured can have a potentially significant impact on the conversion rate. Is there an industry standard for this? Am I missing something really basic? Also, here's a little bit of context for the question: I run an ecommerce website powered by the Magento CMS and I'm trying to measure my conversion rate in Google Analytics for individual products. Google Analytics shows me my site wide conversion rate, but apparently I have to do some customization in order to measure conversion rates on the product level. That's fine, but I want to make sure I'm measuring my product conversions in a standard way. Thanks for any and all help! Adam
Reporting & Analytics | | Adam-Perlman0