Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved URL Crawl Reports providing drastic differences: Is there something wrong?
A bit at a loss here. I ran a URL crawl report at the end of January on a website( https://www.welchforbes.com/ ). There were no major critical issues at the time. No updates were made on the website (that I'm aware of), but after running another crawl on March 14, the report was short about 90 pages on the site and suddenly had a ton of 403 errors. I ran a crawl again on March 15 to check if there was perhaps a discrepancy, and the report crawled even fewer pages and had completely different results again. Is there a reason the results are differing from report to report? Is there something about the reports that I'm not understanding or is there a serious issue within the website that needs to be addressed? Jan. 28 results:
Reporting & Analytics | | OliviaKantyka
Screen Shot 2022-03-16 at 3.00.52 PM.png March 14 results:
Screen Shot 2022-03-15 at 10.31.22 AM.png March 15 results:
Screen Shot 2022-03-15 at 4.06.42 PM.png0 -
Help on how to test the DA, PA of the website
Please help me. How to check the DA, PA of the website https://toolaim.com/ . I want to know any quality website to take care of it more. But at present I do not know any forum quality is good and trust. Thank you
Reporting & Analytics | | gogoanimetp0 -
Since implementing GDPR, has anyone seen website traffic plummet?
On December 1 and 2, I implemented a cookie banner on 5 of my Wix sites (https://www.sanitationsolutionsinc.com is one example) in order to be in compliance with GDPR, CCPA, and LGPD. Since then my traffic according to Wix Analytics and Google Analytics has plummeted. Anyone else have the same issue? How did you fix it?
Reporting & Analytics | | Jason_Taylor0 -
Backlinks Tracking Websites/Tools/Software
I have multiple websites that I need to keep track of their backlinks. How do you guys keep track of your backlinks? What are some cool tools that you use ?
Reporting & Analytics | | AngelosS0 -
Open internal links in a new tab increase bonus rate?
Hello! This week I used a simple method to reduce my blog Google Analytics bounce rate. My blog all the posts are guides, in order to follow them, user need to download a zip file (same zip file). Otherwise they can't. Therefore I added a separate blog post to download all the necessary files. As a result of that I can reduce my bounce rate from 62-70% to 45-50% level. Now I'm thinking to open that zip file download page in a new tab. If I open my blog zip file download page, in a new tab. It will again increase my bounce rate? I reduced my bounce rate using that zip file download page. Thanks!
Reporting & Analytics | | Godad0 -
Blog separate from website?
I have 2 domain names - 1 for a blog - the domain name contains key words related to the main subject of the blog. The 2nd is the main website with a domain name = company name - no key words. The blog and main website are about the same subject. SEOMoz link analysis indicates that the main website is getting domain authority credit from the large number of links from the blog to the main website. The blog website gets more traffic than the main website and Google Analytics data indicates that we are getting a low number of referrals from the blog to the main website. If I can increase the referral traffic from the blog site to the main website do you think the link credit from the blog to the main website warrants keeping the sites separate or do people think the blog and the main website should be under the same domain name? Please share your thoughts - I need advice!
Reporting & Analytics | | Arukor0 -
Using Clients GA account for their over all website to Optimize Their Blog? or is there a better way to compare apples to apples
I would like to optimize my clients blog. If I am using their GA account for their over all website but put in the sub folder for SEOMOz, will it detect that I only want to optimize the blog ex. http://xxx.com/blog or will I be seeing the GA for the entire site on SEOMOZ?
Reporting & Analytics | | CliffordC0 -
Bounce Rate
Howdy Mozzers Does anyone know if the 'average time on site' in Google analytics is calculated with bounce rate included? For example if you have a 50% bounce rate and your average time on site is 2 minutes the actual time would be 4 minutes as the 50% bounce rate time is classed as 0. I hope that is clear! Cheers
Reporting & Analytics | | CraigAddyman2