Why would I suddenly start seeing a spike in hits from particular bots (specifically rogerbot, google, bing, and yahoo)?
-
We have seen consistent network traffic over the past month, then starting yesterday, huge spikes in hits (hits as in crawls to pages causing an increase in megabytes downloaded) started coming in from Rogerbot, Google, Bing, and Yahoo. A specific example from Rogerbot is as follows:
-
rogerbot/1.1+(http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected])
-
Useragent from the bot
-
IP address: 54.226.73.52
-
Domain / hostname: ec2-54-226-73-52.compute-1.amazonaws.com
-
Physical location: United States flag United States, VA, Ashburn
We've have thought about doing a crawl-delay to prevent these bots from hitting us so hard, but that still doesn't help us answer why this even started in the first place.
Any clue on what may be going on here?
-
-
Hi Kasy, did you get to the bottom of this?
-
- No changes yet; however, it's getting worse on our end, particularly from Yahoo, so we're about to update it to add a line for crawl-delay.
- No known changes have been made with any of these.
- No changes have been made to any of our canonical or noindex tags.
- No, everything is the same.
- The only one that we have consistently crawl the site is Moz. I'm familiar with the other tools, but I haven't used them lately to crawl the site.
-
Hi Kasy, sorry to check the obvious first...
- Have there been any updates to your robots.txt file?
- Have you updated sitemaps? In Robots.txt or Google webmaster tools?
- Have you changed any meta information like canonical tags, noindex tags
- Have you changed any internal links from no-follow to follow?
- Have you got any tools regularly set up to crawl your site as Googlebot? Moz, Screaming Frog, DeepCrawl, Xenu etc.
-
Nothing unusual in any of those areas. GA is normal too. The hits/bots did come all at the same time, but since it started, it's been consistent.
-
Anything strange in your link profile or in your social media profile. Did the hits/bots come all at the same time or are spread evenly within 24 hrs?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google not detecting Hreflang
Hey everybody, We recently migrated our .co.uk to .com/en. Google for some reason is saying that the .com/en version has no hfrelang tags - even though they are clearly there and have had the same implementation as other language versions of the website. We also did a previous migration 6 months ago for the german version of our website and no hreflang problems there. We add our hreflang tags to our sitemap - which you can find here:
Technical SEO | | mooj
https://camaloon.com/en/web-sitemap.xml Any help or suggestions would be greatly appreciated!! Thanks 🙏0 -
Webmaster tools not showing links but Moz OSE is showing links. Why can't I see them in the Google Search Console
Hi, Please see attached photos. I have a website that shows external follow links when performing a search on open site explorer. However, they are not recognised or visible in search console. This is the case for both internal and external links. The internal links are 'no follow' which I am getting developer to rectify. Any ideas why I cant see the 'follow' external links? Thanks in advance to those who help me out. Jesse T7dkL5s T7dkL5s OkQmPL4 3qILHqS
Technical SEO | | jessew0 -
Searching in Google using the Site:www.example.com specification - is it in an order?
Hi Gurus, Just a quick searching question. If you do a Google search using the site: specification eg. site:www.example.com Is the list returned by Google in an order of something similar to 'Page Authority' or some other order eg. page first seen date etc. Because you are looking at your single site, is Google listing your pages back to you in it's perceived order of current 'popularity'? Thanks, Brad
Technical SEO | | BM70 -
Google Analytics - Custom Variables
Hi guys, I'd appreciate any advice with this one. At the moment I'm in the process of arranging a URL re-structure. I was wondering what the best way would be to track the performance of the old URLs against new ones? We will be ammending the URLs for any new property pages which go live on our website but leaving the old URLs in play for any old properties listed. We're taking this approach for the moment so we can conduct analysis on the change. It has been mentioned to me that placing a 'setvariable' in the code of pages with the old URLs and ones with the new URLs would be a way of tracking performance. However, my knowledge in this area is a little bit grey. Any advice? Cheers, Mark
Technical SEO | | MarkScully0 -
Google Search Parameters
Couple quick questions. Is using the parameter pws=0 still useful for turning off personalization? Is there a way to set my location as a URL parameter as well? For instance, I want to set my location to United States, can this be done with a URL param the same way as pws=0?
Technical SEO | | nbyloff0 -
Google plus
With Google search plus your world, would i see results ONLY from Google plus followers ? or from someone who is my facebook friend as well.
Technical SEO | | seoug_20050 -
Is this against google rules
Hi i am wanting to know if this is against google rules. I am building a website which will have lots of different sections and i wanted to know if you were allowed to have a new domain name pointing to a section of the site. so for example if i had a site with a domain name of manchester and then i wanted a section of the site to be called www.manchester.com/complimentary health I want to know if to help with traffic to the site and to have a better domain name, if it was allowed to have a new domain name pointing to that section of the site which could be called www.complimentaryhealth.com and have that pointing to the section. would love to hear your thoughts on this
Technical SEO | | ClaireH-1848860 -
Yahoo non www in SERPS
My site never shows the www in the yahoo SERPS. I don't do well with ranking in yahoo. Could these two facts be related? How do I get one canonicalization - preferably the www in yahoo?
Technical SEO | | DavidS-2820610