Why would I suddenly start seeing a spike in hits from particular bots (specifically rogerbot, google, bing, and yahoo)?
-
We have seen consistent network traffic over the past month, then starting yesterday, huge spikes in hits (hits as in crawls to pages causing an increase in megabytes downloaded) started coming in from Rogerbot, Google, Bing, and Yahoo. A specific example from Rogerbot is as follows:
-
rogerbot/1.1+(http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected])
-
Useragent from the bot
-
IP address: 54.226.73.52
-
Domain / hostname: ec2-54-226-73-52.compute-1.amazonaws.com
-
Physical location: United States flag United States, VA, Ashburn
We've have thought about doing a crawl-delay to prevent these bots from hitting us so hard, but that still doesn't help us answer why this even started in the first place.
Any clue on what may be going on here?
-
-
Hi Kasy, did you get to the bottom of this?
-
- No changes yet; however, it's getting worse on our end, particularly from Yahoo, so we're about to update it to add a line for crawl-delay.
- No known changes have been made with any of these.
- No changes have been made to any of our canonical or noindex tags.
- No, everything is the same.
- The only one that we have consistently crawl the site is Moz. I'm familiar with the other tools, but I haven't used them lately to crawl the site.
-
Hi Kasy, sorry to check the obvious first...
- Have there been any updates to your robots.txt file?
- Have you updated sitemaps? In Robots.txt or Google webmaster tools?
- Have you changed any meta information like canonical tags, noindex tags
- Have you changed any internal links from no-follow to follow?
- Have you got any tools regularly set up to crawl your site as Googlebot? Moz, Screaming Frog, DeepCrawl, Xenu etc.
-
Nothing unusual in any of those areas. GA is normal too. The hits/bots did come all at the same time, but since it started, it's been consistent.
-
Anything strange in your link profile or in your social media profile. Did the hits/bots come all at the same time or are spread evenly within 24 hrs?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Penalty and Adwords
Hi guys, I am wondering if the google manual penalty or penalty in general (because of bad backlink profile) also means that your website is blocked for Adwords? Thanks
Technical SEO | | barobijav0 -
How to stop google from indexing specific sections of a page?
I'm currently trying to find a way to stop googlebot from indexing specific areas of a page, long ago Yahoo search created this tag class=”robots-nocontent” and I'm trying to see if there is a similar manner for google or if they have adopted the same tag? Any help would be much appreciated.
Technical SEO | | Iamfaramon0 -
How to rank in Google Places
Normally, I don't have a problem with local SEO (more of a multi-channel sort of online marketing guy) but this one has got me scratching my head. Look at https://www.google.co.uk/search?q=wedding+venues+in+essex Theres two websites there (fennes and quendon park) that both have a much more powerful DA but don't appear in the Google Places (Google + Business or whatever it's labeled as). Why are websites such as Boreham house ranking top in the map listings? Quendon Park has a Google places listing, it's full of content, the NAP all matches up. Its a stronger website. Boreham House isn't any closer to the centroid than Quendon Park Just got me struggling this one
Technical SEO | | jasonwdexter0 -
Exact match and Bing
Hi Folks Fairly new to SEO and had two questions; 1- I have an SEO group doing work for us on a particular keyword, for the site http://bluetea.com.au/ .... however i've noticed that we only rank for the exact match of the keyword we ask them to work on... never a variation or even the plural .... Any thoughts? 2- http://bluetea.com.au/ has been around for almost 2 years however we still have absolutely no presence in Bing / Yahoo.... what am I missing.. any tips? Articles I must read? Thank you in advance 🙂
Technical SEO | | Intrested0 -
Google description problem
Hi all, My website is www.ipbskinning.com I'm having a problem with how my site is appearing in google. I have this in the head of my website: <meta name='<a class="attribute-value">description</a>' content='<a class="attribute-value">Free and Custom IPB Skins for Invision Power Board.</a>'/> Yet when I google 'ipbskinning' it says: Solid Skins. 1We test all our skins in all browsers to insure that they are compatible. This ensures that your users have the best user experience. which is random text from the content of my site. Any idea why this is happening? Thanks a lot all
Technical SEO | | pezza34340 -
When doing the ranking report I see my site showing up on google with out the www in front. So the report is not picking it up how do I fix that?
The ranking report is not picking up my site even though it's there. It would seem that the www. Is missing from the site on google so it's not registering in the report. How do i fix this?
Technical SEO | | ursalesguru0 -
Google Search Parameters
Couple quick questions. Is using the parameter pws=0 still useful for turning off personalization? Is there a way to set my location as a URL parameter as well? For instance, I want to set my location to United States, can this be done with a URL param the same way as pws=0?
Technical SEO | | nbyloff0 -
Seeing non-www in yahoo results - good or bad?
My site ranks for both domain versions but more non-www than www - Should I make it one or the other? How do I tell Yahoo to just choose one? Ehh?
Technical SEO | | DavidS-2820610