Why would I suddenly start seeing a spike in hits from particular bots (specifically rogerbot, google, bing, and yahoo)?
-
We have seen consistent network traffic over the past month, then starting yesterday, huge spikes in hits (hits as in crawls to pages causing an increase in megabytes downloaded) started coming in from Rogerbot, Google, Bing, and Yahoo. A specific example from Rogerbot is as follows:
-
rogerbot/1.1+(http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected])
-
Useragent from the bot
-
IP address: 54.226.73.52
-
Domain / hostname: ec2-54-226-73-52.compute-1.amazonaws.com
-
Physical location: United States flag United States, VA, Ashburn
We've have thought about doing a crawl-delay to prevent these bots from hitting us so hard, but that still doesn't help us answer why this even started in the first place.
Any clue on what may be going on here?
-
-
Hi Kasy, did you get to the bottom of this?
-
- No changes yet; however, it's getting worse on our end, particularly from Yahoo, so we're about to update it to add a line for crawl-delay.
- No known changes have been made with any of these.
- No changes have been made to any of our canonical or noindex tags.
- No, everything is the same.
- The only one that we have consistently crawl the site is Moz. I'm familiar with the other tools, but I haven't used them lately to crawl the site.
-
Hi Kasy, sorry to check the obvious first...
- Have there been any updates to your robots.txt file?
- Have you updated sitemaps? In Robots.txt or Google webmaster tools?
- Have you changed any meta information like canonical tags, noindex tags
- Have you changed any internal links from no-follow to follow?
- Have you got any tools regularly set up to crawl your site as Googlebot? Moz, Screaming Frog, DeepCrawl, Xenu etc.
-
Nothing unusual in any of those areas. GA is normal too. The hits/bots did come all at the same time, but since it started, it's been consistent.
-
Anything strange in your link profile or in your social media profile. Did the hits/bots come all at the same time or are spread evenly within 24 hrs?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google how deal with licensed content when this placed on vendor & client's website too. Will Google penalize the client's site for this ?
One of my client bought licensed content from top vendor of Health Industry. This same content is on the vendor's website & my client's site also but on my site there is a link back to vendor is placed which clearly tells to anyone that this is a licensed content & we bought from this vendor. My client bought paid top quality content for best source of industry but at this same this is placed on vendor's website also. Will Google penalize my client's website for this ? Niche is HEALTH
Technical SEO | | sourabhrana1 -
Pages to be indexed in Google
Hi, We have 70K posts in our site but Google has scanned 500K pages and these extra pages are category pages or User profile pages. Each category has a page and each user has a page. When we have 90K users so Google has indexed 90K pages of users alone. My question is. Should we leave it as they are or should we block them from being indexed? As we get unwanted landings to the pages and huge bounce rate. If we need to remove what needs to be done? Robots block or Noindex/Nofollow Regards
Technical SEO | | mtthompsons0 -
Bing search results
Hi I am getting little or no traffic via Bing. Why would this be? all my traffic is from Google and social media. my site is www.cocoonfxmedia.co.uk
Technical SEO | | Cocoonfxmedia0 -
Hit by a penalty? But which one?
Hi guys, First off, thanks for reading my post! My site, www.doctorloanusa.com suffered a massive drop in Google rankings and traffic on September 27. For the past 4 years, I've ranked in spots 1-3. My main keywords are physician loans and doctor loans. I had a competitor copy almost ALL my text. In early October, I filed a DMCA takedown and had the infringing site removed from the SERPS. I also made text changes to my site. I thought this duplicate content was the problem, but I have yet to see any improvements 2 months later. My site isn't particularly deep content-wise, but it's competitive with others in my niche. I have relatively few links but others in the SERPS have very similar link profiles. I tried to clean up a few questionable links and used the disavow tool. I have used the SEOmoz on-page tools to grade my site. Typically, I receive grade A reports. I still rank for my website URL, and still rank 1 in both Yahoo and Bing. My bounce rate is 43%. If anyone could provide insight, I would be very grateful. I need to get this site ranking again, as this property is my main source of income. Is the site over-optimized? Is it my anchor text profile? Thank you very much in advance!
Technical SEO | | Cary_Forest0 -
Google not showing my website ?
The website is medicare.md. if you search for term "medicare doctors PG county maryland" it is #1 in bing and yahoo but not even showing on google.com first TEN pages, although not banned. Interestingly if you do that search on google.co.pk it is #4. Quite Puzzuling !! Would appreciate any help or advice . Sherif Hassan
Technical SEO | | sherohass0 -
How to start more deep seo.
Okay, lets say that I already fixed all my technical seo stuff. All my crawl errors and warnings are gone. Now I really want to start seo so I have a few questions. 1. I know I should work based on a page and not on a domain, so it means I should choose which keyword to optimize each page for? only one keyword per page? or it might be more? 2. Some keywords are very competitive for me, example for my site www.theprinterdepo.com, the best keyword to rank would be printers, but it would take years a beginner to do that, what should I do? 3. If highly competitive keywords are not an option, which keywords shuold I use? the ones that have more global monthly searches? Thanks
Technical SEO | | levalencia10 -
I have a SERP result thats started returning a dud URL all of a sudden, any ideas why?
SERP "French Furniture" in google.co.uk The URL thats started to be returned P7 is <cite>www.thefurnituremarket.co.uk/ categories.asp?cat=</cite> The meta description is relevant however our CMS system 'ecommerce templates' has some code in it to return a dynamic meta description dependant on what catergory is being searched. It seems that the meta description is being populated from our french furniture category but the URL is not dynamic.. This has only started happening this past week. usually the url http://www.thefurnituremarket.co.uk/french_furniture.asp is returned. in WMT is is nicely indexing with 5 links... on it's own... Any ideas?
Technical SEO | | robertrRSwalters0 -
Google dropping pages from SERPS
The website for my London based plumbing company has thousands of specifically tailored pages for the various services we provide to all the areas in London. It equates to approximately 6000 pages in total. When google has all these pages indexed, we tend to get a fair bit of traffic - as they cater pretty well for long tail searches. However, every once in a while Google will drop the vast majority of our indexed pages from SERPs for a few days or weeks at a time - for example at the moment Google is only indexing 613 whereas last week it was back at the normal ~6000. Why does this happen? We of course lose a lot of organic traffic when these pages don't displayed - what are we doing wrong? Website: www.pgs-plumbers.co.uk
Technical SEO | | guy_andrews0