Why would I suddenly start seeing a spike in hits from particular bots (specifically rogerbot, google, bing, and yahoo)?
-
We have seen consistent network traffic over the past month, then starting yesterday, huge spikes in hits (hits as in crawls to pages causing an increase in megabytes downloaded) started coming in from Rogerbot, Google, Bing, and Yahoo. A specific example from Rogerbot is as follows:
-
rogerbot/1.1+(http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,[email protected])
-
Useragent from the bot
-
IP address: 54.226.73.52
-
Domain / hostname: ec2-54-226-73-52.compute-1.amazonaws.com
-
Physical location: United States flag United States, VA, Ashburn
We've have thought about doing a crawl-delay to prevent these bots from hitting us so hard, but that still doesn't help us answer why this even started in the first place.
Any clue on what may be going on here?
-
-
Hi Kasy, did you get to the bottom of this?
-
- No changes yet; however, it's getting worse on our end, particularly from Yahoo, so we're about to update it to add a line for crawl-delay.
- No known changes have been made with any of these.
- No changes have been made to any of our canonical or noindex tags.
- No, everything is the same.
- The only one that we have consistently crawl the site is Moz. I'm familiar with the other tools, but I haven't used them lately to crawl the site.
-
Hi Kasy, sorry to check the obvious first...
- Have there been any updates to your robots.txt file?
- Have you updated sitemaps? In Robots.txt or Google webmaster tools?
- Have you changed any meta information like canonical tags, noindex tags
- Have you changed any internal links from no-follow to follow?
- Have you got any tools regularly set up to crawl your site as Googlebot? Moz, Screaming Frog, DeepCrawl, Xenu etc.
-
Nothing unusual in any of those areas. GA is normal too. The hits/bots did come all at the same time, but since it started, it's been consistent.
-
Anything strange in your link profile or in your social media profile. Did the hits/bots come all at the same time or are spread evenly within 24 hrs?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Google de-rank a website.
Hi, I was inspecting a website which is covering the topic of best wheelbarrow of 2021, it is a new website and and starts ranking on google. But, after few days it got de-rank automatically and Moz is also not showing any result to that. I was wandering why this just happened and what should I do if I made my website and will not face this kind of situation?
Technical SEO | | Moeen22330 -
Why is Google not indexing my site?
I'm a bit confused as to why my site just isn't indexing on Google. Even if I type in my brand name, my social channels rank and there's no evidence of my website. I've followed all of the advice I've read and gone into webmaster tools and got the Wordpress yoast plug-in but nothing seems to be making a difference!One thing I've noticed, in Google Webmaster Tools it says "Couldn’t communicate with the DNS server." in site errors. I've called GoDaddy and they said that everything is fine. A bit frustrating. Trying to work out what my next steps should be but feeling a bit lost to be honest! Any help GREATLY appreciated!
Technical SEO | | j1066s0 -
Minor SEO for Yahoo and Bing
Over the past year, we have completely optimized one of the sites we manage for Google (removed bad links, reduced keyword count, etc) and we are doing great for Google organic traffic. However, in the process we have seen a steady decline in Yahoo organic traffic and Bing organic traffic. Enough so that I am looking for suggestions for quick fixes that aren't going to interfere with the Google optimizations that we made. Do you have any suggestions? Thanks for your help!
Technical SEO | | airnwater0 -
Sudden MozRank drop?
My site's MozRank dropped abruptly from 3.5 to 3 from September to October. MozRank had been steadily climbing until then and there were no changes made to the site in that time period. What may have happened?
Technical SEO | | mlwilmore0 -
Bing Webmaster tool
Hi Fellas, I wanted to know once you verify the BIng Webmaster tool (via xml file) for a dev site, do you have to do the verification process again for the final site? I thought I needed to do the verification again but once I added the final website (which have almost a similar URL) into the webmaster tool account, it seemed that I didn't have to verify it. I am a bit confused. Thank you for clarifying
Technical SEO | | Ideas-Money-Art0 -
Does Google see page with Trailing Slash as different
My company is purchasing another company's website. We are moving their entire site onto our CMS and the IT guys are working hard to replicate the URL structure. Several of the category pages are changing slightly and I am not sure if it matters: Old URL - http://www.DOMAIN.com/products/adults New URL - http://www.DOMAIN.com/products/adults**/** Notice the trailing slash? Will Google treat the new page as the same as the old one or as completely different (i.e. new) page? P.S. - Yes, I can setup 301s but since these pages hold decent rankings I'd really like to keep it exactly the same.
Technical SEO | | costume0 -
Google places address missing
I have a google places acct that used to rank fairly well. Then i changed addresses and updated the places account. It stopped ranking and whats worse is that the address will not show up in the listing. I have gone back in and edited it, verified it, done everything, but the address does not show on the places page or google results. It shows the city but not the actual address. Ideas?
Technical SEO | | webfeatseo0 -
Why did our site drop in Google rankings?
My site's URL (web address) is: http://tinyurl.com/3svn2l9 Hi there, We operate a travel site that lists numerous tours, accommodation and activities. Since 6th August 2011 we have dropped from top 10 SERP rankings of our pages to around result number 100 (page 10) and losing massive amount of visitors via Google Search. Our Yahoo and Bing rankings are still in the top10. We need your advice and quick! The last changes we have made are the following: -redirected the non-www version to the www version on the 1st August -bought advertising with a follow link in a sidebar that is being populated across the site (+4000 pages) about 2 months ago -added a blog to the website 2 weeks ago and posted 2 posts to date. Additionally, our website structure allows visitors (and bots) to see the same listings via different URLs which caused duplicate content. This has been the case since the launch of our website about 1 year ago. To prevent this duplicate content we have placed canonical tags on the individual listings pages. Why did our site all of a sudden plummet in the rankings?
Technical SEO | | Robbern0