Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.
-
We are trying to figure out why last year we had a huge (80%) and sudden (within two days) drop in our google searches. The only "outlier" in our site that we can find is a huge number of pages reported in MOZ as scanned by search engines. Is this a problem? How did we get so many pages reported? What can we do to bring the number of searched pages back to a "normal" level?
BT
-
Hi. A mystery indeed! Have you recently upgraded or changed Web platforms or changed or upgraded what you are using for your site navigation?
-
Stewart_SEO
Thanks for your quick response. We did review the robots.txt of the competitors. Not line by line - they took surprisingly different approaches to the robots.txt. But there were the usual exclusions for wish lists, etc. We've gone back and tightened up our robots.txt and haven't yet seen any changes. Several months ago we were at about 600,000 pages and it is dropping. Very mysterious.
-
Have you looked at your competitors robots.txt file? they are probably blocking the very same searches you are talking about. if there is a particular bot like a Chinese crawler for example baidu that you don't want to come to your site you can block them via the command: User-agent: Baiduspider
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens if we remove all the links to internal pages from our homepage?
Hi Moz community, We wanna give a try by removing all the links from homepage to internal pages and keep just a free trial button. Will this impact our SEO anyway? We have nearly 15 important internal pages at 2nd and 3rd hierarchy level. They may drop in rankings but we want to risk for few days to understand how it works. Your opinion please! Thanks
Algorithm Updates | | vtmoz0 -
Sitemaps for landing pages
Good morning MOZ Community, We've been doing some re-vamping recently on our primary sitemap, and it's currently being reindexed by the search engines. We have also been developing landing pages, both for SEO and SEM. Specifically for SEO, the pages are focused on specific, long-tail search terms for a number of our niche areas of focus. Should I, or do I need to be considering a separate sitemap for these? Everything I have read about sitemaps simply indicates that if a site has over 50 thousand pages or so, then you need to split a sitemap. Do I need to worry about a sitemap for landing pages? Or simply add them to our primary sitemap? Thanks in advance for your insights and advice.
Algorithm Updates | | bwaller0 -
Bounce rate seems to decrease as site climbs the SERP but SERP ranking always drops down after a few days - WHY ?
Like quite a few sites recently we've seen some large fluctuations for our domain the SERPs over the past 6 or so weeks. One thing ive noticed, is that when the site seems to rank higher in the SERPs we get a lower bounce rate. If the sites average across all of its main keywords in #5 the bounce rate is c. 40% (this site is a creative portfolio site, so i guess the niche has a slightly higher bounce rate than normal as you will get some people who click through an notice straight away that the style of the portfolio isnt what they where looking for) But when the sites ranking averages #2-3 the bounce rate tends to be about 25%. (The thing is that we tend to always drop back down after a fews days or so) Has any one else noticed this ?
Algorithm Updates | | jpeg801 -
Page 2 to page 1
I've found a lot of times it does not take much activity to get a keyword from ranking on page 3 of Google or further down to page 2 but there seems to be a hurdle from page 2 to page 1. It is very frustrating to be between 11 and 15 but not being able to make that push to 9 or 10. Has anyone got or seen any data to justifiy this?
Algorithm Updates | | S_Curtis0 -
Same page but appearing in Google with different titles
I have a page ranking on position 1 for a key phrase. The key phrase is the title of the page as well. I'll use a mock key phrase to aid my question - "Teeth and Gums" So the page is ranking number 1 for "Teeth and Gums" and "Teeth and Gums" is the meta title. However, I went ahead and did a new search adding an additional keyword to the original search. When I did a new search adding an additional keyword to the original search, Google has done something weird.. Let's say the search is "Dentistry - Teeth and Gums", Google has ranked my page again as number 1 but changed the title. The title in the search result is now "Dentistry - Teeth and Gums" How and why? It's kinda like Google PPC's keyword insertion but the title hasn't got anything weird like {KeyWord: Dentistry}. It's just "Teeth and Gums" Has this happened to you guys? Any ideas?
Algorithm Updates | | Bio-RadAbs0 -
Advice on breaking page 3 SERP?
I've worked extremely hard re-ranking for the key word "kayak fishing" after our site migration and panda/penguin our site www.yakangler.com was lost in the doldrums. I couldn't find us on any of the results page 1 thru page 64. The good news is we have managed to make it to page 3 for the past month but I feel like we have hit a wall. I can't seem to get any more moment in my SERP for "kayak fishing" Google US. Anyone have any recommendations on what I should do to move us up more? We have good content that is updated daily, an active community and forum. site: www.yakangler.com key word: kayak fishing
Algorithm Updates | | mr_w0 -
How do these people have so many external back links?
A few of the sites that I 'compete' with have blogger accounts...and according to the research I've done they have over 500 million external, followed, links! These are mom blogs....and they aren't trying to do any SEO. I've noticed that if you are using blogger (as opposed to wordpress) and adding embed youtube videos to your blog you can set it to auto populate to youtube and also get a link in return. From what I can gather this just happens with blogger accounts since bloggers...and youtube...and google are all related. Does that have something to do with the crazy amount of followed external links they are getting? They also have a domain authority of 96 (which is because blogger is a trusted domain.... Is the sub-domain a better number to look at since their blog is a sub domain on blogger? Also, does anyone know who to get the same sort of exposure in youtube as blogger blogs get...or if it's even possible? Thanks.
Algorithm Updates | | NoahsDad0 -
FB Like and G+ on page transferred ?
Hi , As we all know that social signals like FB Like and G+ give a boost to the search engine rankings. I have several pages we have garnerd 100s of FB likes with a 10s of G+ . Now i want to reorganize my site to so that the contents of the blog would be having a different URL . So i have planned for 301 redirections for the older content to the newer ones. But does the FB Like and G+ also move along with the 301 redirection ? If not whats is the best way to handle this... Warm Rgds Avinash MB
Algorithm Updates | | ShoutOut0