Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.
-
We are trying to figure out why last year we had a huge (80%) and sudden (within two days) drop in our google searches. The only "outlier" in our site that we can find is a huge number of pages reported in MOZ as scanned by search engines. Is this a problem? How did we get so many pages reported? What can we do to bring the number of searched pages back to a "normal" level?
BT
-
Hi. A mystery indeed! Have you recently upgraded or changed Web platforms or changed or upgraded what you are using for your site navigation?
-
Stewart_SEO
Thanks for your quick response. We did review the robots.txt of the competitors. Not line by line - they took surprisingly different approaches to the robots.txt. But there were the usual exclusions for wish lists, etc. We've gone back and tightened up our robots.txt and haven't yet seen any changes. Several months ago we were at about 600,000 pages and it is dropping. Very mysterious.
-
Have you looked at your competitors robots.txt file? they are probably blocking the very same searches you are talking about. if there is a particular bot like a Chinese crawler for example baidu that you don't want to come to your site you can block them via the command: User-agent: Baiduspider
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do the back-links go wasted when anchor text or context content doesn't match with page content?
Hi Community, I have seen number of back-links where the content in that link is not matching with page content. Like page A linking to page B, but content is not really relevant beside brand name. Like page with "vertigo tiles" linked to page about "vertigo paints" where "vertigo" is brand name. Will these kind of back-links completely get wasted? I have also found some broken links which I'm planning to redirect to existing pages just to reclaim the back-links even though the content relevancy is not much beside brand name. Are these back-links are beneficial or not? Thanks
Algorithm Updates | | vtmoz0 -
Linking from high ranking sub domain pages to less ranking main domain pages to benefit latter
Hi all, We have our product guide pages on sub domain which are years old, so have some backlinks and high ranking for the beand related queries. Now we created new guide pages on our main website and we want these new pages to rank top beating the old pages from sub domain. Again we can't deindex or rel canonical to solve the issue as there are some part of users still using the old pages. We are planning to give a link from every old page of sub domain to same new page on main domain. Will this linking increases the authority of new pages technically and helps in ranking better? Like we give a link to "Moz guide 1" page to "Moz guide 2" page to rank latter better. Thanks
Algorithm Updates | | vtmoz0 -
Why different pages rank in different countries?
Hi all, I have been investigating on why our log-in page is ranking for primary keyword, but not our homepage. I can see now homepage is ranking from our second important country. I wonder why and what causes to rank different pages in different countries for same keyword. Again the statistics does not vary much between these countries. Thnaks
Algorithm Updates | | vtmoz0 -
Have AMP pages on Chrome disappeared?
Hi, We amplified our site the other day and Google was quick to find our AMP pages. Today, we can´t seem to find any AMP pages on Chrome and it looks like we can´t find any other pages from sites like Wapo and so forth. They´re still visible on Safari. We are searching from Iceland. Is anyone experiencing this and if so, could you please share a link to the news? Thanks and have a nice day!
Algorithm Updates | | guidetoiceland0 -
Do you think this page has been algorithmically penalised or is it just old?
Here is the page: http://www.designquotes.com.au/business-blog/top-10-australian-business-directories-in-2012/ It's fairly old, but when it was first written it hit #1 for "business directories". After a while it dropped but was receieving lots of traffic for long tail variations of "business directories Australia" As of the 4th of October (Penguin 2.1) it lost traffic and rankings entirely. I checked it's link profile and there isn't anything fishy: From Google Webmaster https://docs.google.com/spreadsheet/ccc?key=0AtwbT3wshHRsdEc1OWl4SFN0SDdiTkwzSmdGTFpZOFE&usp=sharing In fact, two links are entirely natural http://blog.businesszoom.com.au/2013/09/use-customer-reviews-to-improve-your-website-ranking/ http://dianajones.com.au/google-plus-local-equals-more-business-blog/ Yet when I search for a close match in title in Google AU, the article doesn't appear within even the first 4 pages. https://www.google.com.au/#q=top+10+Australian+Business+Directories&start=10 Is this simple because it's an old article? Should I re-write it, update the analysis and use a rel=canonical on the old article to the new?
Algorithm Updates | | designquotes0 -
How much is Page Rank really worth?
We are in a position to purchase a domain, made of relevant keywords to our company with a current page ranking of 4 for their home page. However in looking at their analytics and other information they do not do well on significant keywords and have very low site traffic. In fact they do very, very poorly. With their high page ranking would it be relatively easy to conduct a successful SEO campaign on the domain if we were to take it over as our own and attempt to climb in the SERP's? I know Page Rank doesn't mean everything when it comes to your ranking, but 4 is relatively high in our field, so I don't really understand why they do so poorly when it comes to their actual rankings on key words.
Algorithm Updates | | absoauto0 -
Linking to authority / competitor website!? Yes or No?
Here is the dilema .... People say, don't be afraid to link to authority website! What if an authority website in particular niche is actually competing with you in the Search, but you do have a review profile there? Is it ok to link to that profile ? Should I include a no follow tag ? Another case : some other authority high ranked website competing in search with mine directly has a profile page for my company, but this authority website has blocked Google bot to crawl the profile page all the reviews there !? Can I still link to that page and will this be appreciated by Google ? Am I passing PR and link juice from my website to those direct comeptitors / authority websites ?
Algorithm Updates | | montauto0 -
In the body of index page i want to be able to add text that can be picked up by crawlers but I do not want these text to be visible? How can I code this?
in the body of index page i want to be able to add text that can be picked up by crawlers but I do not want these text to be visible? How can I code this?
Algorithm Updates | | FinindDesign0