Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?
-
Hi all,
We 2 sub domains with hundreds of pages where we need only 50 pages to get indexed which are important. Unfortunately the CMS of these sub domains is very old and not supporting "noindex" tag to be deployed on page level. So we are planning to block the entire sites from robots.txt and allow the 50 pages needed. But we are not sure if this is the right approach as Google been suggesting to depend mostly on "noindex" than robots.txt. Please suggest whether we can proceed with robots.txt file.
Thanks
-
Hi vtmoz,
Given the limitations you are telling us, I'd give noindex in robots.txt a try.
I've run some experiments and found that noindex rule in Robots.txt works. It definitely won´t remove from index that pages, but it will stop showing them for search results.I'd suggest you to try using that rule with care.
Also, run some experiments on your own. My first test would be only adding one or two pages, the one that causes more trouble being indexed (maybe due to undesired traffic or due to ranking on undesired search terms).Hope it helps.
Best luck!
GR
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens when we canonical and point to a page which has been redirected to another page? Google response!
Hi all, I would like to know the different scenarios Google going to respond when we use canonical and redirect for duplicate pages. Let's say A to B are duplicate pages with 95% same content and C Doesn't have same content but context wise similar and priority page we expect to rank for. What happens if we canonical from A to B and set redirect from B to C? What if both A and B are pointed to C with canonical? What if A or B deleted and other one is canonical to C? Note: We can noindex or 301 redirect as they have their own visitors. This is more about showing most relevant content to the audience and avoid duplicate content in search results. Thanks
Algorithm Updates | | vtmoz0 -
Linking from high ranking sub domain pages to less ranking main domain pages to benefit latter
Hi all, We have our product guide pages on sub domain which are years old, so have some backlinks and high ranking for the beand related queries. Now we created new guide pages on our main website and we want these new pages to rank top beating the old pages from sub domain. Again we can't deindex or rel canonical to solve the issue as there are some part of users still using the old pages. We are planning to give a link from every old page of sub domain to same new page on main domain. Will this linking increases the authority of new pages technically and helps in ranking better? Like we give a link to "Moz guide 1" page to "Moz guide 2" page to rank latter better. Thanks
Algorithm Updates | | vtmoz0 -
Not showing in Google map listing. Why?
We have a client who's law firm is the highest google reviewed, on page two or three of St. Louis personal injury lawyer, but does not show in the map listing. Any ideas why this would happen or how to ensure they are viewable in the map listing?
Algorithm Updates | | David-Kley0 -
Bounce rates: Google vs others
Hello Mozzers, I was wondering if anyone could share some insight into how Google calculates bounce rates vs other analytics out there. We use both Google analytics and Clicky here. I've made several changes and additions to the website in hopes to increase traffic, optimization and reduce bounce rates. So far so good on all fronts. However I do notice bounce rates are way higher on google analytics than Clicky. While I get a bounce rate of 20%-29% on Clicky, Google has me way up the 50's or 60%s. I've read a few articles on it but I'm still a bit confused. Thanks!
Algorithm Updates | | md30 -
Does Google or Bing use words in the page title beyond the displayed limit for ranking purposes?
Standard good practice for on-page SEO includes keeping page title length below the maximum that Google displays in the SERPs. But words in the title beyond that maximum can be indexed, even if they don't show in the SERPs for end users. For ranking purposes, is there any value in words beyond the character limit in page titles that are truncated in the SERPs?
Algorithm Updates | | KyleJB0 -
Bing Vs Google SERP
I realize the major search engines use different criteria but I don't see how - for the same home page keyword - my site could rank #3 on page 1 for a Bing search and be off the charts (Page 15+)? on Google. Has Google gone so far off the charts with their new Penguins and Pandas so as to be in a different universe? Seems Google is now extremely over-weighting big sites like Wikipedia, WebMD, eHow, etc. and in doing so vastly reducing the diversity of results shown. I am commonly seeing different pages of the same website appear multiple times in the first 2-3 pages of Google results. What's the point?
Algorithm Updates | | veezer0 -
Decent rankings in Google, nothing in Bing and Yahoo
Hi there, I'm in the process of SEOing a site in a very competitive sector, the short term loans market. The URL for the site is http://www.piggy-bank.co.uk. I've managed to get a fair bit of success in Google for some very competitive keywords like short term loans, short term lender etc but in Bing and Yahoo I'm having no luck at all, with only 2 visits in the past month and no decent rankings!! I think I'm doing everything right, with regular new content on the site, decent technical SEO, semantic site structure, regular site map upload, a 10 year old domain, holistic link building through guest blogging etc, but still no luck at all. Looking at the webmaster tools in Bing, 95% of the URLs are indexed, but I'm getting such a low impression count, and obviously, an even lower click through. Am I missing something really obvious? Does anyone have any suggestions to improve my Bing and Yahoo rankings? I've worked on 100s of other sites and Yahoo and Bing tend to be the easy win to make the client happy 😉 Thanks in advance for your help. Dan
Algorithm Updates | | djslimited1 -
Gifts.com - Multiple domain pages in SERPs
One of our big natural search competitors for gift keywords is Gifts.com. We are competing for many keywords like "teen gifts", "gifts for him", "gifts for her". For many of these, the Google SERP has multiple Gifts.com pages on the first page. I have never seen more than one of our pages (uncommongoods.com) on a SERP page. Any clue how/why Gifts.com has multiple pages in search results ? Thanks!
Algorithm Updates | | znotes0