Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?
-
Hi all,
We 2 sub domains with hundreds of pages where we need only 50 pages to get indexed which are important. Unfortunately the CMS of these sub domains is very old and not supporting "noindex" tag to be deployed on page level. So we are planning to block the entire sites from robots.txt and allow the 50 pages needed. But we are not sure if this is the right approach as Google been suggesting to depend mostly on "noindex" than robots.txt. Please suggest whether we can proceed with robots.txt file.
Thanks
-
Hi vtmoz,
Given the limitations you are telling us, I'd give noindex in robots.txt a try.
I've run some experiments and found that noindex rule in Robots.txt works. It definitely won´t remove from index that pages, but it will stop showing them for search results.I'd suggest you to try using that rule with care.
Also, run some experiments on your own. My first test would be only adding one or two pages, the one that causes more trouble being indexed (maybe due to undesired traffic or due to ranking on undesired search terms).Hope it helps.
Best luck!
GR
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Adding / Manipulating Page Meta Titles?
We have a client who is experiencing some heavy google modification to the title tags being displayed on the search engine. It is adding "- 0 Reviews" to an ecommerce site. Obviously a bad start. There were no instances of these keywords anywhere on any of these pages, header tag or otherwise (on only a handful of the affected pages there was a single commented out image with an alt tag 0 reviews - but it was commented out and since removed) We have attempted to rewrite the title multiple times and it will modify the title but still include the non-relevant addition. Has anyone ever experienced anything like this?
Algorithm Updates | | Spindle0 -
Google domain search
Hello all, I'm a newbie to SEO, so you'll have to bear with me. I just started a website LangleyHomeSaerch.com a few months ago and am having trouble ranking with google. When I search "Langley Home Search" with Yahoo or Bing, it comes up on the first page. However when I search it with google it doesn't seem to rank even in the first few hundred pages. The only way I can get a match from google is if I search "Langley HomeSearch" or "LangleyHomeSearch". I know due to google's newer algorithms that there is less importance put on domain name matches, but is this normal, or is there anything I can do to improve it? Thx, Colby Langley, BC
Algorithm Updates | | colbygedak0 -
Why Is The Wrong Page Ranking?
In the past two weeks, I've seen some movement in ranking for "Tampa Personal Injury Attorney." The problem is that this page: http://www.kempruge.com/personal-injury/ is the one that's ranking and not this page: http://www.kempruge.com/location/tampa/tampa-personal-injury-legal-attorneys/ which is the one I've been working on. Also, the former page has made it to page 4 (not great) but better than 7, which is what the latter page was. In addition, the latter page now doesn't rank at all (or at least not in the first 16 pages). Finally, according to Moz, the latter page (the one that no longer ranks) is my second best page after my homepage. I just don't understand this at all. Is this a fluke? Should I just try to work on the page that's ranking higher over the page I've put the time into? Thanks, Ruben
Algorithm Updates | | KempRugeLawGroup0 -
Will increased pagerank increase traffic from google?
I got notified that my domain went from a google pagerank of 3 to 4. When this happens, does google raise me in the searches which can then hopefully get me more traffic, or is it a worthless number. Maybe only google knows 🙂
Algorithm Updates | | BrickPicker0 -
Site not in Google top 50 for key terms
Dear Moz Community, Our site - http://www.sportsdirectnews.com publishes a high volume of daily sport stories and aims to follow Google's Webmaster Guidelines, yet our pages don't appear anywhere in Google's SERP's. We've looked in details at the issue and think it could be something to do with: a) Unusual links or b) High page loading time or c) Too many on-page links If you could have a look at the site - http://www.sportsdirectnews.com - and give your professional opinion as to why our website is not appearing in SERP's, we would be most appreciative. SDN
Algorithm Updates | | BoomDialogue690 -
Struggling with Google Bot Blocks - Please help!
I own a site called www.wheretobuybeauty.com.au After months and months we still have a serious issue with all pages having blocked URLs according to Google Webmaster Tools. The 404 errors are returning a 200 header code according to the email below. Do you agree that the 404.php code should be changed? Can you do that please ? The current state: Google webmaster tools Index Status shows: 26,000 pages indexed 44,000 pages blocked by robots. In late March, we implemented a change recommended by an SEO expert and he provided a new robots.txt file, advised that we should amend sitemap.xml and other changes. We implemented those changes and then setup a re-index of the site by google. The no of blocked URLs eventually reduced in May and June to 1,000 for a few days – but now the problem has rapidly returned. The no of pages that are displayed in a google search request of www.google.com.au where the query was ‘site:wheretobuybeauty.com.au’ is 37,000: This new site has been re-crawled over last 4 weeks. About the site This is a Linux php site and has the following: 55,000 URLs in sitemap.xml submitted successfully to webmaster tools robots.txt file has been modified several times: Firstly we had none Then we created one but were advised that it needed to have this current content: User-agent: * Disallow: Sitemap: http://www.wheretobuybeauty.com.au/sitemap.xml
Algorithm Updates | | socialgrowth0 -
Why is my domain authority (and page authority) plummeting?
In June our domain authority was at a 41. In July we were 38 and ever since then our domain authority is gradually getting worse and worse. We went from a 33 to a 29 in one week! Possible explanations include: Maybe the SEO we hired (for a few months in late 2011) added our domain to some less-than-awesome directories The 301 redirects on our home page are hurting us somehow Duplicate content for URL's with different capitalization (IE: /pages/aboutus and /Pages/AboutUs) Can someone please point me in the right direction? Which of the above possibilities would likely impact domain/page authority? Any other ideas as to why this might be happening? Any suggestions for improving our domain or page authority? Thanks for the help!
Algorithm Updates | | MichaelBrown550 -
Why does Google Alerts call my website a blog?
Our company started a WordPress blog about 14 years ago. It has since added a third-party forum, a user-submitted photo gallery, and a huge database of searchable products. We also have almost 4000 posts. With all that said, Google Alerts often lists our content under blogs rather than websites. Sometimes it shows up in both? Does anyone know what criteria Google uses for determining the type of content, and how we can signal to them that we are a website?
Algorithm Updates | | TMI.com0