Will disallowing in robots.txt noindex a page?
-
Google has indexed a page I wish to remove. I would like to meta noindex but the CMS isn't allowing me too right now. A suggestion o disallow in robots.txt would simply stop them crawling I expect or is it also an instruction to noindex? Thanks
-
Hi
I would also suggest to make sure to change internal links if possible and make them point to the page you want to see in the index
-
Good answer we both submitted this at the same time.
-
Check out this article, it's the Google Requirements for Removing content
There are multiple ways to do this:
Ideally, robots.txt is the way to go, you can also noindex the page. There is no need to do both. If your looking for the page to disappear as well, i'd let it 404.
Another option is to request the URL to be removal request in Google Webmaster Tools along with these things it's under optimization>Remove URLs.
Hope this helps
-
You'd have to add the disallow in robots.txt and then go into the Google Webmaster Console, use the Remove URLs option to get Google to deindex that page.
From Google:
Use **robots.txt **to specify how search engines should crawl your site, or request **removal **of URLs from Google's search results ( have you read our removal requirements? ). Only site owners and users with full permissions can request removals.
-
It the page has already been indexed and you block via robots.txt file what you will most likely see in SERPs is a listing with no meta description (so just one line). I do believe that ranking placements will drop. If you want to remove from SERPs completely you need to use the noindex tag.
I would block by robots.txt file now while you work out how to add the noindex tag.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens to crawled URLs subsequently blocked by robots.txt?
We have a very large store with 278,146 individual product pages. Since these are all various sizes and packaging quantities of less than 200 product categories my feeling is that Google would be better off making sure our category pages are indexed. I would like to block all product pages via robots.txt until we are sure all category pages are indexed, then unblock them. Our product pages rarely change, no ratings or product reviews so there is little reason for a search engine to revisit a product page. The sales team is afraid blocking a previously indexed product page will result in in it being removed from the Google index and would prefer to submit the categories by hand, 10 per day via requested crawling. Which is the better practice?
Intermediate & Advanced SEO | | AspenFasteners1 -
Base copy on 1 page, then adding a bit more for another page - potential duplicate content. What to do?
Hi all, We're creating a section for a client that is based on road trips - for example, New York to Toronto. We have a 3 day trip, a 5 day trip, a 7 day trip and a 10 day trip. The 3 day trip is the base, and then for the 5 day trip, we add another couple of stops, for the 7 day trip, we add a couple more stops and then for the 10 day trip, there might be two or three times the number of stops of the initial 3 day trip. However, the base content is similar - you start at New York, you finish in Toronto, you likely go through Niagara on all trips. It's not exact duplicate content, but it's similar content. I'm not sure how to look after it? The thoughts we have are:1) Use canonical tags 3,5,7 day trips to the 10 day trip.
Intermediate & Advanced SEO | | digitalhothouse
2) It's not exactly duplicate content, so just go with the content as it is We don't want to get hit by any penalty for duplicate content so just want to work out what you guys think is the best way to go about this. Thanks in advance!0 -
Meta NOINDEX and links into the pages?
If I have internal links pointing to pages that are META NO INDEX, will Google still index them? Or does that only apply to pages that are linked to from an external domain? Thanks!
Intermediate & Advanced SEO | | bjs20100 -
Robots
I have just noticed this in my code name="robots" content="noindex"> And have noticed some of my keywords have dropped, could this be the reason?
Intermediate & Advanced SEO | | Paul780 -
Reciprocal Links and nofollow/noindex/robots.txt
Hypothetical Situations: You get a guest post on another blog and it offers a great link back to your website. You want to tell your readers about it, but linking the post will turn that link into a reciprocal link instead of a one way link, which presumably has more value. Should you nofollow your link to the guest post? My intuition here, and the answer that I expect, is that if it's good for users, the link belongs there, and as such there is no trouble with linking to the post. Is this the right way to think about it? Would grey hats agree? You're working for a small local business and you want to explore some reciprocal link opportunities with other companies in your niche using a "links" page you created on your domain. You decide to get sneaky and either noindex your links page, block the links page with robots.txt, or nofollow the links on the page. What is the best practice? My intuition here, and the answer that I expect, is that this would be a sneaky practice, and could lead to bad blood with the people you're exchanging links with. Would these tactics even be effective in turning a reciprocal link into a one-way link if you could overlook the potential immorality of the practice? Would grey hats agree?
Intermediate & Advanced SEO | | AnthonyMangia0 -
Should I index tag pages?
Should I exclude the tag pages? Or should I go ahead and keep them indexed? Is there a general opinion on this topic?
Intermediate & Advanced SEO | | NikkiGaul0 -
Will swear words present on my pages affect my rankings?
Hi There, I am in the process of formulating a listing policy for my site and I'm not sure whether I should add something in there for swear words. My site is an adult site and swear words come with the territory, unfortunately. Will user generated content with swear words affect my ranking? Thank you
Intermediate & Advanced SEO | | Mulith0 -
We are changing ?page= dynamic url's to /page/ static urls. Will this hurt the progress we have made with the pages using dynamic addresses?
Question about changing url from dynamic to static to improve SEO but concern about hurting progress made so far.
Intermediate & Advanced SEO | | h3counsel0