How to remove my site's pages in search results?
-
I have tested hundreds of pages to see if Google will properly crawl, index and cached them. Now, I want these pages to be removed in Google search except for homepage. What should be the rule in robots.txt?
I use this rule, but I am not sure if Google will remove the hundreds of pages (for my testing).
User-agent: *
Disallow: /
Allow: /$ -
Why not just 404/410 those pages?
-
Hi Matt! I've already tried your suggestion. I'll let you know what's the result. Thanks a lot man!
-
why don't you try adding a meta robots tag on those pages with "NOINDEX".
i would also do remove url with WMT
-
These are just test pages and I need them to be private and not visible in Google after I test. I understand that there will be a drop in SERP rankings.
-
I would do
User-agent: *
Disallow: /?
Allow: /But test it first in WMT first to be safe. However, you must be sure that this is the route you want to go down. Robots.txt will prevent all of those pages from being indexed, which means that none of their content will count. Any links to these pages may also be devalued. The result is a potential drop in SERPs.
What is the reason why you don't want them appearing? That way we may find an alternative solution.
-
This is basically a duplicate of your other thread where I gave you that code. Yes, it should block the other pages. Put that in, fetch in WMT and you should be right.
Also, you can test it before you implement in WMT as well. I tried it on my end and it works.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can 'follow' rather than 'nofollow' links be damaging partner's SEO
Hey guys and happy Monday! We run a content rich website, 12+ years old, focused on travel in a specific region, and advertisers pay for banners/content etc alongside editorial. We have never used 'nofollow' website links as they're no explicitly paid for by clients, but a partner has asked us to make all links to them 'nofollow' as they have stated the way we currently link is damaging their SEO. Could this be true in any way? I'm only assuming it would adversely affect them if our website was peanalized by Google for 'selling links', which we're not. Perhaps they're just keen to follow best practice for fear of being seen to be buying links. FYI we now plan to change to more full use of 'nofollow', but I'm trying to work out what the client is refering to without seeming ill-informed on the subject! Thank you for any advice 🙂
Intermediate & Advanced SEO | | SEO_Jim0 -
Competing Pages on Ecommerce Site - Very Frustrating
We have multiple issues with this situation. We rank #1 for "Lace Fabric", #3 for "Lace Trim", and #80 for "Lace". We also rank for "Lace Ribbon", and "Lace Appliques". The Lace Fabric and Lace Trim pages have plenty of backlinks, wherein may lie the problem. We have a similar issue for "Satin". "Silk Satin", "Polyester Satin", "Satin Trim", "Satin Ribbon", etc. This is a very annoying and common pattern. Our backlink profile is sterling, and our competitors with inferior backlink profiles and branded search are outranking us. We outrank them across the board for 2 word terms. Based on my evaluation of TF/CF, PA/DA, Content, etc., we should be on page 1 for "Lace". IMHO, these pages are competing for the head term. Any ideas on how to eliminate this issue to rank for head terms?
Intermediate & Advanced SEO | | GWMSEO0 -
Any issue? Redirect 100's of domains into one website's internal pages
Hi all, Imagine if you will I was the owner of many domains, say 100 demographically rich kwd domains & my plan was to redirect these into one website - each into a different relevant subfolder. e.g. www.dewsburytilers..com > www.brandname.com/dewsbury/tilers.html www.hammersmith-tilers.com > www.brandname.com/hammersmith/tilers.html www.tilers-horsforth.com > www.brandname.com/horsforth/tilers.html another hundred or so 301 redirects...the backlinks to these domains were slim but relevant (the majority of the domains do not have any backlinks at all - can anyone see a problem with this practice? If so, what would your recommendations be?
Intermediate & Advanced SEO | | Fergclaw0 -
Remove unwanted web pages
Hi All, I have a number of web pages that yield little or no traffic. I have analysed the traffic data in both normal SERPs and Google Adwords over a year. All low traffic pages rank on the first page.
Intermediate & Advanced SEO | | Mark_Ch
Redirecting these poor performing pages to the main content page would provide the user with a richer experience. Why do I need to remove these pages?
Cost, time and duplicate content issues is causing untold problems.
Removing the existance of the low/no traffic pages will allow me to provide fresh content on the main content pages. Question
Each main content page has about 20 low/no traffic pages associated with it.
I have about 30 instances of the main page scenario. Would carrying htacces page redirects hurt my ranking or worse? Regards Mark0 -
How to get a site out of Google's Sandbox
Hi I am working on a website that is ranking well in bing for the domain name / exact url search but appears no where in Google or Yahoo. I have done the site search in Google and it is indexed so I am presuming it is in the sandbox. The website was originally developed in India and I do not know whether it had some history of bad backlinks. The website itself is well optimised and I have checked all pages in Moz - getting a grade A. Webmaster Tools is not showing any manual actions - I was wondering what I could do next?
Intermediate & Advanced SEO | | AllieMc0 -
E Commerce site - removing discontinued items
We have been hit with a Panda penalty and the site has slowly been losing rankings since January, I've now realised that we have 4000+ page indexed in Google, but only 2000 live products. We have never deleted any of the pages with discontinued items, most of which were created when keyword stuffing and thin content reigned supreme - which explains the Panda penalty. But which is the best and quickest way to delete them from Google? We have already implemented a 'noindex' across all these pages, but as they are no longer in the 'crawlable' site, how will Google find them to know this? Would a 404 work any better - I'm not concerned about any link juice etc to/from these pages, I just want rid. I'm not sure if we can move all these pages into a dedicated directory which would allow us to use Google's Removal Tool - using it with the individual urls would be a mammoth task. Any advice would be most greatly appreciated.
Intermediate & Advanced SEO | | ElaineAllkids0 -
Killing 404 errors on our site in Google's index
Having moved a site across to Magento, obviously re-directs were a large part of that, ensuring all the old products and categories linked up correctly with the new site structure. However, we came up against an issue where we needed to add, delete, then re-add products. This, coupled with a misunderstanding of the csv upload processing, meant that although the old urls redirected, some of the new Magento urls changed and then didn't redirect: For Example: mysite/product would get deleted re-added and become: mysite/product-1324 We now know what we did wrong to ensure it doesn't continue to happen if we weret o delete and re-add a product, but Google contains all these old URLs in its index which has caused people to search for products on Google, click through, then land on the 404 page - far from ideal. We kind of assumed, with continual updating of sitemaps and time, that Google would realise and update the URL accordingly. But this hasn't happened - we are still getting plenty of 404 errors on certain product searches (These aren't appearing in SEOmoz, there are no links to the old URL on the site, only Google, as the index contains the old URL). Aside from going through and finding the products affected (no easy task), and setting up redirects for each one, is there any way we can tell Google 'These URLs are no longer a thing, forget them and move on, let's make a fresh start and Happy New Year'?
Intermediate & Advanced SEO | | seanmccauley0 -
What's the best way to phase in a complete site redesign?
Our client is in the planning stages of a site redesign that includes moving platforms. The new site will be rolled out in different phases throughout a period of a year. They are planning to put the new site redesign on a subdomain (i.e. www2.website.com) during the roll out of the different phases while eventually switching the new site back over to the www domain once all the phases are complete. We’re afraid that having the new site on the www2 domain will hurt SEO. For example, if their first phase is rolling out a new system to customize a product design and this new design system is hosted on www2.website.com/customize, when a customer picks a product to customize they’ll be linked to www2.website.com/customize instead of the original www.website.com/customize. The old website will start to get phased out as more and more of the new website is completed and users will be directed to www2. Once the entire redesign is completed, the old platform can be removed and the new website moved back to the www subdomian. Is there a better way of rolling out a website redesign in phases and not have it hosted on a different subdomain?
Intermediate & Advanced SEO | | BlueAcorn0