How do you de-index and prevent indexation of a whole domain?
-
I have parts of an online portal displaying in SERPs which it definitely shouldn't be. It's due to thoughtless developers but I need to have the whole portal's domain de-indexed and prevented from future indexing.
I'm not too tech savvy but how is this achieved? No index? Robots?
thanks
-
Fastest way to do it would be through a combination of Google Webmaster Tools and Robots.txt / noindex,nofollow meta tags.
Firstly if your domain hasn't been verified within GWT, go through that process. Then proceed with the remove URL request within GWT. (Image Attached)
You will need to upload a robots.txt and/or implement the noindex meta tag within the header of the pages you no longer want indexed.
Example Robots.txt:
User-agent: *
Disallow: /offendingportal/Example Robots Meta tag:
That should be enough to get rid of the offending content from within the SERPS and prevent it from being re-indexed.
-
Hi TCE,
There are two options in my opinion for this which are the best (as their are several other options including META robots etc):
- Robots.txt: Exclude your portal via your robots.txt file.
- HTTP Auth: Let your developers add HTTP Authentication so you have to log in to see the portal. Google won't be able to see the pages itself and so will de index them.
Hope this helps.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Domain name change
Here's the scenario... Client has two domain names: domain.com - targeting one country (Australia) otherdomain.com - targeting all other countries Both have identical products, but different currencies (AU$ and US$). The problem (as most of you will know) is that without using a sub-domain or country-code top-level domains, Google has no idea which domain should be served for which domain. Furthermore, because the root domain is different, Google doesn't see any connection between the two - other than the fact they have identical products! My recommendation to the client is to change to: domain.com to domain.com.au otherdomain.com to domain.com Arguably, we could leave the second one alone. But I think it's better for the brand to use the same root domain for each. Obviously this means both will need to be redirected. Since NONE of the pages within the sites will change, do we need to redirect every page, or just the root domain? Any other risks or concerns we should know about?
Intermediate & Advanced SEO | | muzzmoz0 -
Better to 301 or de-index 403 pages
Google WMT recently found and called out a large number of old unpublished pages as access denied errors. The pages are tagged "noindex, follow." These old pages are in Google's index. At this point, would it better to 301 all these pages or submit an index removal request or what? Thanks... Darcy
Intermediate & Advanced SEO | | 945010 -
Domain name with a single digit
I have an existing campervan hire website which is being redesigned, rebranded and renamed (including the domain). The website allows businesses and owners to list campervans for rent to customers. There are a huge amount of campervan hire companies and so not many relevant domain names. Also many are for sale as expensive premium domains. We want something different that will standout. We're thinking of https//:camper7.com as its short and we think it would work. Is it best to avoid using a single digit in a domain if it isnt a 2 or 4 or does it not matter if we think it could work as it would stand out? Any help or advice would be appreciated. thanks James
Intermediate & Advanced SEO | | Curran1 -
Can't get auto-generated content de-indexed
Hello and thanks in advance for any help you can offer me! Customgia.com, a costume jewelry e-commerce site, has two types of product pages - public pages that are internally linked and private pages that are only accessible by accessing the URL directly. Every item on Customgia is created online using an online design tool. Users can register for a free account and save the designs they create, even if they don't purchase them. Prior to saving their design, the user is required to enter a product name and choose "public" or "private" for that design. The page title and product description are auto-generated. Since launching in October '11, the number of products grew and grew as more users designed jewelry items. Most users chose to show their designs publicly, so the number of products in the store swelled to nearly 3000. I realized many of these designs were similar to each and occasionally exact duplicates. So over the past 8 months, I've made 2300 of these design "private" - and no longer accessible unless the designer logs into their account (these pages can also be linked to directly). When I realized that Google had indexed nearly all 3000 products, I entered URL removal requests on Webmaster Tools for the designs that I had changed to "private". I did this starting about 4 months ago. At the time, I did not have NOINDEX meta tags on these product pages (obviously a mistake) so it appears that most of these product pages were never removed from the index. Or if they were removed, they were added back in after the 90 days were up. Of the 716 products currently showing (the ones I want Google to know about), 466 have unique, informative descriptions written by humans. The remaining 250 have auto-generated descriptions that read coherently but are somewhat similar to one another. I don't think these 250 descriptions are the big problem right now but these product pages can be hidden if necessary. I think the big problem is the 2000 product pages that are still in the Google index but shouldn't be. The following Google query tells me roughly how many product pages are in the index: site:Customgia.com inurl:shop-for Ideally, it should return just over 716 results but instead it's returning 2650 results. Most of these 1900 product pages have bad product names and highly similar, auto-generated descriptions and page titles. I wish Google never crawled them. Last week, NOINDEX tags were added to all 1900 "private" designs so currently the only product pages that should be indexed are the 716 showing on the site. Unfortunately, over the past ten days the number of product pages in the Google index hasn't changed. One solution I initially thought might work is to re-enter the removal requests because now, with the NOINDEX tags, these pages should be removed permanently. But I can't determine which product pages need to be removed because Google doesn't let me see that deep into the search results. If I look at the removal request history it says "Expired" or "Removed" but these labels don't seem to correspond in any way to whether or not that page is currently indexed. Additionally, Google is unlikely to crawl these "private" pages because they are orphaned and no longer linked to any public pages of the site (and no external links either). Currently, Customgia.com averages 25 organic visits per month (branded and non-branded) and close to zero sales. Does anyone think de-indexing the entire site would be appropriate here? Start with a clean slate and then let Google re-crawl and index only the public pages - would that be easier than battling with Webmaster tools for months on end? Back in August, I posted a similar problem that was solved using NOINDEX tags (de-indexing a different set of pages on Customgia): http://moz.com/community/q/does-this-site-have-a-duplicate-content-issue#reply_176813 Thanks for reading through all this!
Intermediate & Advanced SEO | | rja2140 -
Subdomain and root domain
Hey Everyone, our page has multiple domains and I'm wondering how it affects search rankings today. I saw some stuff from almost a year ago, but I'm not sure if something has changed. We currently have our root domain "www.xyz.com" and started moving some pages over to a different sub-domain "web.xyz.com" because of usability and ease of adjusting content. How much will this affect our seo? Thanks!
Intermediate & Advanced SEO | | josh1230 -
De-indexed by Google! ?
So it looks as though the content from myprgenie.com is no longer being indexed. Anyone know what happened and what they can do to fix it fast?
Intermediate & Advanced SEO | | siteoptimized0 -
Domain with a Virus History
Hi, I have a domain that I am working on that has a past we did not know about. Doing a bit of research it appears that back in 2010 the domain had a link to a virus or had a virus on the domain – because of this certain anti virus sites are blocking the domain. Interestingly Google, Norton, Firefox say the domain is fine.... IE, Kaspersky and a few still block it. I am going through and manually searching and trying to get them to agree the site is safe BUT I am having a problem with mywot.com. They refuse to take down the “reviews” staying its a virus site. Anything I can do? Any suggestions? Any legal action we can take? Is there anything else I can do or should be doing to check else where? Thanks in advance Fresh Fire One
Intermediate & Advanced SEO | | JohnW-UK0 -
Domain Name with Numeric Digit
If I use a Domain Name with a Numeric Digit, would it in any way harm the SEO and Google Ranking of my page? This Question is only from point of view of SEO
Intermediate & Advanced SEO | | KaranBatra0