How do you de-index and prevent indexation of a whole domain?
-
I have parts of an online portal displaying in SERPs which it definitely shouldn't be. It's due to thoughtless developers but I need to have the whole portal's domain de-indexed and prevented from future indexing.
I'm not too tech savvy but how is this achieved? No index? Robots?
thanks
-
Fastest way to do it would be through a combination of Google Webmaster Tools and Robots.txt / noindex,nofollow meta tags.
Firstly if your domain hasn't been verified within GWT, go through that process. Then proceed with the remove URL request within GWT. (Image Attached)
You will need to upload a robots.txt and/or implement the noindex meta tag within the header of the pages you no longer want indexed.
Example Robots.txt:
User-agent: *
Disallow: /offendingportal/Example Robots Meta tag:
That should be enough to get rid of the offending content from within the SERPS and prevent it from being re-indexed.
-
Hi TCE,
There are two options in my opinion for this which are the best (as their are several other options including META robots etc):
- Robots.txt: Exclude your portal via your robots.txt file.
- HTTP Auth: Let your developers add HTTP Authentication so you have to log in to see the portal. Google won't be able to see the pages itself and so will de index them.
Hope this helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Consolidating Multiple Domains into A Single Domain
I have a client who's website is an amalgamation of multiple domains. jacksonhole.net is the main domain but the site passes traffic back and forth from the following domains/sites. My questions is, would it it be better for SEO to consolidate all of these domains under the single high authority domain and 301 redirect the rest or is that a really bad idea? Thanks for your help. jacksonhole.net (Domain Authority 31) jackson-hole-rental-condos.com (Domain Authority 22) jackson-hole-rental-homes.com (Domain Authority 21) j acksonholehotelguide.com (Domain Authority 19)
Intermediate & Advanced SEO | | dbaxa-2613381 -
Drop in Indexed pages
Hope everyone is having an Awesome December! I first noticed a drop in my index in the beginnings of November. My site drop in indexed pages from 1400 to 600 in the past 3-4 weeks. I don't know the cause of it, and would like the community to help me figure out why my indexing has dropped. Thank you for taking time out of your schedule to read this.
Intermediate & Advanced SEO | | BSC0 -
Sub domain on root domain
Hello,
Intermediate & Advanced SEO | | dror999
I have a question that I can't find a good answer on.
I have a site, actually a "portal"/ "directory" for service providers.
Now, for start, we opened every service provider own page on our site, but now we get a lot of applications from those providers that thy want sites from their own.
We want to make every service provider his own site, but on sub domain url. ( they don’t mind… its ok for them)
So, my site is www.exaple.com
There site will be: provider.exaple.com
Now I have two questions:
1. can it harm my site in SEO?
2. if one from those sub domain , punished by google because is owner do "black hat seo" , how it will affect the rood domin? It can make the root domain to get punished?
Thanks!!0 -
Subdomain, subfolder or separate domains?
Hi Mozzers, We're in the process of re-developing and redesigning several of our websites, and moving them all onto the same content management system. At the moment, although the websites are all under the same brand and roughly the same designs, because of various reasons they all either live on a separate domain to the main website, or are on a subdomain. Here's a list of what we have (and what we're consolidating): Main site - http://www.frenchentree.com/ Property database - http://france-property.frenchentree.com/ (subdomain) Forum - http://www.france-forum-frenchentree.com/ (separate domain) Classified ads - http://www.france-classified-ads-frenchentree.com/ (separate domain) My question to you lovely people is: should we take this opportunity through the redevelopment of the CMS to put everything into subfolders of the main domain? Keep things as they are? Put each section onto a subdomain? What's best from an SEO perspective? For information - the property database was put onto a subdomain as this is what we were advised to do by the developers of the system. We're starting to question this decision though, as we very rarely see subdomains appear in SERPs for any remotely competitive search terms. Our SEO for the property database is fairly non-existent, and only ever really appears in SERPs for brand related keywords. For further info - the forum and classifieds were under a separate brand name previously, so keeping them on separate domains felt correct at that time. However, with the redevelopment of our sites, it seems to make more sense to either put them on subdomains or subfolders of the main site. Our SEO for the forum is pretty strong, though has dwindled in the last year or so. Any help/advice would be very much appreciated. Thanks Matt
Intermediate & Advanced SEO | | Horizon0 -
Google Indexing Feedburner Links???
I just noticed that for lots of the articles on my website, there are two results in Google's index. For instance: http://www.thewebhostinghero.com/articles/tools-for-creating-wordpress-plugins.html and http://www.thewebhostinghero.com/articles/tools-for-creating-wordpress-plugins.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+thewebhostinghero+(TheWebHostingHero.com) Now my Feedburner feed is set to "noindex" and it's always been that way. The canonical tag on the webpage is set to: rel='canonical' href='http://www.thewebhostinghero.com/articles/tools-for-creating-wordpress-plugins.html' /> The robots tag is set to: name="robots" content="index,follow,noodp" /> I found out that there are scrapper sites that are linking to my content using the Feedburner link. So should the robots tag be set to "noindex" when the requested URL is different from the canonical URL? If so, is there an easy way to do this in Wordpress?
Intermediate & Advanced SEO | | sbrault740 -
Should I Roll Back Domain Change?
A couple years ago I changed domain names and switched platforms for my site. The traffic dropped dramatically (80-90%). I've tried to get inbound links changed, clean up on-page stuff, but nothing is making a big change. I think most of the problem is loss of link juice with the 301 redirects from the old domain to the new one. Would I be risking bigger losses by switching back to the old domain name?
Intermediate & Advanced SEO | | iJeep0 -
To index search results or not?
In its webmaster guidelines, Google says not to index search results " that don't add much value for users coming from search engines." I've noticed several big brands index search results, and am wondering if it is generally OK to index search results with high engagement metrics (high PVPV, time on site, etc). We have an database of content, and it seems one of the best ways to get this content in search engines would be to allow indexing of search results (to capture the long tail) rather than build thousands of static URLs. Have any smaller brands had success with allowing indexing of search results? Any best practices or recommendations?
Intermediate & Advanced SEO | | nicole.healthline0 -
Domains merging
Hi everyone, The company I work for has two domains, one for the english version of the website and another one for the french version. Example: www.digitalmusic.com (in english) www.musiquedigitale.com (in french) (these are examples***) I would like to know if on SEO standpoint it would be better to only have one domain so all of the links link to the same domain. Would it increase the domain authority and our rankings ? We will then have: www.digitalmusic.com/fr/page1 for pages in french www.digitalmusic.com/en/page1 for pages in english with all the 301 redirects required... Thank you in advance for your answers.
Intermediate & Advanced SEO | | Maxxum0