Disallow indexing of ALL subdomains
-
I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:
project1.domain.com
project2.domain.com
project3.domain.com
...Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.
Is this possible?
-
Hello there!
Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).
Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.
I hope this help!
-
Nobody?
-
Thanks for the link, but I don't think that will solve my problem?
When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?
-
You should edit your .htaccess file as is described here:
http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-repositI hope i answered your question
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
titles length, URL length and meta descriptions on a subdomain effecting SEO on main domain?
Hi all, I am currently evaluating areas for optimization on my main domain. When doing this, Moz has identified multiple titles and urls that should be shortened and missing meta descriptions on my subdomain (a help center of sorts). As far as I am aware, we have not set up any "no-index" rules for this subdomain. Are these items affecting SEO on my main domain? Thanks,
On-Page Optimization | | annegretwidmer
Kasey0 -
Site no indexed after a week loadbalancer, cache-control?
hi All apologies to everybody in advance my level of SEO or technical stuff is quite minimal. we have setup a new site with a top level domain sitting on multiple servers ( https.www.mysite/new/ on one and http.wwww.mysites/old/ on another) different pages,content etc. it's definitely a patchy solution until what is left in the old server will be migrated to the new one. however the new site is still not indexed after around 2 weeks. I have checke on moz and and google console ( fetch and rendering) and nothing seem to block indexing. no issue with robot.txt ( either on https.www.mysite/new/robots.txt or http.wwww.mysites/old/robots.txt) meta-robot. any idea on what could cause the problem? can it be an issue with the cache-control set as : no-store no-cache? Or the loadbalancer that is preventing google to access ? Dario
On-Page Optimization | | Mrlocicero0 -
Should I "No Index" Certain Pages On My Site?
I have some pages on my site that don't really have any content other than some iframes that are embedded from another site. I thought it would be best to tag the page with a no-index so that search engines would leave the page alone since it has no content as far as the search engine can tell (but does provide value to my site visitors). Is this the proper approach or does it do more harm than good?
On-Page Optimization | | Kyle Eaves0 -
Using a subdomain to improve rankings
I have a pretty simple landing page that at present targets several keywords. Is a good strategy to add a blog to the site on a subdomain with pages targeting individual keywords? Will this help the main domain rank? Also is the best strategy to focus on building links to the home page or individual pages on the subdomain? As long as each page on the subdomain links back to the home page it should pass link juice back to the home page shouldn't it? Thanks in advance!
On-Page Optimization | | SamCUK0 -
Why isn't our site being shown on the first page of Google for a query using the exact domain, when its pages are indeed indexed by Google
When I type our domain.com as a query into Google, I only see one of our pages on the homepage, and it's in 4th position. It seems though, that all pages of the site are indexed by google when I type in the query "site:domain.com". There was an issue at the site launch, where the robots.txt file was left active for around two weeks. Would this have been responsible for the fact that another domain ranks #1 when we type in our own domain? It has been around a couple of months now since the site was launched. Thanks in advance.
On-Page Optimization | | featherseo0 -
Latent semantic Indexing - Does this help rankings/relevance?
Hi, Does semantically related words to the target term on a page help with rankings/relevance? If your after the term 'PC Screen' and you use the term 'PC Monitor' will go make the connection and also reward you because of the relevance? Anyone do this and have you seen any positives? I've just started to try this out lately and have been combining it with Wordle.net to give me an indication of where the content piece is heading and how aggressive the content leans towards certain words (makes things a little more interesting then calculating densities).
On-Page Optimization | | Bondara0 -
Subdomain vs. Root Domain design
I know that there are penalties if you have a navigation item in your categories that links to another site (root domain) that yields the same design as the original link. Is there any negative SEO implications with having a subdomain with the same design as the root domain i.e. root.com vs. store.root.com And the navigation functionality within will have links to both store.root.com as well as root.com directories? Thanks in advance.
On-Page Optimization | | customerparadigm.com0 -
How Can I Get Yahoo to Index My Site?
How Can I Get Yahoo to Index My Site? I have installed Bing webmaster tool two months ago -- is Yahoo that slow. My site has been out since May 2010 and for a year and a half, I only have 40 pages index. HELP!!!!
On-Page Optimization | | AppleCapitalGroup0