Disallow indexing of ALL subdomains
-
I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:
project1.domain.com
project2.domain.com
project3.domain.com
...Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.
Is this possible?
-
Hello there!
Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).
Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.
I hope this help!
-
Nobody?
-
Thanks for the link, but I don't think that will solve my problem?
When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?
-
You should edit your .htaccess file as is described here:
http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-repositI hope i answered your question
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Issue to index AMP pages
Hello In googe search console i have about 500 pages indexed for my websites https://horaire-priere.be/ and https://horarios-oracion.es/ but only 20 pages are indexed as AMP. Butmy site is amp only. I can't understand why google indexed the page but not in amp mode? Thank you in advanced
On-Page Optimization | | Zakirou0 -
Meta tags not being displayed for Mobile version of a blog even after the blog is indexed
I have updated the meta title and meta description for my blog. After updating the meta tags, blog URL was submitted in the webmasters and it has been crawled and indexed by Google but only for the desktop version and not for mobile version. Currently, meta tags are not being displayed for mobile version in Google. I have also checked with robots.txt file, there is no problem with this file. PFA screenshots for reference. Kindly help me out with this issue. eWu1k8N PdimKGq
On-Page Optimization | | priyankadesai1230 -
Indexed Link Removal request in GWT, good idea?
Hello, I used a plugin to no-index a lot of pages on my website, and its been couple months and they never disappeared from serps, so i used the google webmaster tool "remove urls" to ask google to remove them. Is that a good idea? Or does it look bad in googles eyes? any thoughts would help a lot. Thanks
On-Page Optimization | | Rank-and-Grow0 -
Subdomain SERPS are different from the root domain SERPS
I am fairly new to SEO so bare with me. I created two campaigns within Moz, one tracking the sudomain www.mysite.com and another tracking just the root domain mysite.com. When I look at the keyword rankings I get different results from Google, Bing and Yahoo. For instance, the root domain shows rankings in Bing and Yahoo, but "Not in top 50" for Google. For my subdomain, it is reversed. I see rankings for Google, but "not in top 50' for Bing and Yahoo. What can I do to solve this issue? I am using canonical tags, but it is a domain redirect issue? Thanks in advance
On-Page Optimization | | BouldersInn0 -
No index, or no index no follow?
Wondering if I could garner some views on this issue please. I'm about to add an affiliate store to a website I own, the site has a couple of pages of unique content (blogs, articles, advice etc on home improvement - all written by my team). Obviously, the affiliate store will not be unique content, it will be made using the datafeeds from cj.com et al, and so I don't want to get any duplicate content type penalties from Google for this store. Should I add a no index to the pages and allow the bots to still crawl them, or should I add no index and no follow? Ideally I would like to get the affiliate store category pages indexed as they will be a mixture of lots of different merchants and be fairly unique. Can Google still mark the site down for duplicate content if it can crawl it, even if it is noindex? Thanks, Carl
On-Page Optimization | | Grumpy_Carl0 -
Anyone know how long it takes Google to Index new site?
Could anyone let me know how long it takes for a NEW site to be indexed in Google please? Am having some robots.txt issues and am keen to see if it got indexed. Thanks!
On-Page Optimization | | Wallander0 -
Advice on why page isn't being indexed in google top 1000 for keyword
Hi, http://www.cgcomposites.com.au/composite-material.html This page isn't listed for keywords 'composite material' It has been live for a few weeks and gets grade A report for onpage optimisation? regards Michael
On-Page Optimization | | bluelilyseo0 -
Subdomains vs. Subfolders Inheriting Authority/Ranking Value
Our website is a continuing education website that is linked to a large university, and our URL is a subdomain of that larger university domain. We offer degrees as well, but because of the modifications we'd like to make to the degree webpages, our content management system won't let them be a part of our website. Now we're trying to figure out if we should create a separate subdomain for all degrees, and put the individual degrees in separate folders (so, all degrees' URLs woudl be degrees.us.university.edu/degree-name/), or if we should give each its own URL, which would be completely separate from ours (degree urls would be degree-name.university.edu). So our question is, how well do subdomains carry the value of the domain? Is it better to have twenty websites that are all separate subdomains of a strong domain, or one subdomain in a subdomain that houses all twenty websites in folders? And, as a side note, will housing the degrees in degrees.us.university.edu pass value to us (us.university.edu)? Thanks!
On-Page Optimization | | UWPCE0