Blocking subdomains without blocking sites...
-
So let's say I am working for bloggingplatform.com, and people can create free sites through my tools and those sites show up as myblog.bloggingplatform.com. However that site can also be accessed from myblog.com.
Is there a way, separate from editing the myblog.com site code or files, for me to tell google to stop indexing myblog.bloggingplatform.com while still letting them index myblog.com without inserting any code into the page load?
This is a simplification of a problem I am running across.
Basically, Google is associating subdomains to my domain that it shouldn't even index, and it is adversely affecting my main domain. Other than contacting the offending sub-domain holders (which we do), I am looking for a way to stop Google from indexing those domains at all (they are used for technical purposes, and not for users to find the sites).
Thoughts?
-
Ah, I see now. Try this out http://moz.com/community/q/block-an-entire-subdomain-with-robots-txt#reply_26992 - basically, when a subdomain is identified, it would pull a different file into the robots.txt location (which would contain the disallow: / syntax)
Read the remaining comments about getting the subdomain removed via GWT.
-
You are correct, but that isn't what I was asking.
user1.bloggingplatform.com and myblog.com point to the same web server files. If I put up a robots.txt on user1.b... I would effectively de-index myblog.com.
The problem we have run accross is that user205.bloggingplatform.com might be doing something shady, but instead of de-listing the subdomain google kills the primary domain from the index as well.
Because user205.bloggingplatform.com should only be used for technical reasons, and not be in Google's index I am looking for a way to tell google not to index the sub-domain.
I think the better way to solve the problem would be to change the technical subdomain's domain though so change it from user205.bloggingplatform.com to user205.bloggingplatformtesting.com.
Then google can kill that URL all it wants as I don't care.
-
bloggingplatform.com/robots.txt
and
user1.bloggingplatform.com/robots.txt
can and should be different. If you disallow at the subdomain level, only the subdomain will be affected. You can search around for other examples of this but i'm certain it works (we have a development domain that is indexed and create subdomains for all clients that aren't indexed and done via individual robots.txt files)
-
I don't think that works. Since both URLs point to the same server the robots.txt file for the test URL would completely kill the main url.
Or am I missing something?
-
Each subdomain should have a robots.txt file that blocks that specific subdomain. e.g. user1.bloggingplatform.com/robots.txt should have:
User-agent: *
Disallow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Main Site and eCommerce Site URLs for SEO
My client currently has a main website on a url and an eCommerce site on a subdomain. The eCommerce site is currently not mobile friendly, has images that are too small and are problematic - and I believe it negates some of the SEO work we do for them. I had to turn off Google Shopping ads because the quality score was so low. That being said, they are rebuilding a shopping cart on a new platform that will be mobile friendly BUT the images are going to be tiny until they slowly replace images over several months. Would you keep the shopping cart on a subdomain, or make it part of the main website URL? Can it negatively impact the progress we have made on the main site SEO.
Technical SEO | | jerrico10 -
Subdomain Ranking Question
Hi All - Quick question that I think I know the answer to, but I feel like I've been going around in circles a bit. My client is launching a new product and wants us to build a microsite for it (product.clientname.com). My client really dislikes their brand website, and wants to use paid media to push their audience to this new microsite. However, they also said want it to rank well organically. I feel uneasy about this, because of the subdomain vs. subfolder argument. I believe that the product will also be listed/featured on their main brand website. What is the best way forward? Thanks!
Technical SEO | | AinsleyAgency0 -
Blog on subdomain of e-commerce site
Hi guys. I've got an e-commerce site which we have very little control over. As such, we've created a subdomain and are hosting a WordPress install there, instead. This means that all the great content we're putting out (via bespoke pages on the subdomain) are less effective than if they were on the main domain. I've looked at proxy forwarding, but unfortunately it isn't possible through our servers, leaving the only option I can see being permenant redirects... What would be the best solution given the limitations of the root site? I'm thinking of wildcard rewrite rules (eg. link site.com/blog/articleTitle to blog.site.com/articleTitle) but I'm wondering if there's much of an SEO benefit in doing this? Thanks in advance for everyone's help 🙂
Technical SEO | | JAR8970 -
Do you think this site has been hit by penguin?
Hi Guys, I need some opinion on a website i am working on www.colourbnners.co.uk They updated their website in August but the company they used did not take into account the URL structure and hence there's a massive loss in links in August time. They also dropped off Google for all their key terms except their brand name 'colour banners'
Technical SEO | | gezzagregz
Since then, they have implemented a 301 redirect. Some key points They have not received any manual warnings in WMT I have disavowed some poor quality links that they have built over the years I am building high quality links quite selectively/slowly There were a lot of duplicate content issues - these have been resolved now.
So my question to you SEO pros is do you think its penguin? or something that i am missing?
If it is penguin, what is the best form of attack to get it removed? regards gezzagrez0 -
I noticed all my SEOed sites are getting attacked constantly by viruses. I do wordpress sites. Does anyone have a good recommendation to protect my clients sites? thanks
We have tried all different kinds of security plugins but none seem to work long term.
Technical SEO | | Carla_Dawson0 -
Site Categorization?
I know getting site categories to appear under the site are dependent on a lot of factors including site mapping. We have a site that does the categorization thing when you type in the sites url name however more people search for the name of the talent to find the site and the short url on the site is just his name, but shorter. However I was just wondering is their a way to optimize the site so that way we could get categorization to show up under the sites URL when they search for the talents full name I ask because the amount of people looking for the talents full name rather than the short name is a lot larger and I would like to see if we can take advantage of the real estate, but I honestly don't think there is a way, however I figured I would open it up to discussion to see if anyone has any ideas. Example: Site name is ABCD you type this into Google and you get ABCD.com about blog how to contact However the actual person whose site it is is ABCDEF and when you type that in you just get: ABCD.com without any of the categories appearing below the url. And that is what I'm asking about. Thanks as I can't seem to find a lot of information on this. However if there is another spot on the site talking about this please let me know I may just not be searching with the right terms.
Technical SEO | | KateGMaker0 -
Local business with multiple sites
I'm auditing a local business' sites (a spa) and I wanted to run my recommendations by everyone. There are 3 sites: www.sitename1.com -- main store location, used for Google Places listing #1 www.sitename2.com -- 2nd store location, used for Google Places listing #2 www.sitename3.com -- used for product sales for both locations Sitename1.com has the most ranking power. I'm going to recommend that they move sitename2.com and sitename3.com to sitename1.com as subfolders, 301 redirecting each page to the corresponding page on sitename1.com/subfolder. Google Places listing #2 would be changed from www.sitename2.com to www.sitename.com/location2. Any risks or problems with this strategy anyone can see?
Technical SEO | | 540SEO0 -
Merging two sites into one
I have two websites, www.joecline.com and www.affinityproperties.com. Originally, I wanted joecline.com to be focused on west austin only. I now realize that I only have time to really handle the promotion of one site. I'm wondering SEOwise, which url would be better to keep. I would consolidate all content on one site and 301 the migrated site to the new URLs in the domain which I would keep. Any ideas would be great. Thanks Joe
Technical SEO | | simplesimon0