Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
-
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare).
We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain.
If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains?
Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error!
Cheers,
Dave.
-
Hi Dave. A workflow checklist should really help with this as well. There are probably a few other items you'll catch by meeting with the others involved and getting everyone on the same page. Cheers!
-
Dave, I had exactly the same issue a month ago with being indexed on subdomains but was able to modify robots.txt in root domain swiftly enough to avoid real damage. Your main root robots.txt can override the subdomains. Simply disallow the subdomain in your robots file disallow: /subdomain url/robots.txt OR, if im not mistaken simply remove it altogether from the subdomain
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can you transcribe your youtube video to just google?
YouTube videos have been very effective for us on our site. The only problem is they have taken part of a fair amount of our content so to google our pages may seem to have minimal content. Transcribing each video and including that on the page would take away from the look of the page and clutter it. Is there a way to let google know the content without having to include it on the page? You can find an example page at https://www.waikoloavacationrentals.com/halii-kai-rentals/
On-Page Optimization | | RobDalton0 -
We have new website abexpress.ae. The website is not ranking in the first page when I search with the Domain name itself. Any tips to improve the SEO for a new website?
We have launched a website abexpress.ae. The website is not ranking well even if we search with the Domain name. The Domain authority of the website it 10. How to increase this?
On-Page Optimization | | AlliedTransport0 -
Is a mobile domain necessary?
Greetings all, I'm starting to build a mobile version of our website to be ready for April 21st. Question is: is it absolutely necessary to have a mobile domain name or can I just direct users to same domain/folder? in other words: www.mywebsite.com/mobile vs m.mywebsite.com - does this even matter? As always, thank you!
On-Page Optimization | | md30 -
Duplicate content issue, across site domains (blogging)
Hi all, I've just come to learn that a client has been cross-posting their blog posts to other blogs (on higher quality domains, in some cases). For example - this is the same post on 3 different blogs. http://thebioethicsprogram.wordpress.com/2014/06/30/how-an-irb-could-have-legitimately-approved-the-facebook-experiment-and-why-that-may-be-a-good-thing/
On-Page Optimization | | ketanmv
http://blogs.law.harvard.edu/billofhealth/2014/06/29/how-an-irb-could-have-legitimately-approved-the-facebook-experiment-and-why-that-may-be-a-good-thing/
http://www.thefacultylounge.org/2014/06/how-an-irb-could-have-legitimately-approved-the-facebook-experimentand-why-that-may-be-a-good-thing.html
And, sometimes a 4th time, on an NPR website. I'm assuming this is doing no one any favors and Harvard or NPR is going to earn the rank most every time. I'm going to encourage them to publish only fresh content on their real blog, would you agree? Can this actually harm the ranking of their blog and website - should we delete the old entries when migrating the blog? They are going to move their Wordpress Blog to hosting on their real domain soon:
http://www.bioethics.uniongraduatecollege.edu/news/ The current set up is not adding any value to their domain. Thank you for any advice! Ketan0 -
Can a page lose ranking because it has too many bold tags?
I run a product search website for the Engineering Sector. Each of the companies listed on the website has a profile page. This page used to rank very well in google. Over the last couple of weeks these pages disappeared for google i.e they fell about 20 to 30 spots in the search results for relevant keywords like the Manufacturer Name. I was trying to analyze the page and found that Each of the Categories that the manufacturer supports was listed in "bold" and linked to the page for that category. A manufacturer can support up to 20 or 30 categories so this results in about 20 or 30 Bold keywords. Is this a bad practice? Could this be a reason for a drop in ranking or do bold keywords on a page not matter? What are your views?
On-Page Optimization | | raghavkapur0 -
Can I add a paragraph of blog content to a product page?
I have a great blog post about one of my products, which means there's some copy in the blog that I would like in the product page. It would constitute perhaps 5% of the copy on the product page if I copied it.
On-Page Optimization | | Brocberry0 -
Where can I find broken link.
Hi there, I got some 404-errors in crawl diagnostics. How can I get the URLs where the broken links were found?
On-Page Optimization | | fleetway0 -
Robots.txt file
Does it serve any purpose if we omit robots.txt file ? I wonder if spider has to read all the pages, why do we insert robots.txt file ?
On-Page Optimization | | seoug_20050