Robots.txt blocking Addon Domains

[eglove]

I have this site as my primary domain: http://www.libertyresourcedirectory.com/

I don't want to give spiders access to the site at all so I tried to do a simple Disallow: / in the robots.txt. As a test I tried to crawl it with Screaming Frog afterwards and it didn't do anything. (Excellent.)

However, there's a problem. In GWT, I got an alert that Google couldn't crawl ANY of my sites because of robots.txt issues. Changing the robots.txt on my primary domain, changed it for ALL my addon domains. (Ex. http://ethanglover.biz/ ) From a directory point of view, this makes sense, from a spider point of view, it doesn't.

As a solution, I changed the robots.txt file back and added a robots meta tag to the primary domain. (noindex, nofollow). But this doesn't seem to be having any effect. As I understand it, the robots.txt takes priority.

How can I separate all this out to allow domains to have different rules? I've tried uploading a separate robots.txt to the addon domain folders, but it's completely ignored. Even going to ethanglover.biz/robots.txt gave me the primary domain version of the file. (SERIOUSLY! I've tested this 100 times in many ways.)

Has anyone experienced this? Am I in the twilight zone? Any known fixes? Thanks.

Proof I'm not crazy in attached video.

robotstxt_addon_domain.mp4

[eglove]

Sort of resolved, maybe the wrong place to ask any further. The above is a working fix for what seems like a legit bug, I'll update if WordPress forums say anything.

[eglove]

No, I don't like to waste memory and bandwidth. If you can do it yourself, you should probably do it yourself. I'm moving this question to WordPress.

[abiondo]

Hi Ethan

One thing I have heard of people trying is a plugin that serves dynamic robots.txt files.  I don't use add-on sites so you will probably have to test the behavior.  He is an example of one of the plugins.

https://wordpress.org/plugins/wp-robots-txt/

hope this helps,
Anthony

[evolvingSEO]

Ethan

It sounds like the issue has been resolved. I'm not too familiar with domain add-ons but if you have any more trouble let us know and I'll be sure another Moz Associate takes a look.

-Dan (Moz Associate)

[eglove]

Then I'll say it again. I want the file for my other sites. I doesn't work to put it in the add-on domain folders. To have a robots.txt for EG.biz, it must be in the LRD.com folder.

[abiondo]

Hi Ethan

Sorry, I wasn't clear.  I was thinking you could drop the use of the robots.txt all together and just use the Meta Tag approach since it seems that the robots.txt is having a global impact to your sites.  Search engines will still crawl the pages, but it should exclude them from the index.

Hope this helps,
Anthony

[eglove]

Anthony, based on your response it's obvious you haven't read the question or follow-up.

[abiondo]

Hi Ethan

One approach may be to try using the Robots Meta Tag.  You can use noindex to tell Google not to index.  This won't prevent crawling, but Google should respect the request to not index your site.  I have included a good guide below to get you started.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com

[eglove]

I've found a quick fix for now: http://ethanglover.biz/using-robots-txt-with-addon-domains/

This is still an issue, and it may be exclusive to WordPress.