Welcome to the Q&A Forum

Thanks Logan,

I have read somewhere that using Host directive in the robots.txt file we can suggest Google bots which is the original version of the website if there are number of mirror sites. So, I was wondering if we can prevent indexing/crawling of HTTPS URLs by using Host directive in robots.txt of HTTP site.

We are using an ecommerce SAAS platform for our website where we have only one robots.txt file that we can use for HTTP site.

Is there any other way to prevent indexing/crawling of HTTPS URLs?

Regards,
Ramendra

Hello Moz Community,

Recently, I have found that Google bots has started crawling HTTPs urls of my website which is increasing the number of duplicate pages at our website.

Instead of creating a separate robots.txt file for https version of my website, can I use Host directive in the robots.txt to suggest Google bots which is the original version of the website.

Host: http://www.example.com

I was wondering if this method will work and suggest Google bots that HTTPs URLs are the mirror of this website.

Thanks for all of the great responses!

Regards,
Ramendra