SSL and robots.txt question - confused by Google guidelines
-
I noticed "Don’t block your HTTPS site from crawling using robots.txt" here: http://googlewebmastercentral.blogspot.co.uk/2014/08/https-as-ranking-signal.html
Does this mean you can't use robots.txt anywhere on the site - even parts of a site you want to noindex, for example?
-
Hi Luke,
Just make sure that your robots.txt file located at https://www.example.com/robots.txt doesn't block search engine spiders. Of course there may be some folders or filetypes you want to block but it certainly shouldn't look like below which would block everything:
User-agent: *
Disallow: /
Hope that helps
-
No that's not what they mean - it means Google recommends you allow the secure version of your site(where applicable) to be crawled. You can still block certain pages/sections should you choose to do so.
With regards to noindexing you could also place this on the actual page as an alternative.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google disavow file
Does anybody have any idea how often Google reads the disavow file?
Intermediate & Advanced SEO | | seoman100 -
Google Update
My rank has dropped quite a lot this past week and I can see from the Moz tools that there is an unconfirmed Google update responsible. Is there any information from Moz on this?
Intermediate & Advanced SEO | | moon-boots0 -
How to make Google index your site? (Blocked with robots.txt for a long time)
The problem is the for the long time we had a website m.imones.lt but it was blocked with robots.txt.
Intermediate & Advanced SEO | | FCRMediaLietuva
But after a long time we want Google to index it. We unblocked it 1 week or 8 days ago. But Google still does not recognize it. I type site:m.imones.lt and it says it is still blocked with robots.txt What should be the process to make Google crawl this mobile version faster? Thanks!0 -
Not ranking on Bing but is on Google?
Hi What are the main differences between Bing and Google in terms of ranking sites? My site is ranking well in Google but in Bing it is very low down and does not deliver much traffic. In Bing webmaster tools there are no warning messages and I had sent in a sitemap back in 2011 and 77 pages are listed, but I had not submitted a URL could this be why my pages are not ranking highly? Or does anybody have a checklist on what a site should offer to get ranking on Bing?
Intermediate & Advanced SEO | | ocelot0 -
GOOGLE Rankings
We built a site for one of our clients about 3 years ago and added Meta tags to each page of the site. But when you do a google search on the keywords the site still does not come up. I thought it would be spidered into the system by now. Web Site - www.specialtysealgroup.com Let me know what we can do to improve our GOOGLE rankings.
Intermediate & Advanced SEO | | thine1230 -
Canonical tag question
Suppose a site has two pages ( Page A ) and Page B. Both of them have pagerank, but duplicate content. The page A is ranked for keyword "seo india" and page B is ranked for keyword "seo services". If i implement canonical tag on page B, does 1. The pagerank of page B will be transfered to Page A ? 2. Does the site A now ranks for keyword "seo servicies " ( for which Page B was ranking earlier )
Intermediate & Advanced SEO | | seoug_20050 -
XML Sitemap instruction in robots.txt = Worth doing?
Hi fellow SEO's, Just a quick one, I was reading a few guides on Bing Webmaster tools and found that you can use the robots.txt file to point crawlers/bots to your XML sitemap (they don't look for it by default). I was just wondering if it would be worth creating a robots.txt file purely for the purpose of pointing bots to the XML sitemap? I've submitted it manually to Google and Bing webmaster tools but I was thinking more for the other bots (I.e. Mozbot, the SEOmoz bot?). Any thoughts would be appreciated! 🙂 Regards, Ash
Intermediate & Advanced SEO | | AshSEO20110 -
10,000 New Pages of New Content - Should I Block in Robots.txt?
I'm almost ready to launch a redesign of a client's website. The new site has over 10,000 new product pages, which contain unique product descriptions, but do feature some similar text to other products throughout the site. An example of the page similarities would be the following two products: Brown leather 2 seat sofa Brown leather 4 seat corner sofa Obviously, the products are different, but the pages feature very similar terms and phrases. I'm worried that the Panda update will mean that these pages are sand-boxed and/or penalised. Would you block the new pages? Add them gradually? What would you recommend in this situation?
Intermediate & Advanced SEO | | cmaddison0