How can I prevent Google and other search engines to crawl my secured pages (https:)?
-
Let me know your thoughts guys. Thanks in advance!
-
Your best bet is to place a meta noindex tag on each secure page. If it's only a few pages, you could just add it by hand. If it's many, you should be able to access each pages protocol with whatever server-side language you're using, and dynamically add it on all secure pages.
If you use robots.txt to exclude the pages, Google can still show them in search results, with the description below them that reads, "A description for this result is not available because of this site's robots.txt – learn more." Personally, I don't care for that.
-
Hi there, blocking the HTTPS version of your pages from being crawled by the search engines is a bit tricky. You might need to come up with a separate robots.txt file to handle the HTTPS requests.
Here you go to know more about the process:
http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html
Hope it helps.
Best,
Devanur Rafi
-
Hi esiow
You have a choice of placing a robots.txt file in the root folder of your website or if blocking individual pages you could use the meta robots tag. See these page for more information: http://moz.com/learn/seo/robotstxt and https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1
I hope that helps,
Peter
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Dealing with the impending Google mobile compliance update - is bMobilized any good as a temporary measure?
We've been caught a bit off guard with the upcoming Google mobile compliance issue and received the warnings in webmasters about fixing mobile usability issues. It will still be some time before we can have the site re-coded as responsive. I have stumbled upon a converter tool (which turns any site into one that's mobile friendly) called bMobilized, which essentially turns your site into one that's mobile friendly. Have you used bMobilized? Is it a safe idea to use this service temporarily until our new responsive site is ready in a few months? Do you have any suggestions for temporarily getting around the mobile compliance issue while our new responsive site is being built? Thanks
Web Design | | Martin_S0 -
H Tags for an Events Page
I wanted to get the thoughts of people here about how to best structure an events listing page for SEO. I have a list of events, all with dates, event titles, location name, city and zip. What I do currently is listed below. I also show a version for how I could revise it, but it would require me to duplicate the event date on the page. Any ideas, suggestions or best practice examples you can point me to would be greatly appreciated. Current Structure <state>Events - H1 Tag
Web Design | | abiondo
Friday, December 5, 2014 - H2 Tag
Event Title 1 - H3
Location Name, City, State - P Tags Event Title 2 - H3
Location Name, City, State - P Tags</state> I was wondering if I would see better results by doing the following instead. The benefits I see of this approach are the event titles are h2 instead of h3 tags and the con I see is duplicating the event dates <state>Events - H1 Tag</state>
Event Title 1 - H2
Friday, December 5, 2014
Location Name, City, State - P Tags
Event Title 2 - H2
Friday, December 5, 2014
Location Name, City, State - P Tags thanks, Anthony0 -
Are these doorway pages or not? Concerned due to Panda 4.0
For a new site we're building, the Products team wants the header (let's call this Product-Header) to have links to every subsection of every section on every page. Since this is a bad idea, I want Product-Header to be coded in such a way that it doesn't appear in the code or the links are nofollow, noindex. I want to instead create static versions of these pages without the Product-Header. The homepage links to the static URL section pages, those main section pages link to static subsection pages, and so on. It's one nice silo. I am concerned though that Google won't like this due to these static pages are being created specifically for search engines. Users could click through to this static parallel site from the homepage, or they could use the dynamic URL site. This is similar to what etsy.com is doing where you can search Google for "mermaid bridal" and get this page https://www.etsy.com/market/mermaid_bridal but the dynamic version of the page does not show up. However you can search on etsy.com for " mermaid bridal" and get https://www.etsy.com/search?q=mermaid bridal&ship_to=US. Could these static versions that show up in search engines be seen as doorway pages? I know ebay.com got spanked for doorway pages and I don't want to do anything that would get this site penalized.
Web Design | | CFSSEO0 -
Moz crawl showing up ?s=keyword pages as errors
Hi all, Hoping someone can she some light on a fix with ref to wordpress and the search function it uses as Moz is craling some pages which reference the search domain.com/?s=keyword Errors showing up are duplicate pages, descriptions and titles. The search function is not important on this site and I have tried to use a plugin which disables the search page which it does but these errors still show up. Can anyone assist as this is the final piece of the puzzle and then we're down to 0 issues on the site.
Web Design | | wtfi0 -
How can a Pincode finder website be SEO optimised?
Guys, I wanted to build a simple Pincode finder website for India. The targeted visitors as is obvious will be from India. Alike other Pincode finder websites, the users in this case too will have to key in the location / area of whose pincode he is looking for and they will get Pincode from that very location / area. Other than this, users will also come to this website when they search for something like " <location name="">pincode</location>" on Google (for instance, users will search for something like "Hiranandani Gardens Powai Pincode") Along with data fethced from our sources via Indian postal departments and other data available in public domain, we shall be using data from Google Maps API too. My question in regards to the same is as follows: What should the page-structure / structure of the website be for ranking well on Google? What should be the URL structure? Other suggestions to rank well on Google in this regards? Competition: (You can search for the term "Hiranandani Gardens Powai Pincode" to know how these sites show data) http://www.getpincode.info http://www.pincode.net.in Pls. help...
Web Design | | ShalinTJ0 -
Site as one page - SEO implications
We may be inheriting a site and will be asked to do SEO for it. We will have control over the development of the site, so this structure is what it is. My question is - how significant of an impact do you think this is going to have and can you think of any workarounds that may help? Basically, the user experience of the site will feel similar to multiple pages. However, this site will, in essence be one page and pull various content through javascript from different locations. I have not seen the site yet (and believe it is still in development), but this is how it has been explained to me. Any thoughts? My first thought was to add a blog to add page depth to the site and expand the content. Any other thoughts are welcome and appreciated. Thanks. (I know this is limited information, I'm sorry. It's just about all I have to work with right now, and I was a little concerned and was hoping for a second opinion)
Web Design | | AdamWormann0 -
Does stock art photo attribution negatively impact SEO by leaking Google Page Rank?
Greetings: Companies such as Shutterstock often require that buyers place credit attribution on their web pages when photos you buy from them appear on these pages.. Shutterstock requests that credit attribution links such as these be added: Songquan Deng / Shutterstock.com Do these links negatively impact SEO? Or do search engines view them as a positive? Thanks,
Web Design | | Kingalan1
Alan0 -
Are slimmed down mobile versions of a canonical page considered cloaking?
We are developing our mobile site right now and we are using a user agent sniffer to figure out what kind of device the visitor is using. Once the server knows whether it is a desktop or mobile browser it will deliver the appropriate template. We decided to use the same URL for both versions of the page rather than using m.websiteurl.com or www.websiteurl.mobi so that traffic to either version of these pages would register as a visit to the page. Will search engines consider this cloaking or is mobile "versioning" an acceptable practice? The pages in essence are the same, the mobile version will just leave out extraneous scripts and unnecessary resources to better display on a mobile device.
Web Design | | TahoeMountain400