How can I prevent Google and other search engines to crawl my secured pages (https:)?
-
Let me know your thoughts guys. Thanks in advance!
-
Your best bet is to place a meta noindex tag on each secure page. If it's only a few pages, you could just add it by hand. If it's many, you should be able to access each pages protocol with whatever server-side language you're using, and dynamically add it on all secure pages.
If you use robots.txt to exclude the pages, Google can still show them in search results, with the description below them that reads, "A description for this result is not available because of this site's robots.txt – learn more." Personally, I don't care for that.
-
Hi there, blocking the HTTPS version of your pages from being crawled by the search engines is a bit tricky. You might need to come up with a separate robots.txt file to handle the HTTPS requests.
Here you go to know more about the process:
http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html
Hope it helps.
Best,
Devanur Rafi
-
Hi esiow
You have a choice of placing a robots.txt file in the root folder of your website or if blocking individual pages you could use the meta robots tag. See these page for more information: http://moz.com/learn/seo/robotstxt and https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1
I hope that helps,
Peter
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why would a developer build all page content in php?
Picked up a new client. Site is built on Wordpress. Previous developer built nearly all page content in their custom theme's PHP files. In other words, the theme's "page.php" file contains virtually all the HTML for each of the site's pages. Each individual page's back-end page editor appears blank, except for some of the page text. No markup, no widgets, no custom fields. And no dedicated, page-specific php files either. Pages are differentiated within page.php using: elseif (is_page("27") Has anyone ever come across this approach before? Why might someone do this?
Web Design | | mphdavidson0 -
Website Home page suddenly disappeared after changing Hosting
HI All, My site was ranking very well and was in 1st page of google for most of my keywords. Last week we did some update to the site and moved it to new hosting and from then onwards I dont see my site home page in Google ranking . My Website Name is : royalevents.com.au. We used to be in 1st of Google for keywords like wedding Mandaps, Indian Wedding Mandaps etc, Would be great if some one helps us to figure out whats gone wrong .. I also did Webmaster Fetch as Google but nothing happened. Thanks
Web Design | | Verve-Innovation0 -
H Tags for an Events Page
I wanted to get the thoughts of people here about how to best structure an events listing page for SEO. I have a list of events, all with dates, event titles, location name, city and zip. What I do currently is listed below. I also show a version for how I could revise it, but it would require me to duplicate the event date on the page. Any ideas, suggestions or best practice examples you can point me to would be greatly appreciated. Current Structure <state>Events - H1 Tag
Web Design | | abiondo
Friday, December 5, 2014 - H2 Tag
Event Title 1 - H3
Location Name, City, State - P Tags Event Title 2 - H3
Location Name, City, State - P Tags</state> I was wondering if I would see better results by doing the following instead. The benefits I see of this approach are the event titles are h2 instead of h3 tags and the con I see is duplicating the event dates <state>Events - H1 Tag</state>
Event Title 1 - H2
Friday, December 5, 2014
Location Name, City, State - P Tags
Event Title 2 - H2
Friday, December 5, 2014
Location Name, City, State - P Tags thanks, Anthony0 -
Will changing product from Grouped to Simple on my magento category page affect my SEO?
Hi all, A category page on my site http://www.porcelainsuperstore.co.uk/wood-effect.html currently ranks number 3 on Google for the keyword "Wood Effect Tiles" We're currently reorganising some of our product and I would like to know if this is going to affect the SEO and ranking for the above page and keyword. The majority of products on that page are magento grouped products. I would like to change the page so that it displays only the different constituent simple products rather than the grouped products on the category page. My question is, will this have any impact on SEO? I intend on leaving all other data on the category page the same - so the metadata and the description/title etc. Any help/comments would be much appreciated! Ben
Web Design | | piazza0 -
Does google prefer expanded text to text that you have to mouse over to show?
Does a long scrolling page of text perform better than a page that has the content in sections that have to be moused over to be seen? Are there any articles or research on this?
Web Design | | SirSud0 -
Any way of showing missed sales in Google Analytics?
Sit down, this might get a little complicated... I was approached by a design company to do some SEO work a client of theirs. Usually, this stuff is white label but I have direct contact with the client as the design agency felt it was easier for me to do this. The website is performing really well and looking at the sales funnel, I'm getting people wanting to buy. BUT, and here's the problem, everything falls apart because of the way the check out works. It's appalling. The customer has to register to buy a product, there's no guest check out or anything. The checkout button is right below the fold and you'd miss it completely if you didn't actually search for it. Basically, it's losing the client money. Last month alone there were 300~ people entering the conversion funnel and NONE of them complete it. I've been talking with the design company and they basically saying that it's too much work for them to change it, it's a signed off project blah blah. UI reports have been conducted and sent to them but still nothing. I have the client asking (a great client, obviously wondering why there is a lack of return on his investment) why he isn't making money. He's asking me because I'm the guy thats meant to be getting him the cash back. I keep saying to the design agency the problems and that it's never going to make money. The potential is massive. But thats my problem. Is there ANY way in GA to calculate the missed sales? I know that I can view the total amount made when the customer successfully checks out but I need figures to present that I'm leading the horse to water, but the check out system is preventing it from drinking. tl;dr I need to show client/design agency missed sales due to poorly built checkout system. Cheers!
Web Design | | jasonwdexter0 -
How can we improve our e-commerce site architecture to help best preserve Page Authority?
Today I installed the SEOMoz toolbar for Firefox (very cool, highly recommended). I was comparing our site http://www.ccisolutions.com to this competitor: http://www.uniquesquared.com For the most part, the deeper I go in our site the more the page authority drops. We have a few exceptions where the page authority of a subcategory page is actually better than the cat. page one level up. In comparison, when I was looking at http://www.uniquesquared.com I noticed that their page authority stays at "21" on every single category page I visit. Are you seeing what I'm seeing? Is this potentially a problem with the tool bar or, is there something significantly different about their site architecture that allows them to maintain that PA across all category and sub category pages? Is there something fundamentally wrong with our (http://www.ccisolutions.com) site architecture? I understand that we have longer URLs, but this is an old store with a lot of SKUs, so we have decided not to remove the /category/ and /product/ from the URLs because the 301 redirects that would result wouldn't pass all of the authority they've built up over the years. Interested to know viewpoints on the site architecture and how it might be improved. Thanks!
Web Design | | danatanseo0 -
Dynamic pages and code within content
Hi all, I'm considering creating a dynamic table on my site that highlights rows / columns and cells depending on buttons that users can click. Each cell in the table links to a separate page that is created dynamically pulling information from a database. Now I'm aware of the google guidelines: "If you decide to use dynamic pages (i.e., the URL contains a "?" character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them few." So we wondered whether we could put the dynamic pages in our sitemap so that google could index them - the pages can be seen with javascript off which is how the pages are manipulated to make them dynamic. Could anyone give us a overview of the dangers here? I also wondered if you still need to separate content from code on a page? My developer still seems very keen to use inline CSS and javascript! Thanks a bundle.
Web Design | | tgraham0