Googlebot soon to be executing javascript - Should I change my robots.txt?
-
This question came to mind as I was pursuing an unrelated issue and reviewing a site's robots/txt file.
Currently this is a line item in the file:
Disallow: https://* According to a recent post in the Google Webmasters Central Blog: [http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better") Googlebot is getting much closer to being able to properly render javascript. Pardon some ignorance on my part because I am not a developer, but wouldn't this require Googlebot be able to execute javascript? If so, I am concerned that disallowing Googlebot from the https:// versions of our pages could interfere with crawling and indexation because as soon as an end-user clicks the "checkout" button on our view cart page, everything on the site flips to https:// - If this were disallowed then would Googlebot stop crawling at that point and simply leave because all pages were now https:// ??? Or am I just waaayyyy over thinking it?...wouldn't be the first time! Thanks all! [](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better")
-
Excellent answer. Thanks so much Doug. I really appreciate it! Adding a "nofollow" attribute to the Checkout button is a good suggestion and should be fairly easy to implement. I realize that internal nofollows are not normally recommended, but in this instance, may not be a bad idea.
-
Hi Dana,
When you click on the checkout button - what's the mechanism for taking people to the https:// site. Is it just that the checkout link uses https:// in it's link? Is there some javascript wizardry you're particularly concerned about?
Even though googlebot follows this one link to the https version of the cart, it will still have all the other links on the previous page queued up to follow (non-https) so I don't think this will stop the crawl at that point. It would be a nightmare if googlebot stopped crawling hte entire site everytime it went down a rabbit hole!
That's not to say that you wouldn't want to consider no-following your checkout button. I'm sure neither you, nor google want to the innards of the cart pages to be indexed? There's probably other pages you'd rather Googlebot spent it's time finding right?
My take on the Google blog about understanding Javascript is that the aim is to try and do a better job discovering content that might be hidden by Javascript/Ajax. It's a problem for google when the raw html that they're crawling doesn't accurately reflect the content that is displayed in front of a real visitor.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Have you ever changed the logo anchor text from "logo" to "keyword"? How Google considers?
Hi all, We know that generally logo with the website homepage link is the first link crawled by Google and other search engines. Can we change the anchor text from "logo" to "keyword"? Have any one tried or seen others doing? How Google considers it? Thanks
Algorithm Updates | | vtmoz1 -
Does personalization that changes meta data display in SERPs impact SEO?
My company has been rolling out personalization at the page level across our site using behavior paths embedding content from cross pathed pages as well as customer journey mapping. The dynamically generated content doesn’t change the URLs. In the SERPs I’m seeing that our title tags and meta descriptions also seem to be dynamically generated even though we have these elements crafted. The way our elements are crafted: Title tag: descriptive Keyword rich phrase | Brand Meta description: Keyword rich, grammatically correct description tied to title tag and page content for consistency. I search a specific URL: Title tag display: Keyword rich phrase | Brand – Brand Meta description display: Random content pulled from the page I search a phrase that includes Brand + keywords in the URL: Title tag display: Title tag we crafted Meta description display: Meta description we crafted I search a phrase that includes Brand + keywords in the title tag: Title tag display: Title tag we crafted Meta description display: Random content pulled from the page Does Google crawl the page and digest the title tag and meta description we crafted? Or is Google going to ding us for having the brand twice, exceeding the length of the title tag, etc.? I have been searching the interwebs, forums and the cosmos, but the only information I’m finding is related to the fact that URLs are changing and how that would impact SEO. That’s not the case for us. Thoughts on how all this is impacting our SEO efforts?
Algorithm Updates | | NStarJM0 -
Help Me Change My Client's Mind
My client wants to build a second site to provide targeted links for SEO to his main site. He's interested in buying a TLD with some near topic authority/links and then build the second site's authority up from there. He is clear that this could get him in trouble for a link scheme, but thinks it can all be hidden from Google. Off the top of my head I was able to recall a few of the pain-in-the-neck things you'd have to do to not get caught, but he seemed unconvinced. I recall you'd have to have: Different registrar Different contact/WhoIs Different site host Different G/A, GWT Logging into second's site's G/A, GWT with different IP address not used for main domain With the exception of the last one, he didn't seem to think it would be too hard. Aren't there more difficult maneuvers required for hiding this from Google? I want to be able to point out to him how ridiculous this low integrity effort will be, without losing the client. Thanks! Best... Darcy
Algorithm Updates | | 945010 -
Question about Google Algo Change on June 26
I have a client who's Google Organic visits dropped significantly on June 26th. I used a chart overlay called ChartIntelligence. It says that there was an SEOF update on 6/26/2013. Does anyone know what this update (or any other updates) would be? Also, where might I find additional info on this update. I did notice that Moz's algo change tracker listed a multi-week update on June 27, but I'm not sure where to find info on what types of things were impacted by this update. Any info would be helpful.
Algorithm Updates | | TopFloor0 -
50% of my keywords just had a significant drop for no reason no changes?
I just looked at a report for one of my sites in moz pro and Improvement: 35 slots 35/75 keywords just dropped on google, I didn't make any changes recently Ranking Changes Improvement: 12 slots 12 Improved Improvement: 35 slots 35 Declined Many dropping 5 - 22 places! Any ideas on whats up, thank fully it didn't effect traffic, its pretty low anyways, but it has stayed the same... But still dropped to page 5 for lots of stuff. Strange. Did google just update or something? Thanks for any ideas.
Algorithm Updates | | syndicate0 -
Is changing your meta titles frequently good SEO Practice
Greetings, Im a new SEO and really knew nothing until signing up to SEOMoz. After reading the SEO101 and gathering as much information in a short period of time things started to become a little clearer. So I started my first campaign used my new SEO knowledge and input all of my meta information. Then I waited a few days to see what happened with my search result. We had never ranked for a single keyword before mind you. So a couple/few of days go by and I started punching in my keywords and looking through the pages. There I was page three. I was SO happy. I read the entire SEO101 again, realized a little more about what I had to do. So I started changing everything up, adding pictures, I found out what a IMG ALT Attributes were in the HTML editor, bolded text and all the other things I missed the first time around. Three days go by and I move up again. I start to notice my traffic is increasing and I am actually getting organic hits through search traffic. This has never happened before. I am over the moon. But I realize that I have my main focus keyword as the second key word in my title tag. So I switch the two words around, wait a few more days. Here's why I ask my question. The original title tag was still showing up and I was on the first page for both keywords, and I could see both title tags when searching for either keyword. So; Is changing your meta titles frequently good SEO Practice ? Warmest regards, Michael S&M Warning: adult site, NSFW
Algorithm Updates | | Sexandmetal0 -
Did google change their algorithm over the past week?
I did some home page optimization with the seo moz on page key word optimization tool and we are now back in the top three in the past week (after dropping to page 3 a month or so ago). It seems that google has gone back to combining google places with organic searches. Has anyone else noticed this type of change? I did read some posts about panda 2.2, which seems to explain some of these findings. I am wondering if things are in flux or they may be more stable this way? Thanks for the insights.
Algorithm Updates | | fertilityhealth0 -
Recent changes to suggested search algorithm?
Our company recently had a "company name + scam" listing as #2 in suggested search, and yesterday, it miraculously disappeared. Has anyone else noticed similar changes in suggested search results? I hope it stick, I'm just trying to understand exactly what caused that 1 listing to vanish.
Algorithm Updates | | CareerBliss0