Hide messenger for crawlers
-
At Magnet.me we are using Intercom to communicate with our users. This means that we are actively adding javascript code which will load the Intercom javascript on each page, and render the button afterwards.
However, this button has no value for crawlers, and slows the page down as the javascript is big and fairly slow. Therefore I considered to ship some code which disables this button, such that performance would improve. To give a ball pack estimate, the buttons javascript is around 3x bigger than the actual entire react application...
Unfortunately this would result in giving users and crawlers slightly different content on the page. I'm unsure about the possible SEO impact:
- Would Google mark the page as faster due to less resources to load?
- Or would it penalize the page for showing slightly different content to users and search engines?
-
In general, I don't think that this is a great idea. Although Google does meter out crawl-allowance, Google also wants a realistic view of the pages which it is crawling. Your attempt at easing the burden of Google's crawl-bots may be seen as an attempt to 'fake' good page-speed metrics, for example (by letting Google load the web-page much faster than end users). This could cause some issues with your rankings if uncovered by a 'dumb' algorithm (which won't factor in your good intentions)
Your efforts may also be unrequired. Although Google 'can' fire and crawl JavaScript generated elements, it doesn't always do so and it doesn't do that for everyone. If you read my (main) response to this question, you'll get a much better idea of what I'm talking about here. As such, the majority of the time - you may be taking on 'potential' risk for no reward
Would it be possible to code things slightly differently? Currently you state that this is your approach:
"This means that we are actively adding javascript code which will load the Intercom javascript on each page, and render the button afterwards"
Could you not add the button through HTML / CSS, and bind a smaller script to the button which then loads the "Intercom javascript"? I am assuming here that the "Intercom javascript" is the large script which is slowing the page(s) down. Why not load that script, only on request (seems logical, but also admit I am no dev - sorry)? It just seems as though more things are being initiated and loaded up-front than are really required
Google want to know which technologies are deployed on your page if they choose to look, they also don't want people going around faking higher page-speed loading scores
If you really want to stop Google wasting time on that script, your basic options would be:
- Code the site to refuse to serve the script to the "googlebot" user agent
- Block the script in robots.txt so that it is never crawled (directive only)
The first option is a little thermonuclear and may mean you get accused of cloaking (unlikely), or at the least 'faking' higher page-speed scores (more likely). The second option is only a directive which Google can disregard, so the risks are lower. The down-side is that Google will pick up on the blocked resource, and may not elevate your page-loading speed. Even if they do, they may say "since we can't view this script or know what it does, we don't know what the implication for end-users is so we'll dampen the rankings a little as a risk assessment factor"
Myself, I would look for an implementation that doesn't slow the site down so much (for users or search-bots). I get that it may be tricky, obviously re-coding the JS from Intercom would probably break the chat entirely. Maybe though, you could think about when that script has to be loaded. Is it really needed, on page-load, all the time for everyone? Or do people only need that functionality, when they choose to interact? How can you slot the loading of the code into that narrow trench, and get the best of both worlds?
Sorry it's not a super simple answer, hope it helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Critical crawler errors...4xx
Hey fam, I ran the Critical Crawler Issues and found 9 pages with critical crawler issues. I'm running a wordpress site and looked in the dashboard for Pages and Posts but the links aren't in the dashboard. Can you help fix? Thank you!
Technical SEO | | Myflgreen0 -
Hide mobile content element
Hello, We are optimizing our mobile (responsive) website at this moment and we want to change some elements on the productpage of our webshop in order of rank. This is the outcome of a data analysis which proves a change in rank of order will result in a higher conversion rate. No doubt. By technical limitations there is no other option but duplicating the element 'product description' in the source of the page to be able to show in on the spot on the product page we like to. Because our webshop is responsive we can not just move the element to the right spot for mobile because the tablet and desktop version then will change as well. My question is: will it be a problem for Google if we hide the original element of the product description on mobile pages by using the bootstrap class "hidden-xs" and duplicate it on another spot in the page to show it with the "visible-xs" class? My concern is that this will create duplicate content and hiding content for Google is not particularly good. On the other hand, I think Google is smart enough to understand that this is not to manipulate visitors or rankings, but this is only for a different look of the mobile website. I hope you guys can give me some good advice.
Technical SEO | | MarcelMoz
Thanks in advance.
Marcel0 -
Why are crawlers not picking up these pages?
Hi there, I've been asked to audit a new subdomain for a travel company. It's all a bit messy, so it's going to take some time to remedy. However, one thing I couldn't understand was the low number of pages appearing in certain crawlers. The subdomain has many pages. A homepage, category pages then product pages. Unfortunately, tools like Screaming Frog and xml-sitemaps.com are only picking up 19 pages and I can't figure out why. Google has so far indexed around 90 pages - this is by no means all of them, but that's probably because of the new domain and lack of sitemap etc. After looking at the crawl results, only the homepage and category (continent pages) are showing. So all the product pages are not. for example, tours.statravel.co.uk/trip/Amsterdam_Kings_Day_(Start_London_end_London)-COCCKDM11 is not appearing in the crawl results. After reviewing the source code, I can't see anything that would prevent this page being crawled. Am I missing something? At the moment, the crawl should be picking up around 400+ product pages, but it's not picking up any. Thanks
Technical SEO | | PeaSoupDigital0 -
Hiding Price html component for all countries except US
Hello everybody, We are planning to have a new website soon, which will be an E-Commerce website for people from the US, and non E-Commerce website for people from other countries. In other words, in the poduct pages, we would like to have the price of the product shown to the users from the US, and on the other hand we would like it to be invisible for users outside of the US. We thought about setting the html elelment of the price to be visible only for US users (by ip). My question is - can Google crawler see this as potential cloacking, since we hide some of the content to some of the users (while google might scan it from US iip address)? Thanks in advance...
Technical SEO | | skifr0 -
Why is it the crawler saying I have 9 Duplicate Page Titles?
Hi, I received my weekly web crawl and it is saying this: | 4 | Duplicate Page Content |
Technical SEO | | afrohairsolutions
| 22 | Missing Meta Description Tag |
| 9 | Duplicate Page Title |
| 1 | Title Element Too Long (> 70 Characters) |
| 1 | Title Element Too Short |
| 1 | 301 (Permanent Redirect) | I'm new to SEO and don't know how to fix this, I don't really see how I have Duplicate Page Content or Duplicate Page Title. This is my website: afrohairsolutions.co.uk Thank you in advance.0 -
Will a google map loaded "on scroll" be ignored by the crawler?
One of my pages has two Google maps on it. This leads to a fairly high keyword density for words like "data", "map data" etc. Since one of the maps is basically at the bottom of the page I thought of loading it "on scroll" as soon as its container becomes visible (before loading the map div should be empty). Will the map then still be craweld by google (can they execute the JS in a way that the map is loaded anyways?) or would this help to reduce the keywords introduced by the maps?
Technical SEO | | ddspg0 -
Hiding Duplicate Content using Javascript
We have e-commerce site selling books. Besides basic information on books, we have content for “About the book” , “Editorial Reviews”, “About the author” etc. But the content in all these section are duplicate and are available on all sites selling similar books. Our question is: 1.Should we worry about the content being duplicate?2.If yes, then will it by a good idea to hide this duplicate content using javascript or iframe?
Technical SEO | | CyrilWilson0 -
CSS Hiding Text - Does this matter to search engine crawlers
Hello, I'm working on a site and a developer is using CSS to mask crawlable links below. Then, java, advanced search links go on top of this. So, if you disable Java, but have CSS enabled, you don't have a lefthand nav. With both CSS and Java disabled you have a fully crawlable website. Is this a red flag? I understand a user without java would have a problem since most people don't disable CSS. But, is this a problem for search crawlers? Thanks!
Technical SEO | | Blenny0