Split test content experiment - Why can't GA identify a winner here?
-
I have been running a content experiment for a short while now and GA has just ended it saying it cannot determine a winner.
Looking at the images (links below), without any form of analysis I can already see a pattern of greater success in Variation 1. It ended with a 93% probability of outperforming the original yet the content experiment ended with no winner. Does this mean the 95% confidence threshold I set should've been lowered?
Ultimately I'm going to choose this as my winner but why didn't GA push it as the winner? Is there something I am missing?
Image 1 - Showing e-commerce performance (objective of split test was transactions)
Image 2 - Showing conversions (same split test, same objective, just different report)
Your thoughts and comments will be appreciated.
-
Hey Emeka,
Short Answer:
You're correct. Effectively what Google is saying here is that they don't have enough statistical confidence to definitely tell you that the variation is outperforming the original at a 95% confidence level, but they do at a 93.8% confidence level.
Quick Note: 95% is the lowest setting in GA Experiments.Long Answer:
The math behind this statistical significance calculation is:
Full credit to vwo.com for their A/B Testing Significance Calculator & doing all the work here.
Link to Image One - This is simply the data of the Control vs Variation & the Conversion Rate & Standard Error Rates
- Conversion Rate is: Conversions/Sessions.
- Standard Error is: √(Conversion Rate*(1-Conversion Rate)/Visitors)
Link to Image Two - Confidence Levels, Z-score, & P-value
To find if something is truly significant at a specific confidence level, we need to calculate the Z-score then use that value to find the P-value and from there we can determine the confidence level.- Z-score is: (Control Conversion Rate-Variation Conversion Rate)/√((Control Standard Error^2)+(Variation Standard Error^2))
- For the P-value, we need to calculate the normal distribution of the z-score with a mean of 0 and a standard deviation of 1. The easiest way to do this is to use an online tool, here's a link to your specific example.
Finally, we take the Confidence % expressed as a decimal (i.e. 0.90, 0.95, 0.99) and 1 minus these values (i.e. 0.1, 0.05, 0.01)
If the P-value is greater than the Confidence % or less than 1 minus the Confidence %, then it is significant, otherwise it is not. Let me explain that using our example:
at 95% confidence, our P-value needs to be <0.05 or >0.95. Since our P-value is .05799, it doesn't fit either of those requirements and such is not significant at that confidence level.I know that's a lot of math, but this is why Google Experiments is saying that the result is not statistically significant.
Hope this helps! Let me know if you have any further questions on this!
Trenton
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to fix canonicalization so Google shows our company's website and not LinkedIn
I work for a company that builds and manages apartments. I would like to know how we can fix canonicalization. You see, we have been posting our company's blog articles not only on our company's website, but also on LinkedIn. When people search for one of our articles, Google is displaying the article for LinkedIn. Except, that's not what we want. I'd prefer for Google to display our company's website, to drive more traffic to our website and not LinkedIn. For instance I just Googled the title to one of our company's blog article "Great Leadership is about Influence not Authority." The first thing to show up was a link to see the article on LinkedIn. There was nothing referring to our company's website. Here is a link to the LinkedIn account where we have been posting the blogs: https://www.linkedin.com/pulse/great-leadership-influence-authority-mike-kaeding/ Here is a link to the same article, but on our company's website: https://www.norhart.com/blog/2018/08/08/great-leadership-is-about-influence-not-authority/ Why is Google showing the article for Linkedin and not our Company's website? Might LinkedIn have canonicalization over Google? If so, how might we be able to reverse this and drive people to our company's blog instead. Any ideas I'd greatly appreciate it! Thanks.
Conversion Rate Optimization | | mikleing0 -
Please can anyone recommend any GOOD 'TRAFFIC BUYING COMPANIES ?
Hi Guys I really need some help here I am looking for some good traffic buying companies where we can buy traffic from that can will help us to generate leads from our own website. We want to 'buy traffic' that is targeted and generate leads from our website. We are running Google ad words but it is now becoming more and more expensive to run! Please can anyone recommend any good companies ? Thanks Gareth
Conversion Rate Optimization | | GAZ090 -
What's the most effective web marketing tactic you've seen or used that very few people know about?
I wanted to start a thread to share some of the really cool marketing tactics I've seen on the web that I think few folks are using, AND ask the community here what you've seen, too! Some of my favorite undiscovered or less-used tactics include: Making smart use of bios for conferences, events, interviews, etc. where folks ask you or your team members for a "bio" and you get to control the links, link targets, and anchor text. This is super powerful in my experience, so long as you have a moderately strong profile or regular participation in this type of stuff. Price anchoring on conversion pages, e.g. http://www.trackur.com/options - note how they start with the highest price to help "anchor" the audience to bigger numbers. A great principle of psychology in action. Using re-marketing to draw people to content rather than just purchase/conversion pages. The effectiveness of these is, I've heard, dramatically higher than the usual re-marketing campaigns that take you to a squeeze or purchase page. I can't share the example I'm thinking of, unfortunately, but I'd urge you to try it! Get more social shares and clicks by SHARING MORE THAN ONCE! A lot of folks feel like they are burdening their audience on Twitter/Facebook/G+ or frustrating them if they post multiple times, when in fact, very, very few of your followers are online at any given time. I've tested this myself and I get almost no negative feedback but can triple or more the number of shares/+1s/likes/visits/etc I get just by sharing 2-3X! The key is not to be too repetitive or annoying, and to acknowledge past shares (at least for me). e.g. I'll say "my blog post from last night on XYZ" and get a ton more clicks. What are your favorites? Please share!
Conversion Rate Optimization | | randfish16 -
How can I track the traffic source/keyword of form submissions?
I'm using Google Analytics on www.nhfinehomes.com and would like to track the source of leads submitted via form submissions. Ideally, I could track if the lead originated from organic search (and what keyword phrase), PPC, referral links, etc. Is this possible with Google Analytics and if so, can anyone point to some documentation on how to do this or better yet someone who can help me set this up? This site is running ASP.NET and uses JQuery so there is not a 'traditional' thank you page that I can reference. I'm not a programmer and do not pretend to be one! Thanks in advance. Link
Conversion Rate Optimization | | LinkMoser0 -
How Do I Create a Google Analytics Dashboard for My Designer To Monitor Landing Page A/B Testing
Hello All, We recently started doing some "AdWords Experiments" using A/B testing of our search landing pages. My web design team does not have access to our AdWords account, but they do have "user" access to our Google Analytics account. What I need to figure out how to do is setup an easy dashboard (or custom report) that will show them at a quick glance how the two versions of their page are performing in terms of: Goal Completions (Conversions) where the specific page is the entrance/landing page. Bounce Rate Time spent on site where the specific page is the entrance/landing page. Pages viewed where the specific page is the entrance/landing page. Possibly a way to see the most popular page visited 'next' after starting on the specific entrance/landing page Anything else that might be useful The two URLs would be like: http://www.domain.com/search/testa/
Conversion Rate Optimization | | Robert-B
http://www.domain.com/search/testb/ Any insight about the best way to do this is greatly appreciated! Cheers!0 -
SEO For Personalized Content Pages
Hi, We are building a site that has content that personalizes based each visitor. This is being done to serve the most relevant content for conversion purposes and NOT for SEO purposes. So, there are many diferent versions of each page based on the visitor's location, device, browser, keyword-used, traffic source, etc. Obviously, we don't want this to appear like cloaking (it is genuine content personalization) and wondered how Google would view this? What is the safeguard for protecting the website from potential penalties? Any help would be greatly appreciated! Thanks in advance.
Conversion Rate Optimization | | Clickthroo0 -
Which A/B test software do you prefer? Howcome?
I am about to start some research on software for experimental design on websites. So I'd really like to know which A/B test software do you prefer? Howcome?
Conversion Rate Optimization | | ThomasHgenhaven1 -
Using overlay content on landing pages - possible? recommended?
We want to use an overlay, like a modal dialog, in a landing page to show more information to support conversion without linking to another page or using popups. Anyone have any stats on success or risks? We've considered simple roll-overs to provide some more info, but everyone has raised alarms that we may be introducing a potential technical or usability hurdle for users. The overlay would display when someone rolled over or clicked a "Learn More" link on the page to provide additional offer reinforcement points to drive the visitors from consideration to conversion. Aside from our own testing, we don't have any sources to cite as to whether this approach can or will impact or aid conversion. Any insights would be appreciated.
Conversion Rate Optimization | | lh27121