Split test content experiment - Why can't GA identify a winner here?
-
I have been running a content experiment for a short while now and GA has just ended it saying it cannot determine a winner.
Looking at the images (links below), without any form of analysis I can already see a pattern of greater success in Variation 1. It ended with a 93% probability of outperforming the original yet the content experiment ended with no winner. Does this mean the 95% confidence threshold I set should've been lowered?
Ultimately I'm going to choose this as my winner but why didn't GA push it as the winner? Is there something I am missing?
Image 1 - Showing e-commerce performance (objective of split test was transactions)
Image 2 - Showing conversions (same split test, same objective, just different report)
Your thoughts and comments will be appreciated.
-
Hey Emeka,
Short Answer:
You're correct. Effectively what Google is saying here is that they don't have enough statistical confidence to definitely tell you that the variation is outperforming the original at a 95% confidence level, but they do at a 93.8% confidence level.
Quick Note: 95% is the lowest setting in GA Experiments.Long Answer:
The math behind this statistical significance calculation is:
Full credit to vwo.com for their A/B Testing Significance Calculator & doing all the work here.
Link to Image One - This is simply the data of the Control vs Variation & the Conversion Rate & Standard Error Rates
- Conversion Rate is: Conversions/Sessions.
- Standard Error is: √(Conversion Rate*(1-Conversion Rate)/Visitors)
Link to Image Two - Confidence Levels, Z-score, & P-value
To find if something is truly significant at a specific confidence level, we need to calculate the Z-score then use that value to find the P-value and from there we can determine the confidence level.- Z-score is: (Control Conversion Rate-Variation Conversion Rate)/√((Control Standard Error^2)+(Variation Standard Error^2))
- For the P-value, we need to calculate the normal distribution of the z-score with a mean of 0 and a standard deviation of 1. The easiest way to do this is to use an online tool, here's a link to your specific example.
Finally, we take the Confidence % expressed as a decimal (i.e. 0.90, 0.95, 0.99) and 1 minus these values (i.e. 0.1, 0.05, 0.01)
If the P-value is greater than the Confidence % or less than 1 minus the Confidence %, then it is significant, otherwise it is not. Let me explain that using our example:
at 95% confidence, our P-value needs to be <0.05 or >0.95. Since our P-value is .05799, it doesn't fit either of those requirements and such is not significant at that confidence level.I know that's a lot of math, but this is why Google Experiments is saying that the result is not statistically significant.
Hope this helps! Let me know if you have any further questions on this!
Trenton
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to fix canonicalization so Google shows our company's website and not LinkedIn
I work for a company that builds and manages apartments. I would like to know how we can fix canonicalization. You see, we have been posting our company's blog articles not only on our company's website, but also on LinkedIn. When people search for one of our articles, Google is displaying the article for LinkedIn. Except, that's not what we want. I'd prefer for Google to display our company's website, to drive more traffic to our website and not LinkedIn. For instance I just Googled the title to one of our company's blog article "Great Leadership is about Influence not Authority." The first thing to show up was a link to see the article on LinkedIn. There was nothing referring to our company's website. Here is a link to the LinkedIn account where we have been posting the blogs: https://www.linkedin.com/pulse/great-leadership-influence-authority-mike-kaeding/ Here is a link to the same article, but on our company's website: https://www.norhart.com/blog/2018/08/08/great-leadership-is-about-influence-not-authority/ Why is Google showing the article for Linkedin and not our Company's website? Might LinkedIn have canonicalization over Google? If so, how might we be able to reverse this and drive people to our company's blog instead. Any ideas I'd greatly appreciate it! Thanks.
Conversion Rate Optimization | | mikleing0 -
Please can anyone recommend any GOOD 'TRAFFIC BUYING COMPANIES ?
Hi Guys I really need some help here I am looking for some good traffic buying companies where we can buy traffic from that can will help us to generate leads from our own website. We want to 'buy traffic' that is targeted and generate leads from our website. We are running Google ad words but it is now becoming more and more expensive to run! Please can anyone recommend any good companies ? Thanks Gareth
Conversion Rate Optimization | | GAZ090 -
GA expert for publishers
Hi. I'm looking for a GA expert who has experience working with magazine publishers. Anyone have a lead?
Conversion Rate Optimization | | RicardoSalcedo0 -
How I can exclude "Events" & "Traffic" from a specific IP?
I would like to know how I can exclude the events triggered from a specific IP for example my office' IP address. I've already added a filter to exclude my office's IP address, but the "events" are still showing in the new view. TIA!
Conversion Rate Optimization | | zpm20140 -
Any PPC Companies out there who can work with small accounts
I have a story , 18 months ago we had our own PPC adwords account, we had been running at £3.20 CPA for quite some time and we were very happy. Sometimes CPA would go up , sometimes down, but not my very much. One unfortunate day google rang, and told us , that our CPA should ne £2.50 and that we needed expert advice that they couldn't actually give us, we needed an expert. To cut a long story short and after 5 very expert companies have messed with our account, we had varying results. One company left us with 2 leads per day at £10 a lead !!!! , one did better than that and setup a £3k adwords account using their expert advice, blew it all and said "times are tough" sorry !!! The last company was actually the best, great guy , but I think we are two small for him, we have varying results, one day we get 30 leads, the next we get 10 , one day its £3.50 CPA , next its £7.00 . But on the whole very happy with his initial work We just want a company that doesn't Lie to us , tells us straight , and can use their expertise to get us a CPA of £3.50 and help us get upto 50 leads a day, if not more. Thanks Hope that makes sense Nick
Conversion Rate Optimization | | NickKer
systems direct ltd0 -
Does the word next to domain gives more relevancy to the page of it's URL?
Whether putting a (category or brand) word next to domain is better than on the end of an URL? (i.e. domain.com/sony/tvs or domain.com/tv/sony) Which one would get higher result on SERP "tv" or "sony" in both cases? Or maybe they both serves the same?
Conversion Rate Optimization | | komeksimas0 -
Can I track user journeys from a particular source?
We want to track users from a particular campaign to see what they do when they get to the site. The site generates revenue from clicks out so we want to be able to see whether visitors from a particular source click on multiple deals. i.e. enters from the source and then clicks on 3 different deals that take the user to other sites. Using GA, can I track this by implementing UTM on both the campaign links and outgoing links? Can I see exactly how many deals the average user is clicking on? As far as I know I can: Track visitors from the campaign Track overall clicks on deals Divide click outs by visitors But I can't track how many deals the visitors from campaign A clicked on... or can I?
Conversion Rate Optimization | | Red_Mud_Rookie0 -
Can Changing Meta Descriptions Negatively Impact SERP's?
I have just had a page start ranking well in key SERP's and I would like to change the meta description and add a price as we are extremely competative in that line. Could changing the meta description now a page is ranking negatively impact the SERP placing? Does anyone have any experience with this?
Conversion Rate Optimization | | robertrRSwalters0