When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Split test content experiment - Why can't GA identify a winner here?
I have been running a content experiment for a short while now and GA has just ended it saying it cannot determine a winner. Looking at the images (links below), without any form of analysis I can already see a pattern of greater success in Variation 1. It ended with a 93% probability of outperforming the original yet the content experiment ended with no winner. Does this mean the 95% confidence threshold I set should've been lowered? Ultimately I'm going to choose this as my winner but why didn't GA push it as the winner? Is there something I am missing? Image 1 - Showing e-commerce performance (objective of split test was transactions) Image 2 - Showing conversions (same split test, same objective, just different report) Your thoughts and comments will be appreciated.
Conversion Rate Optimization | | OptiBacUK0 -
How to track conversions?
Hi Mozers, Anyone know how I can go about finding what my conversion is on my site? Or how to go about setting up tracking?
Conversion Rate Optimization | | edward-may0 -
Landing page experts, for lead gen conversion
If I am wanting to hire a conversion expert, especially for PPC landing pages in the insurance space. Do you think that the expert should be specific to the insurance space or just to landing pages to get the maximize conversion optimization?
Conversion Rate Optimization | | jdcline0 -
Adwords conversion data is contradicting
I am looking at my Adwords data from a segment of last year. It shows my total for all ad groups in this campaign at 18. I only have one ad and it shows 7 conversions. Where are the other 11 coming from?
Conversion Rate Optimization | | EcommerceSite0 -
Does adding a well scripted and produced video help significantly in conversions from visits to my divorce lawyer website to calls for my services?
Should the addition of well scripted and produced video to my divorce lawyer website increase my conversions and, if so, to what extent?
Conversion Rate Optimization | | hildebrandlaw0 -
Has anyone heard about Facebook's Revenue Problems? Duh right. Well have you seen how they are losing money yet?!?!
So about a year ago now (maybe more/less), Facebook changed the way their ads are displayed. On the desktop version of Facebook (for me at least), the Sponsored ads in the right sidebar display anywhere between 3 and 7 ads at a time. Well in the case of 7 ads, that is a lot of ads to look at and read through. All seems pretty ok right?? Well, here is the problem. I cannot tell you how many times I have seen an ad that interests me, and then read through the others below them, then before I can go back to click on the one that interested me, the ads automatically refresh, and I don't get to click on the ad. Maybe they figure that by refreshing ads, it gets them more chances to catch your interest, and that the value of that is higher than the value of me getting to click what I wanted. (It kinda upset me honestly. I thumbs down things that tease me.) Either way, they could have both. My solution? A small button that you can click to view the ads that were previously displayed. (imagine how much money that could bring them. Especially if I saw the ad yesterday, left the window open for 24 hours with the computer on, and just sat there clicking until i found it. You are BOUND to find MORE ads that interest you, and click on those too.) ((ok, ok, ok, maybe 24 hours is a little ridiculous, but at least SOME amount of time and a button is necessary)) If we were really going to build a marketing playground, then we could also throw in a "More Like This" button. It would open the doors to some AMAZING data. Full to the brim with advertisement related feedback coming happily from your users. When you are talking about positioning your brand, it doesn't get much more important than to talk to them while they are enjoying themselves. (i.e. they click button, and next page has 10 links. you show the links, based on how many times people are clicking them. Not to mention, when you are that big, you can look at the value of what they do immediately after, the next day, and even the next week, and really tell what is the most valueable add, in terms of egagement, and further CTRs and Conversions for later in your relationship ) Anyways, I thought everyone would find that interesting, and hopefully it would inspire some people to get analytical with their thoughts today. Do you think that Facebook could use some conversion rate optimization? Do you think that Facebook could be doing more than they are to make more money? Do you think that Facebook (being such a powerful site), could make a lot of money by implementing an SEO/Content strategy?
Conversion Rate Optimization | | TylerAbernethy0 -
Does the word next to domain gives more relevancy to the page of it's URL?
Whether putting a (category or brand) word next to domain is better than on the end of an URL? (i.e. domain.com/sony/tvs or domain.com/tv/sony) Which one would get higher result on SERP "tv" or "sony" in both cases? Or maybe they both serves the same?
Conversion Rate Optimization | | komeksimas0 -
Conversion Rate Optimization (CRO): Best Approach/Plan Documentation
Hey Mozzers - First off, I am well aware of what CRO is, its benefits, and why it is essential for any inbound or performance marketing campaigns. However i am a total newbie at the actual planning and implementation of CRO, so any advice/feedback on the questions below are much appreciated! When planning a CRO experiment/test what is the best way to document it? Should i develop the full plan ahead of time laying out each element? If so, what elements should i include in this plan? What are the KPIs i should report on? What tools should i use for reporting (i was thinking a blend of Google and Clicktale)? Any lessons learned through out your campaigns? Thank you all so much for your input! Regards - Kyle
Conversion Rate Optimization | | kchandler0