When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Conversion Rate Benchmarking and Optimization
Hey everyone! I run the marketing for an ecommerce marketplace and we're doing good on the SEO front of things, with 60%+ of our traffic coming in via search and we're ranking for some competitive keywords. I'm really curious about the conversion rate though. Despite intensive googling, I've not been able to find a rough benchmark for conversion rates. Our conversion rate is about 1.2%, increasing as we continue to do more optimization work with regards to the site's usability. While we track other forms of conversions, the 1.2% conversion rate is for bookings through the site, which is our main source of revenue. Does anyone have a reference guide for conversion rates, particularly for ecommerce marketplaces? I'd like to know if we're doing ok or really bad, and also to take reference from other similar sites to improve our conversion. Thanks!!
Conversion Rate Optimization | | NgEF0 -
Is my destination url affecting QS?
Hello, I'm running a PPC campaing targeting "Storenvy App" using the ads below and the following DESTINATION LINK: http://www.yotpo.com/landing_page?utm_source=google&utm_medium=cpc&utm_campaign=s-storenvy&redirect=http://lp.yotpo.com/storenvy-app.html As you can see there is a 301 redirect from http://www.yotpo.com/landing_page?utm_source=google&utm_medium=cpc&utm_campaign=s-storenvy TO http://lp.yotpo.com/storenvy-app.html How this will affect QS? Thanks in advance for all your answers! AdDzQi0.jpg
Conversion Rate Optimization | | JustinButlion0 -
How Would You Market A Photo Of Jay-Z Wearing Your Client's Apparel?
If Jay-Z was spotted wearing your client's sweatshirt, how would you maximize the marketing potential of this photo and opportunity to sell more sweatshirts?
Conversion Rate Optimization | | farmiloe0 -
Any tips and tricks to increase Conversion Rates on Google PLA?
I have recently organized my Adwords (Product Listing Ads) campaign according to this link: http://certifiedknowledge.org/blog/how-to-create-a-state-of-the-art-pla-campaign While I like the organization and control this provides, my fundamental concern is to increase conversions. I understand conversions are dependent on other factors such as landing pages etc. Are there any Google Adwords tips someone has had success with to increase CTR and increase conversion rates? Pating $10/day to Google Adwords while only getting a sporadic business every 3 days is getting a bit frustrating. Are there any tools in MOZ that can help find faults and generate simple reports on the Google Adwords campaign? Any help is much appreciated.
Conversion Rate Optimization | | amitramani1 -
How to find average organic traffic growth rate for ecommerce startup?
Happy Friday Everyone, I have a quick question: I am in the process of crunching some numbers to try and figure out an organic traffic forecast for a client. One of the crucial metrics I can't seem to figure out is traffic rate of growth over the next year or so. The reason this is complicated for me is that I have only ever done this sort of thing for an established business before, and had plenty of past organic traffic data to work with. What I am looking for is a source that could give me a reasonable idea of what kind of growth rates I might expect for a startup with practically no data; I would be perfectly happy with a national average for online retail startups. Any help would be greatly appreciated. Thanks!
Conversion Rate Optimization | | G2W0 -
Tracking Adwords Conversions for eCommerce ROI
I'm working with a rather large eCommerce customer who needs to be able to track their Adwords conversions for the purpose of determining the ROI of their Adwords campaigns. I've installed code from Google Adwords Conversions on their eCommerce page, but it doesn't seem to be calculating values correctly. Also, where would setting goals in Analytics come in to play if I'm trying to track the conversions of specific Adwords conversions?
Conversion Rate Optimization | | SmokewagonKen0 -
Has anyone heard about Facebook's Revenue Problems? Duh right. Well have you seen how they are losing money yet?!?!
So about a year ago now (maybe more/less), Facebook changed the way their ads are displayed. On the desktop version of Facebook (for me at least), the Sponsored ads in the right sidebar display anywhere between 3 and 7 ads at a time. Well in the case of 7 ads, that is a lot of ads to look at and read through. All seems pretty ok right?? Well, here is the problem. I cannot tell you how many times I have seen an ad that interests me, and then read through the others below them, then before I can go back to click on the one that interested me, the ads automatically refresh, and I don't get to click on the ad. Maybe they figure that by refreshing ads, it gets them more chances to catch your interest, and that the value of that is higher than the value of me getting to click what I wanted. (It kinda upset me honestly. I thumbs down things that tease me.) Either way, they could have both. My solution? A small button that you can click to view the ads that were previously displayed. (imagine how much money that could bring them. Especially if I saw the ad yesterday, left the window open for 24 hours with the computer on, and just sat there clicking until i found it. You are BOUND to find MORE ads that interest you, and click on those too.) ((ok, ok, ok, maybe 24 hours is a little ridiculous, but at least SOME amount of time and a button is necessary)) If we were really going to build a marketing playground, then we could also throw in a "More Like This" button. It would open the doors to some AMAZING data. Full to the brim with advertisement related feedback coming happily from your users. When you are talking about positioning your brand, it doesn't get much more important than to talk to them while they are enjoying themselves. (i.e. they click button, and next page has 10 links. you show the links, based on how many times people are clicking them. Not to mention, when you are that big, you can look at the value of what they do immediately after, the next day, and even the next week, and really tell what is the most valueable add, in terms of egagement, and further CTRs and Conversions for later in your relationship ) Anyways, I thought everyone would find that interesting, and hopefully it would inspire some people to get analytical with their thoughts today. Do you think that Facebook could use some conversion rate optimization? Do you think that Facebook could be doing more than they are to make more money? Do you think that Facebook (being such a powerful site), could make a lot of money by implementing an SEO/Content strategy?
Conversion Rate Optimization | | TylerAbernethy0 -
Top 5 Conversion Rate Tips for a "Beds Shop E-commerce"
Which are in your opinion the "Top 5 things to do" to increase the conversion rate in a beds e-commerce site? The conversion is "to sell a bed" 🙂 The website is http://www.idealetti.com Thank you in advance to everyone that share his experience or give some advice!
Conversion Rate Optimization | | YESdesign0