When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We've a client that wants to repurpose some of the content on there site but not lose rankings. However its changing the content to an alternative to ....
The Current URL Example would be test.co.uk/prodcut-catogory/product-1 . This side of the business is being sold however they have products elsewhere in the range that can be used as alternatives to the product range that will be moved, so they want to amend these pages with an " alternative " slant. Example idea would be to change to the following : test.co.uk/prodcut-catogory/alternative-to-product-1, trying not to change the URL to much, keeping the majority of the content the same but adding new copy stating the benefits of using the alternative product. So it becomes a comparison between the two. I appreciate I will have to amend the URLs, titles, meta, H1s, add new copy & preform a 301's. Option 2 Simply write a second page which is all brand new content talking about it being an alternative to the products and doing a 301 from the original page but not amending that. Be great to get peoples feedback on this approach and if there is anything else that I could do to minimise the drop in rankings once I complete the changes. Thank you D
Conversion Rate Optimization | | SDOwner0 -
Tracking Facebook Mobile Visits to Desktop Conversions?
Hi. If I set an ad in Facebook, I can obviously track visits, conversions and so on from that ad and see if they where mobile or desktop. However, how do I, if possible track a mobile visitor who the converts on a desktop? Eg. I go on XXX Facebook Ad on my mobile whilst in work, look around. Like what I see. Then I go home from work, open my laptopn and go on XXX website and send an email enquiry form in. Is this possible to track? Thanks
Conversion Rate Optimization | | YNWA0 -
Facebook Connect login preventing traffic source conversion data. Any solutions?
We're buying traffic to the website via Adwords, and when they hit the website and go to sign up for an application (enter into our conversion funnel), the session concludes as a bounced session, a new session starts and that session now has a referral from Facebook rather than the original referring source. This means we can't monitor adwords conversions properly (or any other traffic source conversions) this seems to be quite a common problem with Facebook/Google account logins, in the case of Google, people block account.google.com but we can't afford to block Facebook outright as we're wanting to see other Facebook referrals. Has anyone come across this problem? Any suggestions of how to fix this?
Conversion Rate Optimization | | Startupfactory0 -
Google Analytics: Do 'Goals' actually work?
Hi, Hoping someone can clarify this for me. I've set a goal up whereby the rule is to pick up all pages with "Thank-you" in the destination URL (regular expression). So, when someone fills out a form, it will count as a lead. I'm about 8 weeks in to the campaign now and measuring the last 4 weeks I've had a total of 30 goals completed. I have two issues with this and I really hope someone can help clarify: Using a custom report I can see that I've actually had 52 Visits to the Thank-You pages, 49 Unique Visits and 60 pageviews. Where is Google getting the goals from? I have 51 leads in my inbox from the campaign so far. So, much closer to the above stats in the custom report than in the Goal report. Why is this? Thanking you in advance!
Conversion Rate Optimization | | IndustrySteve0 -
Suggestions for Conversion Rate Optimization low cost companies?
Does anyone have a list of suggestions for lower cost companies to do conversions. I am a new business and just wanted my "Cart" check out with maybe some suggestions for improvement and a little testing to see if it is done correctly. Thank you, Boo
Conversion Rate Optimization | | Boodreaux0 -
How to improve my conversions
Hey all, As always thanks in advance to any help given. So I think we haven't done a bad job, but I know there is always more that can be done with conversions. We have a website which provides guestlist to clubs in London, we are competitive on our keywords and ppc and get the clicks through with very very relevant adverts and meta descriptions.... The questions is 'How do I improve our conversion rate on page?' The said pages are setup like the examples below: www.capitalalist.com/movida-guestlist/ www.capitalalist.com/chinawhite-guestlist/ www.capitalalist.com/mahiki-guestlist/ The conversion comes when the click 'join guestlist' and fill out the form. Any help with be GREATLY appreciated!
Conversion Rate Optimization | | AdenBrands1 -
Any idea why my bounce rate could be so high?
Hello Everyone, I'm stumped. My primary website has a horrendous bounce rate of around 75% My domain is www.radicalmarketingsolutions.com My home page has a bounce rate of around 68%. The funny thing is, I have a page on Google Analytics basics for small business owners that is my 3rd top page for content, I have an average time on page of over 3 minutes yet my bounce rate is 90%. I have another page on how to depersonalize your google searches in Chrome and Firefox that is a top page with a time on page over 5 minutes and this page has a bounce rate of 96%! I felt that my old design was really hurting me so I changed the design in the beginning of this month and it does not look like it's helped much. I really simplified the site, created a lot more white space and added some links to my other important pages in hopes to get more click throughs. I would really love some honest "constructive" feed back on my site as to why the bounce rate could be so high. I have a ton of content on my blog and I get a lot of positive feedback. One thing I did not do was exclude my IP from Analytics so I could be influencing the numbers. I've added this filter and hoping this will help. Another issue is I do email my list with each new blog post. This could mean those folks visit the post and then leave without reading anything else on my blog. So this could be hurting me as well. Would love your suggestions. Thanks, Bill Parlaman
Conversion Rate Optimization | | wparlaman0 -
Lookiing for examples of local "small biz" sites with phone or contact form conversion points
Hi Guys, I'm looking for any good examples of small business sites that you've come across that do a good job at converting potential visitors to clients. I'm primarily thinking about service oriented business such as Doctors, Dentists, Salons, Clothing stores, small restuarants, car detailing services, local photographers, etc. I'm just trying to digest other examples and look for new inspirations.
Conversion Rate Optimization | | DotCar0