When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Facebook conversions - issue in calculation
Hi, I have a few campaigns for conversions on Facebook, but any of them don't show to me real numbers. There are 4 times more conversions on Facebook than in GA. I did a little research. The problem is that - Facebook pixel calculates all possible conversions from all sources, and because my campaign results are wrong e.g.: Facebook campaign results: Conversions: 10 | Cost per conversion: 1 GA for that campaign (real numbers): Conversions 1 | Cost per conversion: 1 GA from all campaigns and sources: 10 It's that correct? Pls help. Br. Marco
Conversion Rate Optimization | | marcop21 -
Should we change branded keywords based on audience knowledge?
Good Morning Moz, Wanted to pose a question to the community that our company has been wrestling with internally for a while now. I think your insights would be tremendously valuable. Our company effectively has two business units, one that is online and one that is offline. Each appeals to a different customer base as well. In one business unit, the larger, more offline and better established of the two, we refer to ourselves as a "manufacturer of high end automobiles" (not true, but using as an example). We've carried this through to the other business unit which is very small, web-based, and growing, but have quickly realized that the majority of would-be customers do not search for "automobiles", they search for "cars." And the keyword volume is night and day. On the order of 5000 searches/mo for automobiles, and 1,000,000 searches/mo for cars (again, not true for these example keywords but this is the actual ratio against our real keywords). So, do we optimize heavily against a lesser known and used keyword, or do we try to target "cars" which sees higher search volume even though it's significantly more competitive? We have this problem with a few different branded keywords, actually, and our search performance reflects as much. If someone were to search for these lesser known keywords they would find us, and we rank well. But the volume isn't there. Thoughts? Thanks in advance.
Conversion Rate Optimization | | J-Me0 -
Meta Descriptions - Does Cutt's comment still hold true?
When looking through my old research on Meta's came across this article on Search Engine Land where Matt Cutt's stated... "In fact, Matt said for his own blog, he doesn’t bother to make meta descriptions for his own site. In short, it is better to let Google auto-create snippets for your pages versus having duplicate meta descriptions."(http://searchengineland.com/googles-matt-cutts-dont-duplicate-your-meta-descriptions-177706) Recently, I have seen Google continuously play around with my metas, pulling from some really weird places on the page, discrediting the meta that I have there..and more-or-less really f them up.. Does the community think that maybe what Matt said in 2013 still holds weight and we are just wasting time with metas since Google does what they want in that space anyways?
Conversion Rate Optimization | | GoAbroadKP2 -
How to track conversions?
Hi Mozers, Anyone know how I can go about finding what my conversion is on my site? Or how to go about setting up tracking?
Conversion Rate Optimization | | edward-may0 -
Does adding a well scripted and produced video help significantly in conversions from visits to my divorce lawyer website to calls for my services?
Should the addition of well scripted and produced video to my divorce lawyer website increase my conversions and, if so, to what extent?
Conversion Rate Optimization | | hildebrandlaw0 -
Adwords Conversion Optimiser
OK Mozers! Here's the situation (my parents went away on a weeks vacation...) • I am managing an online furniture retail website Infurn .com • Our budget is 500 euro per day and we need basically 20% cost per sale and im hovering at 30 - 35% • I recently was given the opportunity to participate in The Conversion Optimiser which allows you to set a desired CPA and Google works its magic geting you that target by adjusting your bids accordingly using historical campaign data. • Is this something I should use? Or should I continue to hone my ad groups, bids, products, landing pages, and ad copy until I reach my 15 - 20% CPA goal? Thanks!
Conversion Rate Optimization | | TheGrid0 -
Reviews not shown on google search & seller rating extensions (adwords)
Hello, we have 2 questions, regarding reviews from rating websites. 1. Reviews on google Search We have several reviews for our website www.theprintspace.co.uk from various rating sites (e.g. qype, yelp, remotegoat, etc.). When searching for theprintspace on google, only some reviews and only some rating website are shown, most are not. For qype, for instance, we have 8 reviews, but google only indicates that we have 3. The yelp reviews (and many others) are not shown at all. According to google, it takes a maximum of 2 months for the correct number of reviews to be shown on google, but the reviews have been online for over 2 months now. Is there any way this problem can be solved, so google shows the correct number of reviews? 2. Seller rating extensions on google adwords We would like the reviews to be displayed in our google adwords as a seller rating extension. These will include the merchant star rating for advertisers that are highly rated on Google Product Search. We are setting up a google merchant account to do so. Google says: "Ratings are based on the user reviews collected by Google Product Search, which aggregates reviews from a broad base of sources across the web, including Reseller Ratings, Bizrate, ReviewCentre.com and Viewpoints." Do you know if the following rating website are in google's "base of sources" and will thus be recognised and used in google adwords as rating extensions? Yelp Qype Scoot Remotegoat viewlondon tipped Or does google only source from big e-commerce rating websites? How do we get the ratings into our adwords? Thank you for your help!
Conversion Rate Optimization | | Waplington0 -
Conversion optimization with Optimizely, impact on SEO
I am looking into signing up with www.optimizely.com for conversion testing. They put scripting on my site which will then redirect half of my site visitors to an alternate home page. The site I want to test on is ranking quite well with Google and I do not want to hurt my rankings. And with this set up,what Google will think is my home page is not and so I am essentially sending visitors to a different page than Google reads as my home page in source code. So, my concern is whether this will have a negative impact on my SEO rankings to redirect 50% of site visitors to a different page using this testing tool? I would use Google Web Site Optimizer, but many of my sites are in Wordpress and it seems that Wordpress and web site optimizer are not so compatible. Advice would be appreciated. Thank you, Robert
Conversion Rate Optimization | | Robertnweil11