It’s been short of two weeks since I’ve had the privilege to speak about some of the most interesting experiments that me and my teams have been conducting that generally led us to very insightful results to say the least.
What’s cooler about it is that the conference was hosted and organized by one of our team’s favorite online publisher of tests and results, Which Test Won.
My communication goal was essentially to get everybody thinking about testing in terms of subsequent increments, based on hypotheses, the champion-challenger model, rather than generic “let’s see what happens” strategies, which is by the way, the reason why I’m not such a big fan of MVT.
Since I’ve had almost an hour at my disposal, I’ve obviously spiced up my talk with examples ranging from basic, more theoretical examples, to new and innovative stuff that, to be hones, I have yet to see elsewhere in the industry.
The basic examples were centered around the option to do AB in either sequential or split environments, which the pro and cons of each. Slides attached at the end of the article.
The more advance ones were discussing tracking conversions further down the funnel, by stressing that’s obviously not good enough to look at CTR or any intermediate metrics, and that you have to take a look at tral conversions.
One example that I particularly enjoyed presenting is the way that we managed to connect a couple of funnels which start and live off of different media. That is Avira’s download flow, which thanks to this implementation can now be optimized also in terms of how the visual representation on the website can influence the open rates of the downloaded software.
How we achieve that? Well, we use the source system (the website) to generate a unique ID which is then passed to the downloaded file as an argument in the filename, which can be read by the software upon installation, and by making calls to the same backend system (mixpanel), from the two frontends (website and product) the two fonnels become connected. What’s more to it, is that since you pretty much customize your calls you can send variables in those calls that make reference to say, a variant that is being tested on the website, performance of which can later be tracked and interpreted throughout the funnel across the two different media. This is also present in one of the last slides, and it is something that has helped us leap open rates of our downloads by considerable amounts.
