什么是实验?
在数字世界中，最经典的实验是AB测试. 但除了AB测试，还有更多的实验方法. To measure the appetite for a new idea, you might fake door test it. 了解忠实客户对新功能的反应, 你可以进行beta测试(封闭或开放)。. 如果你要发布对网站的修改, you might have phased rollouts or feature flags as part of your release process (arguably, AB测试的一种).
The point is: think of an experiment as a method of measuring the effect of change. AB测试，beta邀请，假门，所有这些.
So, now we’re clear there are multiple types of experiment, let’s go back to this. There are two concepts of experiment – confirmatory, 和 exploratory.
I’m going to do my best to explain the two concepts 和 give you some pros 和 cons. 绑在…
验证性实验
I’ve started with confirmatory experiments because these are the types of experiments you’re probably familiar with.
Imagine you’ve just watched a day’s worth of usability testing 和 7 out of 10 users got to your product page 和 missed the free delivery message. You know from a prior survey that free delivery is important to your customers.
You decide that you need to make the free delivery message more obvious on the product page. You ask your designer to design a new version of the product details section 和 this time, 让免费送货更加明显!
你的假设可能是这样的:
Because we have observed 7/10 of customers missed the free delivery message on product pages in usability testing, 虽然受访客户表示免费送货很重要.
我们相信让免费送货的信息更加突出.
This will result in more customers seeing the message 和 being aware of our free delivery, 因此更有可能购买.
当产品页面的购买率增加时确认.
Your designer goes to work 和 produces a lovely new design that makes free delivery really… freakin’ …obvious.
你测试它. 它赢了. 购买率提高8%. 96%的意义. 波什. 为了对数据的热爱，请有人打开香槟!
这是一个验证性实验.
你推测B版本(更大), 更大胆的免费送货)比版本A(弱小)更好, 迷你免费送货). 你检验了那个理论. 你将从这个实验中学到以下三件事之一:
1. 版本A更好(该死) )
2. 版本B更好 )
3. 版本A和B是一样的 )
现在让我们看看另一种情况.
探索性实验
我们又回到了一天结束的时候, you’ve just watched all those people miss the free delivery message. 你知道这是一条重要的信息.
假设是一样的. 但是，这一次你需要向设计师提供以下要点:
- 人们错过了产品页面上的免费送货信息, 但我们知道让他们看到这一点很重要.
- 你能给我一些信息的变体吗? First, try a design making the message more prominent early in someone’s journey. Let’s have a banner on the product results pages after the second row of results.
- 接下来，试着让它在产品页面上更加突出. 把它放在接近价格的地方.
- 最后，向购物篮页面添加一条免费投递消息. Show it near the total price so it’s clear it won’t be added later.
Your designer does three different versions… Let’s call them variations. 您运行AB/CD测试与所有三个针对当前站点.
以下是你可能从这个实验中学到的东西:
1. A是最好的，然后是B, C, D
2. B最好，然后是A, C, D
3. B最好，然后是C, A, D
4. D是最好的，A, B, C是相等的
5. C是最好的. B和D是相等的. A是最差的.
6. 等等......。
My point is, your capacity for learning from this experiment has increased enormously. With some smart analysis you could learn that showing the message earlier in the journey is better. 你可以学到完全相反的东西:越晚越好. You could learn that the pop up has a negative impact – probably because it’s an annoying pop up – that the improved product page banner has a positive impact (8%, 还记得?)，但是…… 篮子 信息具有巨大的积极影响(14%)!).
You might even iterate on this experiment 和 run version two with both the product page 和 篮子页面横幅. You might find that combined you get an even greater uplift of 16%!
关键是:你学会了.
“天哪，我被卖了.”
有什么隐情?? 没有什么大. 但也有一些考虑.
你必须有足够的交通才能灵活
谷歌“ab样本大小计算器” 并使用其中一个免费工具. 如果你需要10个,每个变化有000个回合，你得到10个回合,000 sessions per month…you may want to explore usability testing 和 confirm through an AB.
不要把设计扔到墙上，希望它们能坚持下去
That’s like when you first learned what a multivariate test was 和 made a 21-variation split test.
Instead, articulate what you want to teach your designer (or yourself, if you are your designer). 想想在光谱上进行测量. Examples include: early vs late in the journey; subtle vs LOUD; emotional vs rational; price-driven vs quality-driven.
You want to be able to come away from this (series of) experiment(s) 和 tell your team something like:
“We tested free delivery messaging throughout the journey in different ways 和 we found it’s best to include it everywhere there’s mention of a price, 和 to include it in a subtle way rather than an obnoxious banner.”
最终的想法
别以为我在说验证性实验不好! 他们绝对有一席之地. 有时候设计是不可行的 并建立 三到四个变体. Sometimes you need to ask questions 和 it’s better to explore through usability studies. Sometimes you just have one clear idea 和 you want to see if it’s better to speak a whole thread of other ideas.
如果你从这篇文章中得到什么, it’s that you should know the difference 和 look for opportunities to explore. 如果这次不行，下次再试.
保持试验.
