You can’t A/B test to success

Companies like Netflix, Booking and Amazon test everything. They try to make decisions as much as possible based on data. It almost makes you believe that extensive A/B testing is key to success.

People were renting movies by visiting stores and driving back to return rentals. After Netflix offered the option to rent DVDs by mail, nobody had to leave their home to rent a movie anymore.

Nokia was leading the race for smaller, better and faster mobile phones. Every new model improved upon the previous one. Until Apple came along and changed the competition for everyone. Apple transformed what people expected from their phone.

Bach was not appreciated by music critics when performed on the piano. As Bach wrote the original pieces for performance on the harpsichord. Until Glenn Gould was able to perform Bach beautifully on the piano in a way even the critics could cherish.

A/B testing polishes what already exists. It offers a great path for incremental improvement. It helps finding the local optimum of the existing paradigm you are working in. You limit yourself to the degrees of freedom that exist in your box.

If you want to disrupt, A/B testing is not an option. You need to step out of the box. You need to have guts and take risks. You need to be brave to discover a new global optimum. Launch something that does not fit an A/B test. Go for the crazy A and worry about the B later.

You can’t A/B test to success was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.