1. Beware of Outliers

Recent years have been anything but predictable. Often, to evaluate the success of a change, companies rely on historical business data and compare results – sales, bookings, churn – before and after a product update, a new customer success initiative, a marketing campaign or another operational adjustment. However, history can be an unreliable comparison, particularly in the wake of a pandemic or in the face of volatile, inflationary, or recessionary conditions. Data in a volatile economic climate is especially challenging as it can only accurately represent a single moment in time. Even under normal circumstances, external “outlier” events such as unexpected press coverage, changing customer demographics, or seasonality can temporarily skew your analysis. Straightforward before-and-after comparisons will inevitably lead to incorrect conclusions if qualitative factors are not considered. Most businesses need a more sophisticated analytical toolkit to account for the unpredictable and unexpected.

2. Experimentation Leads to Enlightenment

Experimenting or trying, failing, and then trying again, is not a new concept. Yet, testing an unknown result can still feel risky to businesses. In the early 17^th century Age of Enlightenment, French philosopher, and creator of the first encyclopedia, Denis Diderot outlined the three principles of knowledge: observation to collect the facts, reflection to combine them and most importantly, experimentation to test and verify them.  Any data-driven culture should be grounded in experimentation: testing changes and letting the data guide you. We see this approach regularly applied to marketing. You may have heard about – or even tried! – A/B testing on website button colors or email headlines. But the power of experimentation also matters in product development, customer support, operations, and in countless areas across your business. With experimentation, some customers receive a new experience while others receive the existing “control” experience. Results between the two groups can be compared because they occur under the exact same external conditions – allowing you to see the true effect. The ability to understand what is currently working and why will help your team find signal through the noise so they can invest their time and resources more efficiently.

3. Validate, Invest, Succeed

Change is inevitable in any economic environment. The value of experimentation is in understanding that change and steering it toward growth. Resources, whether time or budget, are scarce even in the best of times. As businesses continue to navigate changing market conditions, they need to be highly conscious of how resources are allocated. Here too, experimentation is key.

While teams often consider experimentation in the context of optimizing success – picking the option customers like best – A/B testing can also be used to avoid failure; quantifying the impact of product, process and other changes that can challenge results. When was the last time someone in your product organization told you, “We changed something, we think customers like it, but we don’t know how much sales have changed?” Or, “it was only a back-end change, customers won’t even see it!” Experimentation can eliminate risk and be an early identifier of the difference in conversion, sales or any other metric used to measure the new experience against the control group or status quo. Informed by the data, your team can invest talent and budget in ideas and experiences that work and drop the ones that don’t.