OperationsFramework / modelModelAccessible

Business experiments/experimental design/AB testing

When and how should business experiments/experimental design/ab testing be applied?

AccessibleStrategicIndividual3 min read

Contents

Business experiments, experimental design and AB testing are all techniques for testing the validity of something – be that a strategic hypothesis, new product packaging or a marketing approach.

Business experiments use controlled comparison to test a strategic assumption, product change, package design or marketing treatment before a full commitment is made. Experimental design defines how the evidence will be produced; A/B testing is one common form comparing two variants.

When to use it

Innovation and strategic change rely on assumptions that customers may not confirm. A well-designed experiment reduces uncertainty by exposing a limited population to alternatives and measuring the difference before the organisation scales one choice.

Use experimentation when two or more viable options exist, a wrong decision would be costly and the outcome can be measured safely on a smaller scale. Results can identify the stronger option and reveal how to refine it.

Experiments can answer:

Which treatment causes a meaningful increase in sales?

Which product should be released first?

Which offer do target customers prefer in behaviour, not only in stated opinion?

Which campaign produces the highest incremental response?

Which recruitment channel generates the best qualified hires at acceptable cost?

Origins

Controlled experiments have roots in scientific and agricultural research. During the 1920s, statistician Ronald A. Fisher formalised randomisation, replication and experimental design at Rothamsted Experimental Station. Randomised controlled trials later became central to medicine and social science. Businesses adopted direct-mail and market tests throughout the twentieth century, while digital products made large-scale A/B tests faster and cheaper from the 1990s onward. The underlying logic remains counterfactual: estimate what changed because of the treatment by comparing otherwise equivalent groups.

What it is

“Business experiment” is the broad term. Experimental design specifies treatments, assignment, samples and measurement; A/B testing compares a control A with a changed version B and is common in digital and direct marketing. Whatever the label, the objective is maximum decision-relevant information with minimum bias. A television pilot applies a looser version of the same principle by testing audience response before financing a full series.

How to use it

Thomas H. Davenport summarises the process in four stages:

State a hypothesis.

Design the experiment.

Run it.

Analyse, decide and follow up.

Create a hypothesis. Specify the treatment, expected causal effect, primary outcome and decision threshold before seeing results. Examples include “reducing packaging will increase unit sales” or “moving the buy button from bottom left to top right will increase completed purchases.” Ensure that:

the outcome and guardrail metrics can distinguish improvement, harm and inconclusive evidence;

the experiment fits the organisation’s strategy, values, consent obligations and reputation;

the decision is valuable enough to justify the test.

Design the experiment. Choose the population, control, treatment, random-assignment method, sample size and duration. A simple before-and-after comparison is vulnerable to seasonality and other changes occurring at the same time; a concurrent randomised control is normally stronger. In a basic A/B test, B differs from A in one deliberate respect so the effect can be attributed more credibly. Keep the design as simple as the decision permits, but use factorial designs when interactions among multiple changes are the question.

Run the experiment. Inform affected employees and participants as required, document the protocol and monitor execution without repeatedly peeking and stopping at a convenient result. Record events that threaten validity. If the reduced-packaging product is unavailable for four days, lower sales may reflect stockout rather than customer preference.

Analyse and follow up. Compare the observed effect and uncertainty with the pre-specified hypothesis and threshold. Being wrong can still produce a valuable experiment; an unchanged purchase rate after moving the button prevents investment in an ineffective idea. Look for unintended effects and differences among relevant segments, but distinguish planned analysis from exploratory findings. A clear result may support rollout; an inconclusive one may require a better-powered follow-up. Test subsequent modifications against the same control until the evidence justifies a choice.

Practical example

Imagine managing fundraising for a large environmental charity with a successful but expensive direct-mail control. You want to test three hypotheses:

a handwritten sticky note with a personal request increases response;

a prepaid return envelope increases response;

presenting a high donation amount first increases average gift size.

Randomly select four groups of 2,000 from 500,000 active supporters. One receives the control. The other groups receive the same pack with only one change: a sticky note, a prepaid envelope or donation options beginning at £50 rather than £20 and descending to £5 plus “Other.” Mail every group on the same day.

After three weeks, the sticky note produces the largest response lift but its manual cost creates a slight net loss, so reserve it for high-value donors. The prepaid envelope changes nothing, allowing that cost to be removed. The higher opening ask increases average gift substantially and is rolled out. The example shows why response, net contribution and gift size must all be assessed rather than declaring a winner from one metric.

Top practical tip

Pre-register the hypothesis, primary outcome, guardrails, sample rule and stopping rule. Randomise comparable units concurrently and keep treatment delivery identical except for the intended change.

Top pitfall

A poorly implemented prototype tests execution quality rather than the concept. Also guard against underpowered samples, repeated peeking, novelty effects, interference between groups and optimising a local metric at the expense of customers or long-term value.