OperationsFramework / modelModelAccessible

Regression analysis

When and how should regression analysis be applied?

AccessibleStrategicTeam3 min read

Contents

Regression analysis is a statistical tool for investigating the relationship between variables.

Regression analysis estimates how an outcome varies with one or more explanatory variables. It can describe associations, support prediction and—under a credible research design—help estimate causal effects. The calculation alone does not turn an association between price and demand, or any other pair of variables, into causation.

When to use it

Use regression when a decision requires a quantified relationship, a conditional prediction or a test of how an outcome differs as selected inputs change.

It can help investigate questions such as:

How are loyalty and satisfaction associated with profitability after accounting for relevant differences?

Does brand perception predict sales beyond price, distribution and seasonality?

Which dimensions of product quality are associated with customer satisfaction?

Is employee engagement related to retention, and what alternative explanations must be considered?

Regression is appropriate only when the variables can be measured meaningfully, the sample supports the model and the assumptions fit the intended inference. For causal questions, design matters more than statistical significance.

Origins

Regression emerged from nineteenth-century work in statistics. Francis Galton introduced the idea of “regression toward mediocrity” while studying inherited characteristics; Karl Pearson formalised correlation and regression, while Adrien-Marie Legendre and Carl Friedrich Gauss contributed the least-squares method used in linear models. The technique later became foundational in economics, science, social research and policy analysis.

What it is

A regression expresses a dependent variable as a function of one or more independent variables plus unexplained variation. In a simple linear case, the slope estimates the average change in the outcome associated with a change in the predictor.

Correlation summarises the direction and strength of association between variables symmetrically. Regression assigns different roles to predictors and outcome, estimates conditional relationships and can include several predictors. Neither method, without additional assumptions or design, proves what causes what.

A model can be used for explanation or prediction, but those goals require different validation. A coefficient may be statistically distinguishable from zero yet too small to matter, and a model that fits historical data may predict new cases poorly. In legal, government and business settings, results must therefore be interpreted with effect sizes, uncertainty, model assumptions and the provenance of the data.

How to use it

Begin with a decision question and a causal or predictive diagram. Define the outcome, predictors, population, time order and plausible confounders before inspecting results. Specify what evidence would challenge the hypothesis.

A regression model contains:

unknown parameters, denoted as b;

independent variables, X;

a dependent variable, Y.

The model relates Y to a function of X and b, together with an error term representing variation not captured by that function.

Select a model that matches the outcome and data-generating process. Linear regression is not automatically appropriate for binary outcomes, counts, time series, grouped observations or non-linear relationships.

A complete analysis may involve:

estimating coefficients through least squares or another suitable method;

checking functional form and interactions;

examining predicted values and residuals;

assessing heteroskedasticity, dependence and influential observations;

reporting uncertainty, effect size and goodness of fit such as R-square;

testing performance on data not used to fit the model; and

conducting sensitivity analysis for omitted variables and alternative specifications.

Software can perform the arithmetic, but it cannot decide whether the sample is biased, a variable is measured badly, a future period resembles the past or a coefficient has a causal interpretation. Preserve code, definitions and model versions so the analysis is reproducible.

Practical example

An estate agent wants to understand the factors associated with sale price. The dataset includes floor area, bedrooms, reception rooms, neighbourhood income, time on market and a structured condition or appeal assessment.

A regression can estimate how price varies with these attributes conditional on the others. Bedrooms may add little once floor area is included, or a location effect may interact with size. The agent should account for selection into listings, changing market conditions, spatial dependence and the fact that asking strategy can affect both time on market and final price. Test predictive accuracy on later or held-out sales before using the model for valuation, and retain a human review for unusual properties.

Top practical tip

Write down the intended interpretation of every important coefficient before fitting the model. Then check data lineage, missingness, time order, confounding, residuals and out-of-sample performance. A familiar technique still requires a defensible design.

Top pitfall

Do not translate statistical significance into causation, importance or predictability. A very small, biased or unstable effect can be highly significant in a large sample, while data leakage can make a poor forecasting model look excellent.