Data mining

Data mining is often used loosely to mean any large-scale data processing, but its purpose is more specific: discovering useful, previously unrecognised patterns and relationships. The name can be misleading because the objective is not to extract more data; it is to turn data into validated knowledge that improves a decision.

When to use it

Use data mining when a substantial dataset may contain patterns capable of improving prediction, classification, detection or prioritisation. The resulting insight can reduce cost, support planning, change strategy and lower decision risk.

A discovered association does not explain its own cause. The method can expose a pattern, anomaly or dependency, but additional analysis or experimentation may be needed to establish why it occurs and whether acting on it will work.

Typical questions include:

• Which characteristics are shared by the most profitable customers?
• How can customers in the smart-watch market be grouped?
• Which signals recur in fraudulent transactions?
• What navigation paths do website users follow?

Origins

Data mining emerged from the convergence of statistics, database systems, artificial intelligence and machine learning. The knowledge-discovery-in-databases community formalised the field through workshops beginning in the late nineteen eighties. Usama Fayyad, Gregory Piatetsky-Shapiro and Padhraic Smyth later distinguished data mining—the algorithmic pattern-discovery step—from the broader process of selecting, preparing, interpreting and deploying useful knowledge.

What it is

Data mining is the automatic or semi-automatic exploration of data, often very large business datasets, to find patterns, anomalies and dependencies that are both previously unknown and potentially useful. It combines computational techniques with domain judgement: a pattern must survive validation and lead to a relevant action before it becomes business insight.