Predictive Modeling and Machine Learning Fundamentals Questions
Applied predictive-modeling cases worked end to end on tabular business data. Covers choosing a model family for a business target (churn, lifetime value, lead scoring, conversion, support-ticket escalation), setting a baseline, defining the label, observation window and prediction horizon, catching leakage, sanity-checking and interpreting results for non-technical stakeholders (feature importance and its traps with correlated features, explaining a single flagged customer), handling censored outcomes with survival methods such as Kaplan-Meier and Cox models, turning model scores into thresholds and business actions with cost-sensitive reasoning, targeting for incremental effect with uplift modeling, and clustering and unsupervised methods in practice: k-means and its initialization (k-means++), empty clusters, scaling, distance choices and mini-batch variants, choosing and validating the number of clusters without labels (elbow, silhouette, stability), density-based, hierarchical and mixture-model alternatives, mixed numeric and categorical features, high-dimensional preparation and visualization, and customer segmentation carried through to stakeholder communication, testing whether segment-based targeting works, keeping segments stable over refits, and downstream use. Excludes regression and classification algorithm internals, metric and validation theory, feature engineering mechanics, and large-scale or production ML systems (including distributed or streaming clustering), which are covered elsewhere.
How does a Gaussian mixture model differ from k-means in practice? When would soft assignments and per-cluster covariance shapes change your conclusions, and how are the two methods related?
Sample Answer
The idea in plain words
k-means puts every point in exactly one cluster: the one with the nearest centre, using ordinary distance. That quietly assumes every cluster is a round ball of the same size. A Gaussian mixture model (GMM) says the data were drawn from a blend of bell-shaped clouds, each with its own centre, its own spread and tilt (a covariance matrix, which records how wide the cloud is along each direction and how the directions co-vary) and its own share of the data (a mixing weight). Instead of a hard label, each point gets a responsibility for each cloud: the probability it came from that cloud. k-means is the special case where the clouds are forced to be equal, round and so tight that every responsibility becomes 0 or 1.
The mechanism
Both methods alternate two steps. For the GMM the loop is called expectation-maximisation (EM): the E-step (expectation) computes each point's responsibilities from the current parameters, and the M-step (maximisation) refits the parameters from those responsibilities. Each pass can only raise the likelihood (the probability the model assigns to the observed data), until it settles at a local optimum, so the result depends on where it started.
- Assign. k-means: each point goes to the nearest centre. GMM (E-step): point xi gets responsibility for component k
where πk is the mixing weight, μk the centre, Σk the covariance matrix and N(xi∣μk,Σk) the Gaussian density: the height of the bell-shaped cloud with centre μk and shape Σk at the point xi, which is large when the point is typical of that cloud. In words: a component's responsibility for a point is its weight times its density at the point, divided by the same quantity summed over all components, so the responsibilities of one point add up to 1. Part (d) of the worked example below does this arithmetic for one point.
- Update. k-means: each centre becomes the mean of its points. GMM (M-step): μk is the responsibility-weighted mean, Σk the responsibility-weighted covariance, and πk the average responsibility.
If you fix πk=1/K and Σk=σ2I for all components and let σ→0, the responsibility of the nearest centre tends to 1 and all others to 0, so the E-step becomes k-means' assignment step and the M-step becomes its mean update. That is the sense in which k-means is a limiting case of EM.
The reason is one line of algebra. With equal weights and the same covariance for every component, the weights and the normalising constants cancel, leaving
rik=∑jexp(−∥xi−μj∥2/(2σ2))exp(−∥xi−μk∥2/(2σ2))a softmax of minus the squared distance, divided by 2σ2. Divide the top and bottom by the nearest centre's term: every other term becomes exp(−extra squared distance/(2σ2)), which goes to 0 as σ→0, so the nearest centre's responsibility goes to 1.
Practical differences
The covariance types in the table are the choices for how much shape each component may have: spherical (one variance in every direction, round clouds), diagonal (a separate variance per feature, axis-aligned ellipses), tied (all components share one full covariance) and full (each component has its own tilted ellipse). For an interview the core rows are assignment, cluster shape and covariance type; the BIC and collapse rows are follow-up territory.
| k-means | GMM | |
|---|---|---|
| Assignment | Hard (one label) | Soft (probabilities per cluster) |
| Cluster shape | Round, similar size and spread | Per-component covariance: round, axis-aligned or tilted ellipses, depending on the covariance type chosen |
| Cluster sizes | Implicitly similar | Free mixing weights |
| Parameters per component in d dimensions | d (the centre) | d plus d(d+1)/2 for a full covariance, plus a weight |
| Choosing the number of clusters | Elbow, silhouette | Likelihood-based criteria such as BIC (the Bayesian information criterion: the number of parameters times ln of the number of points, minus twice the log-likelihood; lower is better, and the penalty stops it adding components just to fit noise) |
| Risks | Wrong on elongated or unequal clusters | More parameters (a full covariance in 20 dimensions has 210 per component), sensitive to initialisation, can collapse onto one point (a component shrinking onto a single data point, its covariance going to zero and its density to infinity) unless a small ridge, a tiny constant added to the covariance diagonal, keeps it from collapsing |
A worked example you can run
Four runs on synthetic data, all with fixed seeds.
(a) Two long thin clusters tilted 35 degrees, side by side, 10 different datasets. Score: adjusted Rand index (ARI), 1.0 for a perfect match to the true groups and about 0 for a random split.
(b) Two overlapping round clusters and three probe points, to compare soft and hard assignment.
(c) The k-means limit: with equal weights and a shared variance σ2, how responsibilities harden as σ shrinks. The code computes the softmax of −d2/(2σ2) from the formula above, with d the distance to each centre.
(d) The responsibility formula applied by hand to the probe point x = 1.25: component weight times density, then normalised.
import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
from sklearn.mixture import GaussianMixture
th = np.deg2rad(35)
R = np.array([[np.cos(th), -np.sin(th)], [np.sin(th), np.cos(th)]])
def stretched(rng, n=300, off=3.0):
"""Two long, thin clusters tilted 35 degrees, side by side (true labels known)."""
a = (rng.normal(size=(n, 2)) * [2.0, 0.4]) @ R.T
b = (rng.normal(size=(n, 2)) * [2.0, 0.4] + [0, off]) @ R.T
return np.vstack([a, b]), np.r_[np.zeros(n), np.ones(n)]
scores = {k: [] for k in ("k-means", "GMM spherical", "GMM full, k-means init", "GMM full, random-point init")}
for seed in range(10):
X, y = stretched(np.random.default_rng(seed))
fits = {
"k-means": KMeans(2, n_init=10, random_state=seed).fit(X).labels_,
"GMM spherical": GaussianMixture(2, covariance_type="spherical", n_init=10, init_params="random_from_data", random_state=seed).fit(X).predict(X),
"GMM full, k-means init": GaussianMixture(2, covariance_type="full", n_init=10, init_params="kmeans", random_state=seed).fit(X).predict(X),
"GMM full, random-point init": GaussianMixture(2, covariance_type="full", n_init=10, init_params="random_from_data", random_state=seed).fit(X).predict(X),
}
for k, lab in fits.items():
scores[k].append(adjusted_rand_score(y, lab))
print("tilted elongated clusters: adjusted Rand index against the true labels, 10 datasets")
for k, v in scores.items():
print("%-28s mean=%.2f runs above 0.9: %2d of 10" % (k, np.mean(v), sum(s > 0.9 for s in v)))
# one start at a time: how often a single start finds the true split (20 starts on each of the 10 datasets)
print("\nshare of single starts reaching ARI above 0.9, and datasets recovered with 50 restarts")
for init in ("kmeans", "random_from_data"):
single, fifty = [], 0
for seed in range(10):
X, y = stretched(np.random.default_rng(seed))
one = [adjusted_rand_score(y, GaussianMixture(2, covariance_type="full", n_init=1, init_params=init, random_state=100 * seed + j).fit(X).predict(X)) > 0.9 for j in range(20)]
single.append(np.mean(one))
many = GaussianMixture(2, covariance_type="full", n_init=50, init_params=init, random_state=seed).fit(X).predict(X)
fifty += adjusted_rand_score(y, many) > 0.9
print("%-18s single start: %.2f 50 restarts: %2d of 10" % (init, np.mean(single), fifty))
# Soft assignment: two overlapping round clusters, three probe points
rng = np.random.default_rng(0)
X2 = np.vstack([rng.normal([0, 0], 1.0, (300, 2)), rng.normal([2.5, 0], 1.0, (300, 2))])
gm = GaussianMixture(2, covariance_type="full", n_init=5, random_state=0).fit(X2)
km = KMeans(2, n_init=10, random_state=0).fit(X2)
order = np.argsort(gm.means_[:, 0]) # left cluster first
kmo = np.argsort(km.cluster_centers_[:, 0])
pts = np.array([[-0.5, 0.0], [1.25, 0.0], [3.0, 0.0]])
print("\nsoft versus hard assignment on overlapping clusters")
print("x P(left) P(right) k-means picks")
for x, pr, kl in zip(pts[:, 0], gm.predict_proba(pts)[:, order], km.predict(pts)):
print("%5.2f %7.3f %8.3f %s" % (x, pr[0], pr[1], "left" if kl == kmo[0] else "right"))
# the same number by hand for the probe at x = 1.25: weight times density for each component, then normalise
from scipy.stats import multivariate_normal
probe = pts[1]
print("\nhand trace at x = 1.25 (components ordered left, right)")
wd = []
for k in order:
dens = multivariate_normal(gm.means_[k], gm.covariances_[k]).pdf(probe)
wd.append(gm.weights_[k] * dens)
print("mean x=%5.2f weight=%.3f density=%.4f weight*density=%.4f" % (gm.means_[k][0], gm.weights_[k], dens, wd[-1]))
print("responsibilities = weight*density / sum = %.3f and %.3f" % (wd[0] / sum(wd), wd[1] / sum(wd)))
# k-means as a limit: equal weights, shared spherical variance sigma^2, centres fixed at the k-means centres
C = km.cluster_centers_
d2 = ((X2[:, None, :] - C[None, :, :]) ** 2).sum(axis=2)
print("\nresponsibilities with equal weights and shared spherical variance (k-means centres)")
print("sigma mean max-responsibility points below 0.99")
for sigma in (2.0, 1.0, 0.5, 0.25, 0.1):
logw = -d2 / (2 * sigma ** 2)
r = np.exp(logw - logw.max(axis=1, keepdims=True))
r /= r.sum(axis=1, keepdims=True)
assert (r.argmax(axis=1) == km.labels_).all() # the most responsible component is the nearest centre
print("%5.2f %22.3f %17d" % (sigma, r.max(axis=1).mean(), (r.max(axis=1) < 0.99).sum()))
Output:
tilted elongated clusters: adjusted Rand index against the true labels, 10 datasets
k-means mean=0.11 runs above 0.9: 1 of 10
GMM spherical mean=0.02 runs above 0.9: 0 of 10
GMM full, k-means init mean=0.80 runs above 0.9: 8 of 10
GMM full, random-point init mean=1.00 runs above 0.9: 10 of 10
share of single starts reaching ARI above 0.9, and datasets recovered with 50 restarts
kmeans single start: 0.39 50 restarts: 10 of 10
random_from_data single start: 0.62 50 restarts: 10 of 10
soft versus hard assignment on overlapping clusters
x P(left) P(right) k-means picks
-0.50 0.978 0.022 left
1.25 0.328 0.672 right
3.00 0.005 0.995 right
hand trace at x = 1.25 (components ordered left, right)
mean x=-0.20 weight=0.455 density=0.0542 weight*density=0.0247
mean x= 2.33 weight=0.545 density=0.0925 weight*density=0.0505
responsibilities = weight*density / sum = 0.328 and 0.672
responsibilities with equal weights and shared spherical variance (k-means centres)
sigma mean max-responsibility points below 0.99
2.00 0.699 600
1.00 0.901 399
0.50 0.975 102
0.25 0.994 25
0.10 1.000 2
Reading it:
- Shape. k-means splits the pair across the long axes and reaches ARI above 0.9 on only 1 of 10 datasets (mean 0.11). A spherical-covariance GMM has the same round-cloud assumption and does no better (mean 0.02). A full-covariance GMM, which can represent a tilted ellipse, is the one that recovers them. Its result depends on the start. Initialising from k-means (
init_params="kmeans") starts the GMM from one of k-means' own solutions, which on these data often cut across the long axes, so a start there frequently ends on the wrong split: a single k-means-initialised start reaches the true groups 39% of the time here, against 62% for a start from randomly chosen data points (init_params="random_from_data"). Restarts keep the fit with the best likelihood, so more restarts raise the chance that one of them lands on the right split: with 10 restarts the k-means start recovers the groups on 8 of 10 datasets and the random-point start on all 10, and with 50 restarts both recover all 10. These rates belong to this tilted geometry, not to a general ranking of initialisations. On well-separated round clusters the k-means start is a good default; in practice use several restarts and keep the fit with the highest likelihood (or lowest BIC). - Soft assignment. At x = -0.50 the GMM is 97.8% sure of the left cluster; at x = 1.25, in the overlap, it gives 0.328 and 0.672 while k-means just says "right". The probe at 1.25 is the midpoint of the true centres (0 and 2.5) but not of the fitted ones: the fitted left mean is -0.20 and the right mean 2.33, so 1.25 sits nearer the right one, whose weight is also larger (0.545 against 0.455). The hand trace shows it: weight times density is 0.455 x 0.0542 = 0.0247 for the left component and about 0.0505 for the right (0.545 x 0.0925 gives 0.0504 from the rounded figures shown; the program multiplies unrounded values), and 0.0247 / (0.0247 + 0.0505) = 0.328. The extra information is exactly the uncertainty a stakeholder needs: that point is a borderline case, not a typical member.
- Limit. The mean maximum responsibility rises from 0.699 at σ=2.0 to 1.000 at σ=0.10, and points that are still ambiguous (maximum responsibility below 0.99) fall from 600 to 2. The
assertin the program checks that at every σ the most responsible component is the k-means nearest-centre assignment.
When soft assignments and covariance shapes change your conclusions
- Borderline customers. With hard labels a customer 55% in one segment is reported like one 99% in it. Responsibilities let you treat the uncertain ones separately (a different message, or no action) and give an honest size for each segment (sum of responsibilities, not a head count).
- Elongated or rotated groups. If one feature is strongly correlated with another within a group (spend and order count rising together), the group is a tilted ellipse. k-means chops it or merges it with a neighbour. A full or per-cluster covariance keeps it whole.
- Different sizes and spreads. A tight small group next to a large diffuse one is a common pattern (loyal core, casual fringe). k-means draws the boundary midway between the two centres, so the outer part of the diffuse group, which lies beyond the midpoint, gets labelled as the tight group; the GMM can give the diffuse group a wide covariance and a different weight.
Choosing between them
For quick, interpretable segmentation on standardised numeric data with roughly round groups, start with k-means. Move to a GMM when clusters visibly differ in shape or size, when you need probabilities, or when you want likelihood-based model selection. Cap the covariance complexity (diagonal or tied) when data are few or dimensions many. Neither suits categorical variables or strongly non-Gaussian shapes; density-based methods (DBSCAN, which grows clusters outward from dense regions and leaves isolated points unassigned as noise) and hierarchical clustering (which builds a tree of merges you can cut at any level) are alternatives there.
Common mistakes
- Reading GMM responsibilities as certainty about the person: they are model-based probabilities, only as good as the Gaussian assumption.
- Fitting a full covariance with few points per component and getting a degenerate fit.
- Running one initialisation (one starting guess for the centres) and calling the result "the" GMM.
- Comparing BIC across different feature sets or scalings.
You must cluster records that combine numeric, ordinal, nominal and high-cardinality categorical features. What pipeline and distance or encoding approach would you use, and how would you validate the result?
Sample Answer
The idea in plain words
k-means averages numbers, so it only makes sense when every column is a number on a comparable scale. Records that mix numbers, ranked categories (ordinal), unordered categories (nominal) and categories with hundreds of levels (high cardinality) need two decisions made explicitly: how each column type is turned into a contribution to "how different are these two records", and how the clustering algorithm can use that. Then, because there are no labels, the result must be checked by stability and by whether the segments can be described and used, not by accuracy.
What I would try first
For this scenario I would commit to this order. First, deal with the high-cardinality column explicitly (coarsen it, for example city to region, or drop it from the first pass), standardise the numeric columns after a log transform, rank-code the ordinal column, and encode the few-level nominal column; run k-means on that. It is cheap, it scales, and every choice is visible. Second, run Gower distance with k-medoids on the same records as an independent check, because it handles each column type natively with a fixed rule (a range-scaled difference for numbers, a rank difference for ordinal, a mismatch for nominal) and needs no one-hot block or block weight; its range scaling and equal column weights are still choices, and the run below shows they matter. Segments that appear under both pipelines are more credible, and segments that appear in only one are suspect; where the two disagree strongly, find which scaling or column causes it before trusting either. Switch to the first pipeline alone when the record count makes an n x n distance matrix impractical. The experiment below supports this order: the standardise-and-drop-city pipeline was the best on average, and the frequency-coded city column made it worse.
The pipeline
- Audit each column. Count missing values, rare levels, and skew. Take the logarithm of heavy-tailed amounts such as income, group levels that appear in fewer than a handful of records into "other", and decide what missing means (a separate level, or imputation: filling the gap with an estimate such as the column's median).
- Give each type its own treatment so that no single column dominates:
- Numeric: rescale so the range or spread is comparable (standardise, or divide by the range).
- Ordinal (for example plan tier basic < plus < pro): use the rank, rescaled to [0,1] by rank/(L−1) where L is the number of levels. Treating it as nominal throws away the order; treating the raw integer as a measured quantity assumes equal gaps, which is a judgement call.
- Nominal with few levels (for example 4 channels): a mismatch costs a fixed amount (distance 1 if different, 0 if the same), or one-hot columns (a 0/1 column per level).
- High cardinality (for example 60 cities or thousands of merchants): see the next section.
- Pick a distance and an algorithm that agree with each other.
- Gower distance with a medoid method. Gower distance is the average over columns of a per-column difference scaled to [0,1]: absolute difference divided by the range for numbers, rank difference over (L−1) for ordinal, and 0 or 1 for nominal. Because the mean of two records is not defined for categories, pair it with k-medoids (centres are real records, the medoid is the record with the smallest total distance to its cluster) or with hierarchical clustering (which starts with every record alone and repeatedly merges the two closest groups, and needs only a distance matrix, not a mean). The cost is memory: the full distance matrix is n×n (about 80 GB for 100000 records, as the run below prints).
- Encode and standardise, then k-means. Cheaper and scales to large n, but the encoding choices (scales, the weight given to one-hot blocks) quietly set the distance. A sensible default is to make each one-hot block carry about as much weight as one numeric column.
- k-prototypes style methods run k-means on the numeric part and a mismatch count on the categorical part, with a weight balancing the two, and are one way to avoid building the full matrix.
- Validate (below), then choose k and profile the segments.
High-cardinality columns: what each encoding does to the distance
- One-hot gives one 0/1 column per level. Any two records with different levels sit at the same squared distance, and levels with few records contribute almost nothing. It also inflates the number of columns, which makes the pairwise distances more alike (distance concentration: as the number of columns grows, the nearest and farthest neighbours of a record end up almost the same distance away, so "close" loses its meaning) and the result more sensitive to how the block is weighted.
- Frequency encoding replaces the level by how common it is (a city with 40 records becomes the number 40, or 40 divided by the total). One column, no explosion, but different levels with the same frequency become identical, so it measures "popularity", not identity.
- Target encoding replaces a level by the average of an outcome. Clustering has no outcome. Using one (say churn) pushes that outcome into the segments, so any later comparison of the segments on that same outcome is circular (the outcome leaks into the groups), and it needs out-of-fold computation (computing each record's encoding from the other folds of the data, never from its own row) to avoid overfitting rare levels. Reserve it for supervised segmentation, where the goal is explicitly "groups that differ on this outcome".
- Embeddings map each level to a short list of numbers (a vector) learned from co-occurrence (which levels appear with which other values) or from an auxiliary prediction task, so levels that behave alike get nearby vectors. Distances between levels become graded: two similar cities are near each other, rather than a flat 0 or 1. The cost is a training step and a choice of dimension.
- Hierarchy or coarsening (city to region) is often the simplest and most explainable option, and the segment profiles stay readable.
Worked example from a run
The data are synthetic (600 records, 3 hidden segments, 10 different random seeds): age and income (numeric, income heavy-tailed), tier (ordinal, 0 to 2), channel (nominal, 4 levels) and city (60 levels, where half the records come from a block of 20 cities tied to their segment). The hidden segment is used only to score the result, with the adjusted Rand index (ARI: 1 means the clusters match the hidden groups exactly, 0 is what random labelling gives on average). The silhouette score below is computed on the Gower distance (for each record, how much closer it is to its own cluster than to the nearest other cluster, between -1 and 1). Rows F and G drop the categorical columns and differ only in the scaling (range or standard deviation). Stability is the ARI between a clustering of all records and a clustering of a random 80% subsample, over 20 draws.
Reading the dense one-liners: make() draws the hidden segment z first and then builds each column from it, so every column carries some signal about z (the city column only half the time). gower() builds one n x n matrix per column (v[:, None] - v[None] is every record minus every record) and averages them. scaled_matrix() turns each numeric or ordinal column into a z-score (subtract the mean, divide by the standard deviation), one-hot codes channel at weight 0.7, and adds the city's frequency as one more z-scored column. scaled_no_city() is the same without that last column. kmedoids() starts from random records as centres, assigns every record to its nearest centre in the matrix D, then, for each cluster, picks the member with the smallest total distance to the other members (D[np.ix_(...)].sum(0).argmin()) as the new centre, and repeats until the centres stop changing; the best of 10 random starts wins.
A hand check of the Gower formula comes first in the printed output: records 0 and 1 differ by 0.333 of the age range, 0.032 of the log-income range, 0 on tier, 0 on channel (same level) and 1 on city (different cities), and (0.333 + 0.032 + 0 + 0 + 1) / 5 = 0.273, the same as the function. Note that one mismatched city contributes more than all the numeric differences together, which previews the city finding below.
import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score, silhouette_score
def make(seed, n=600):
rng = np.random.default_rng(seed)
z = rng.integers(0, 3, n) # hidden segment, used only to score the result
age = np.choose(z, [28, 42, 58]) + rng.normal(0, 5, n) # numeric
income = np.exp(rng.normal(np.choose(z, [10.2, 10.8, 11.4]), 0.35)) # numeric, heavy right tail
tier = np.clip(np.round(np.choose(z, [0.3, 1.0, 1.7]) + rng.normal(0, 0.45, n)), 0, 2).astype(int) # ordinal 0<1<2
chan_p = np.array([[.7, .15, .1, .05], [.1, .7, .1, .1], [.05, .1, .15, .7]])
chan = np.array([rng.choice(4, p=chan_p[k]) for k in z]) # nominal, 4 levels
city = np.where(rng.random(n) < 0.5, z * 20 + rng.integers(0, 20, n), rng.integers(0, 60, n)) # 60 levels
return z, age, income, tier, chan, city
def gower(age, income, tier, chan, city, use_city=True):
parts = []
for v in (age, np.log(income)): # numeric: |difference| / range
parts.append(np.abs(v[:, None] - v[None]) / (v.max() - v.min()))
parts.append(np.abs(tier[:, None] - tier[None]) / 2.0) # ordinal: rank difference / (levels - 1)
parts.append((chan[:, None] != chan[None]).astype(float)) # nominal: 0 if equal else 1
if use_city:
parts.append((city[:, None] != city[None]).astype(float))
return np.mean(parts, axis=0) # equal weight per feature, result in [0, 1]
def raw_matrix(age, income, tier, chan, city): # raw units, ordinal as 0/1/2, one-hot nominals
return np.column_stack([age, income, tier, np.eye(4)[chan], np.eye(60)[city]])
def scaled_matrix(age, income, tier, chan, city): # standardised numerics, city by frequency
zs = lambda v: (v - v.mean()) / v.std()
freq = np.array([np.mean(city == c) for c in city])
return np.column_stack([zs(age), zs(np.log(income)), zs(tier), np.eye(4)[chan] * 0.7, zs(freq)])
def kmedoids(D, k, seed=0, restarts=10):
rng = np.random.default_rng(seed); best = (np.inf, None)
for _ in range(restarts):
med = rng.choice(len(D), k, replace=False)
for _ in range(50):
lab = D[:, med].argmin(1)
new = np.array([np.flatnonzero(lab == j)[D[np.ix_(lab == j, lab == j)].sum(0).argmin()] if (lab == j).any() else med[j] for j in range(k)])
if np.array_equal(new, med): break
med = new
cost = D[np.arange(len(D)), med[D[:, med].argmin(1)]].sum()
if cost < best[0]: best = (cost, D[:, med].argmin(1))
return best[1]
def scaled_no_city(age, income, tier, chan, city): # same as scaled_matrix but without the city column
zs = lambda v: (v - v.mean()) / v.std()
return np.column_stack([zs(age), zs(np.log(income)), zs(tier), np.eye(4)[chan] * 0.7])
def numeric_ordinal(age, income, tier, scale): # age, log income and tier only, no categorical columns
return np.column_stack([scale(v) for v in (age, np.log(income), tier.astype(float))])
km = lambda X, k=3: KMeans(k, n_init=10, random_state=0).fit_predict(X)
names = ["A raw units + one-hot, k-means", "B standardised + frequency code, k-means", "C Gower + k-medoids", "D Gower + k-medoids, city dropped", "E standardised, city dropped, k-means",
"F age, income, tier range-scaled, k-means", "G age, income, tier standardised, k-means"]
res = {n: [] for n in names}
for seed in range(10):
z, age, inc, tier, chan, city = make(seed)
res[names[0]].append(adjusted_rand_score(z, km(raw_matrix(age, inc, tier, chan, city))))
res[names[1]].append(adjusted_rand_score(z, km(scaled_matrix(age, inc, tier, chan, city))))
res[names[2]].append(adjusted_rand_score(z, kmedoids(gower(age, inc, tier, chan, city), 3)))
res[names[3]].append(adjusted_rand_score(z, kmedoids(gower(age, inc, tier, chan, city, False), 3)))
res[names[4]].append(adjusted_rand_score(z, km(scaled_no_city(age, inc, tier, chan, city))))
res[names[5]].append(adjusted_rand_score(z, km(numeric_ordinal(age, inc, tier, lambda v: v / (v.max() - v.min()))))) # the scaling Gower uses
res[names[6]].append(adjusted_rand_score(z, km(numeric_ordinal(age, inc, tier, lambda v: (v - v.mean()) / v.std()))))
z0, *_ = make(0)
print("reference: ARI of random labels vs the hidden segments", round(adjusted_rand_score(z0, np.random.default_rng(99).integers(0, 3, len(z0))), 2) + 0.0, "| of a perfect match", adjusted_rand_score(z0, z0))
print("adjusted Rand index against the hidden segments: mean over 10 seeds (min..max)")
for n in names:
print(f"{n:46s} {np.mean(res[n]):.2f} ({np.min(res[n]):.2f}..{np.max(res[n]):.2f})")
# Gower ablation: which columns help or hurt Gower + k-medoids (same 10 datasets; "region" is the block of 20 cities, city // 20)
def gower_subset(age, income, tier, chan, city, keep):
li = np.log(income)
cols = {"age": np.abs(age[:, None] - age[None]) / (age.max() - age.min()),
"income": np.abs(li[:, None] - li[None]) / (li.max() - li.min()),
"tier": np.abs(tier[:, None] - tier[None]) / 2.0,
"channel": (chan[:, None] != chan[None]).astype(float),
"city": (city[:, None] != city[None]).astype(float),
"region": ((city // 20)[:, None] != (city // 20)[None]).astype(float)}
return np.mean([cols[c] for c in keep], axis=0)
subsets = [("age, income", ["age", "income"]), ("age, income, tier", ["age", "income", "tier"]), ("age, income, channel", ["age", "income", "channel"]),
("age, income, tier, channel", ["age", "income", "tier", "channel"]), ("all five columns (row C)", ["age", "income", "tier", "channel", "city"]),
("all five + region", ["age", "income", "tier", "channel", "city", "region"])]
abl = {n: [] for n, _ in subsets}
for seed in range(10):
z, age, inc, tier, chan, city = make(seed)
for n, keep in subsets:
abl[n].append(adjusted_rand_score(z, kmedoids(gower_subset(age, inc, tier, chan, city, keep), 3)))
print("Gower + k-medoids with different column sets: mean ARI over 10 seeds (min..max)")
for n, _ in subsets:
print(f"{n:46s} {np.mean(abl[n]):.2f} ({np.min(abl[n]):.2f}..{np.max(abl[n]):.2f})")
z, age, inc, tier, chan, city = make(0)
D = gower(age, inc, tier, chan, city)
# Gower by hand for records 0 and 1: per-column difference, then the plain average
li = np.log(inc)
cols = {"age": abs(age[0] - age[1]) / (age.max() - age.min()),
"log income": abs(li[0] - li[1]) / (li.max() - li.min()),
"tier": abs(tier[0] - tier[1]) / 2.0,
"channel": float(chan[0] != chan[1]),
"city": float(city[0] != city[1])}
print("Gower by hand, records 0 and 1:", {k: round(float(v), 3) for k, v in cols.items()}, "-> average", round(sum(cols.values()) / 5, 3), "| gower() gives", round(D[0, 1], 3))
print("pairwise distance matrix size: n=600 ->", 600 * 600 * 8 / 1e6, "MB as float64; n=100000 ->", 100000 * 100000 * 8 / 1e9, "GB")
freq0 = np.array([np.mean(city == c) for c in city])
print("mean share of records in the record's own city, by hidden segment:", [round(float(freq0[z == s].mean()), 4) for s in range(3)])
print("silhouette of the TRUE segments on the Gower matrix:", round(silhouette_score(D, z, metric='precomputed'), 3))
print("share of record pairs with equal value: channel", round(float(np.mean(chan[:, None] == chan[None])), 2),
"city", round(float(np.mean(city[:, None] == city[None])), 3))
rng = np.random.default_rng(0)
print("k silhouette(Gower) stability (ARI over 20 subsamples of 80%)")
for k in range(2, 6):
full = kmedoids(D, k)
st = []
for _ in range(20):
idx = rng.choice(len(D), int(0.8 * len(D)), replace=False)
st.append(adjusted_rand_score(full[idx], kmedoids(D[np.ix_(idx, idx)], k)))
print(k, f"{silhouette_score(D, full, metric='precomputed'):.3f} {np.mean(st):.2f}")
Output:
reference: ARI of random labels vs the hidden segments 0.0 | of a perfect match 1.0
adjusted Rand index against the hidden segments: mean over 10 seeds (min..max)
A raw units + one-hot, k-means 0.29 (0.27..0.32)
B standardised + frequency code, k-means 0.62 (0.39..0.82)
C Gower + k-medoids 0.51 (0.47..0.57)
D Gower + k-medoids, city dropped 0.51 (0.47..0.58)
E standardised, city dropped, k-means 0.83 (0.81..0.86)
F age, income, tier range-scaled, k-means 0.30 (0.23..0.34)
G age, income, tier standardised, k-means 0.77 (0.74..0.79)
Gower + k-medoids with different column sets: mean ARI over 10 seeds (min..max)
age, income 0.81 (0.77..0.86)
age, income, tier 0.31 (0.23..0.38)
age, income, channel 0.50 (0.48..0.56)
age, income, tier, channel 0.51 (0.47..0.58)
all five columns (row C) 0.51 (0.47..0.57)
all five + region 0.60 (0.55..0.64)
Gower by hand, records 0 and 1: {'age': 0.333, 'log income': 0.032, 'tier': 0.0, 'channel': 0.0, 'city': 1.0} -> average 0.273 | gower() gives 0.273
pairwise distance matrix size: n=600 -> 2.88 MB as float64; n=100000 -> 80.0 GB
mean share of records in the record's own city, by hidden segment: [0.0195, 0.0186, 0.018]
silhouette of the TRUE segments on the Gower matrix: 0.242
share of record pairs with equal value: channel 0.27 city 0.019
k silhouette(Gower) stability (ARI over 20 subsamples of 80%)
2 0.299 0.89
3 0.318 0.97
4 0.355 0.87
5 0.303 0.73
Reading it:
- How to read ARI here. The first printed line anchors the scale: random labels score 0.0 and an exact match scores 1.0. The mean over 10 seeds is reported with its min and max.
- Scaling and typing matter most. Raw units with one-hot reach a mean ARI of 0.29 because income in dollars swamps every other column; the prepared pipelines B, C, D and E all beat it by a wide margin. Standardising with no city column (E) is best at 0.83 (0.81 to 0.86), and the scaling rows F and G below show how much of the difference is scaling. Gower with k-medoids (C) is 0.51, with a narrow spread (0.47 to 0.57). Standardising and adding a frequency-coded city (B) falls to 0.62 on average and varies a lot across seeds (0.39 to 0.82).
- Why B is worse than E. The only difference between them is the frequency-coded city column. The printed shares show that records of every hidden segment live in cities of almost the same size, so how common a city is tells you nothing about the segment; the column is noise that gets the same weight as age. That is the "popularity, not identity" problem in numbers: 0.83 without the column, 0.62 with it. Gower with the 0/1 city mismatch behaves like another uninformative column, but dropping it changes nothing for Gower (C and D are both 0.51).
- Gower scores lower than E here, and the cause is the tier column under range scaling with equal weights. Rows F and G use only age, log income and tier and differ only in how they are scaled before the same k-means: dividing by the range gives 0.30 (0.23 to 0.34), standardising gives 0.77 (0.74 to 0.79). Gower divides each numeric difference by the column's range, which is set by its extreme values, not by how spread out the segments are, and a three-level tier moves by 0.5 per level while the age gap between segments is a small fraction of the age range. The column-set table shows the effect directly: Gower with age and income alone reaches 0.81 (0.77 to 0.86), about the level of E, and adding the noisy tier column drops it to 0.31 (0.23 to 0.38). Adding the channel mismatch on top of age, income and tier lifts it back to 0.51, but channel is no free gain either: age, income and channel without tier gives 0.50, below age and income alone, because a full-weight 0/1 mismatch on a column that agrees only about 70% of the time inside a segment outweighs the small numeric differences. So with equal weights, every added column with a large per-column gap pulls the average distance away from the two columns that carry most of the signal. This is a property of this generator and of equal column weights with range scaling, which is why the order above is to run both families and compare, not a claim that one always wins. Down-weighting columns (the weights are a choice, not a fact) and a standard-deviation scaling in place of the range are the Gower variants worth trying.
- The city column barely moved Gower (C and D both 0.51). Only 1.9% of record pairs share a city, so under a 0/1 mismatch the column gives almost every pair the same penalty and carries little signal. Coarsening cities into regions is what lets it help: adding a region column (the block of 20 cities tied to a segment) to the five Gower columns lifts C from 0.51 to 0.60 (0.55 to 0.64), though still short of the 0.81 of age and income alone. An embedding is the other route.
- Validation without labels, and what values mean. On this draw (seed 0) silhouette is highest at k=4 (0.355) while stability is highest at k=3 (0.97, against 0.87 at 4). The hidden truth here is 3, and the two indicators disagree like this because they measure different things: silhouette rewards compact, separated groups, stability rewards groups that survive resampling. For calibration, the silhouette of the hidden true segments themselves on this Gower matrix is 0.242 (printed line before the table), so a silhouette near 0.3 means these segments are about as well separated as the truth is, not a poor result; silhouette values depend on the data, so compare them across settings on the same data rather than against a universal cutoff. A stability of 0.97 means that re-clustering 80% of the records returns almost the same groups (1 would be identical). Neither number is decisive on its own: use them together, then check that the segments are describable (profile each one on the original columns) and actionable.
- Cost of the full matrix. The printed sizes are 2.88 MB for 600 records and 80.0 GB for 100000, which is why Gower plus k-medoids does not scale and the encode-then-k-means pipeline is the fallback for large n.
Validating the result in practice
- Stability across subsamples and across reasonable alternatives (a different scaling, a different feature weight, a different algorithm). Segments that appear only under one setting are artefacts.
- Internal measures computed with the same distance used to cluster (silhouette on the Gower matrix, as above).
- Profile each cluster on the original columns, with a table of means and category shares, and check that the differences are large enough to name.
- Downstream use: does the segment predict something that matters (response to an offer, churn rate) on a holdout period, and can new records be assigned by nearest medoid or centroid?
- Sensitivity to column weights, because equal weights are a choice, not a fact.
Common mistakes
- One-hot encoding a 1,000-level column and clustering with Euclidean distance, so the block drowns out everything else.
- Using the raw integer codes of a nominal column (1 = red, 2 = blue, 3 = green) as numbers.
- Using target encoding with no stated outcome, or with the outcome you later claim the segments "discovered".
- Calling the segmentation valid because silhouette is high, without a stability check or a profile a stakeholder can read.
How does the starting position of the centroids affect k-means, and what initialization strategies would you use? Explain why a bad start hurts the result and how a smarter seeding scheme helps.
Sample Answer
Direct answer
K-means only moves centroids (cluster centres, each the mean of its points) downhill from where they start, so it finds the local optimum nearest to the start, not the best one. A bad start, for example two centroids inside one natural group and none in another, gets stuck there with a high total squared distance (inertia). Smarter seeding (k-means++) picks starting centroids that are spread apart, and running several starts and keeping the lowest inertia protects against the rest.
Why a bad start hurts
Each iteration assigns points to the nearest centroid and then moves the centroid to its members' mean. A centroid never jumps across empty space to a far-away group, because points far from it are claimed by other centroids. Suppose the data has three groups and two initial centroids land in group A while one sits between B and C. The two in A split A in two, the third stays between B and C and averages them into one cluster. Every step is an improvement, no step can fix the mistake, and the loop stops at a fixed point (a state where one more round changes nothing) with higher inertia than the correct three-group solution. The result also changes from seed to seed, which is a reproducibility problem on its own.
Here is that failure with coordinates, on nine points on a line: group A at 0, 1, 2, group B at 20, 21, 22 and group C at 40, 41, 42. The bad start puts two centroids in A (at 0 and 2) and one at 31, between B and C. The good start puts one centroid in each group.
import numpy as np
# three groups on a line: A around 1, B around 21, C around 41
pts = np.array([0.0, 1.0, 2.0, 20.0, 21.0, 22.0, 40.0, 41.0, 42.0])
def run(centres):
c = np.array(centres, dtype=float)
for it in range(1, 10):
labels = np.abs(pts[:, None] - c[None, :]).argmin(axis=1) # nearest centroid
new = np.array([pts[labels == j].mean() if (labels == j).any() else c[j] for j in range(3)])
inertia = ((pts - new[labels]) ** 2).sum()
print(f" iteration {it}: centroids {np.round(new, 1).tolist()}, inertia {inertia:.1f}")
if np.allclose(new, c):
break
c = new
print("bad start: two centroids in group A, one between B and C")
run([0.0, 2.0, 31.0])
print("good start: one centroid in each group")
run([1.0, 21.0, 41.0])
Output:
bad start: two centroids in group A, one between B and C
iteration 1: centroids [0.5, 2.0, 31.0], inertia 604.5
iteration 2: centroids [0.5, 2.0, 31.0], inertia 604.5
good start: one centroid in each group
iteration 1: centroids [1.0, 21.0, 41.0], inertia 6.0
The bad start stops after one update at inertia 604.5, because the centroid at 31 is the mean of B and C together (each of those six points is about 10 away from it), while the two centroids in A split three points that were already tight (point 1 is exactly as close to both, and the tie goes to the first). Nothing in the loop can move a centroid from A across to separate B from C. The good start reaches inertia 6.0 immediately.
Initialisation strategies
| Strategy | How it picks centroids | Strength | Weakness |
|---|---|---|---|
| Random points | k distinct data points chosen uniformly | Simple, fast | Can start two centroids in one group |
| k-means++ | Spread-out data points, chosen sequentially (below) | Far fewer bad starts; has a proven guarantee: its expected inertia is within a factor that grows only like lnk of the best possible inertia | Sequential, so k passes over the data; still random, so still restart |
| Farthest-first | Always take the point farthest from the chosen ones (so each new centre is the most isolated point left) | Spreads well, and is deterministic once the first point is fixed | Picks outliers first |
Multiple restarts (n_init) | Run any of the above several times, keep the lowest inertia | Works with any init; the restarts do not depend on each other, so they can run side by side on separate cores | Multiplies the cost by the number of restarts |
| Informed seeds | Centroids from a domain rule or from a hierarchical clustering of a sample (a method that starts with every point alone and repeatedly merges the two closest groups) | Meaningful, reproducible | Needs prior knowledge or an extra step |
k-means++ in implementable steps
- Choose the first centroid uniformly at random from the data.
- For every point, compute D(x)2: the squared distance to its nearest already chosen centroid.
- Choose the next centroid from the data with probability D(x)2/∑x′D(x′)2. Far points are much more likely, points already close to a centroid almost never.
- Repeat steps 2 and 3 until k centroids exist, then run ordinary k-means.
In practice, k-means++ followed by several restarts is the standard default (k-means++ is scikit-learn's default init); farthest-first and informed seeds are specialised choices. Squaring matters: it makes the chance grow faster than distance, so a point twice as far is four times as likely.
Worked example, and measured effect
The program prints the hand-sized step, then clusters 600 points from six blobs 100 times with a single start per seed, once with init="random" and once with init="k-means++" (scikit-learn 1.5.2), and compares each run's inertia with the best found in 50 restarts.
import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
# 1) One k-means++ seeding step by hand on six points on a line
pts = np.array([0.0, 1.0, 2.0, 10.0, 11.0, 20.0])
first = 0.0 # suppose the first centre is the point at 0
d2 = (pts - first) ** 2 # squared distance to the nearest chosen centre
prob = d2 / d2.sum()
print("points :", pts.tolist())
print("D^2 :", d2.tolist())
print("probability :", np.round(prob, 3).tolist())
# 2) Many single starts: random vs k-means++ on six blobs, k = 6
X, _ = make_blobs(n_samples=600, centers=6, cluster_std=0.9, random_state=3)
best = KMeans(6, n_init=50, random_state=0).fit(X).inertia_
print("best inertia found with 50 restarts:", round(best, 1))
for init in ["random", "k-means++"]:
inertias = np.array([KMeans(6, init=init, n_init=1, random_state=s).fit(X).inertia_
for s in range(100)])
near_best = (inertias <= 1.01 * best).mean()
print(f"{init:9s} n_init=1 over 100 seeds: share within 1% of best = {near_best:.2f}, "
f"median = {np.median(inertias):.1f}, worst = {inertias.max():.1f}")
Output:
points : [0.0, 1.0, 2.0, 10.0, 11.0, 20.0]
D^2 : [0.0, 1.0, 4.0, 100.0, 121.0, 400.0]
probability : [0.0, 0.002, 0.006, 0.16, 0.193, 0.639]
best inertia found with 50 restarts: 866.8
random n_init=1 over 100 seeds: share within 1% of best = 0.27, median = 935.6, worst = 2760.6
k-means++ n_init=1 over 100 seeds: share within 1% of best = 0.45, median = 884.4, worst = 948.4
On the six points, squared distances are 0, 1, 4, 100, 121 and 400, giving selection probabilities 0, 0.002, 0.006, 0.16, 0.193 and 0.639: the second centroid is most likely the point at 20, and almost never one near 0. On the six-blob data, with one start each, random initialisation landed within 1% of the best inertia in 27 of 100 seeds, k-means++ in 45 of 100. The median inertia was 935.6 for random and 884.4 for k-means++, and the worst run was 2760.6 against 948.4. So on this dataset k-means++ shrinks the damage of an unlucky start a lot (worst case) but does not remove it (only 45 of 100 near best), which is why restarts remain standard. These numbers come from one dataset. On other datasets of the same kind k-means++ puts more of its single starts near the best inertia, but how much the worst run improves varies from a large gap to almost none, so take away the direction (more good starts, fewer terrible ones, never zero bad ones), not the exact shares.
Pitfalls
- k-means++ spreads centroids by distance, so with outliers it can seed on an outlier. Remove or down-weight extreme points first.
- Seeding quality does not fix a wrong k or unscaled features.
- Report the seed and number of restarts so the segmentation is reproducible.
When would you use density-based clustering instead of k-means? Explain how it decides what is a cluster and what is noise, and what its settings control.
Sample Answer
Use density-based clustering (DBSCAN, density-based spatial clustering of applications with noise) when you do not know how many clusters there are, when the groups are not round blobs, or when some customers belong to no group at all. DBSCAN calls a cluster a crowded region and noise a lonely point. Its two settings are a radius (eps) that defines "near" and a count (min_samples) that defines "crowded".
How it decides
- For every point, count the points within distance
eps, including the point itself (the convention scikit-learn uses, which the code below confirms by recomputing the core points by hand and comparing). - A point with at least
min_samplespoints in that count is a core point: it sits in a dense region. - Two core points within
epsof each other are in the same cluster; clusters grow by chaining through core points, so a cluster can be any shape, such as a crescent or a ring. - A non-core point within
epsof a core point is a border point and joins that cluster. - Every other point is noise and gets the label -1. It is not forced into the nearest cluster.
There is no k to choose, and no centroid.
The rules on seven points you can check by hand
Seven customers on a single line, described by annual spend in thousands of dollars. With eps 1.0 and min_samples 3, count the points within 1.0 of each one (itself included). Points 1.6 and 2.2 each have three, so they are core points. Point 1.0 has only two (1.0 and 1.6; 2.2 is 1.2 away), so it is not core, but it lies within 1.0 of the core point 1.6, which makes it a border point of that cluster. Point 2.9 has two (2.2 and 2.9) and lies within 1.0 of core point 2.2, so it is a border point too. The two core points are 0.6 apart, so they are in one cluster, and 1.0 and 2.9 (1.9 apart) end up in the same cluster only by chaining through them. Points 4.4 and 5.0 have two neighbours each and no core point within 1.0, and 9.0 has one: all three are noise. The code reproduces the counts and checks the labels against scikit-learn, then prints the distances that eps is read from.
import numpy as np
from sklearn.cluster import DBSCAN
from sklearn.datasets import make_moons
from sklearn.neighbors import NearestNeighbors
# seven customers on one line (annual spend in thousands of dollars), eps = 1.0, min_samples = 3
x = np.array([1.0, 1.6, 2.2, 2.9, 4.4, 5.0, 9.0])
eps, ms = 1.0, 3
dist = np.abs(x[:, None] - x[None, :])
count = (dist <= eps).sum(axis=1) # neighbours within eps, the point itself included
core = count >= ms
print("point value neighbours-within-eps core?")
for i in range(len(x)):
print(f"{i:5d} {x[i]:6.1f} {count[i]:21d} {core[i]}")
db = DBSCAN(eps=eps, min_samples=ms).fit(x[:, None])
print("scikit-learn labels:", db.labels_.tolist())
print("core flags match scikit-learn:", bool((np.isin(np.arange(len(x)), db.core_sample_indices_) == core).all()))
# k-distance values on the crescent data: distance from each point to its 5th nearest neighbour (itself counts as the 1st)
X, y = make_moons(n_samples=400, noise=0.06, random_state=0)
rng = np.random.default_rng(0)
X = np.vstack([X, rng.uniform([-1.5, -1.0], [2.5, 1.5], size=(30, 2))])
kd = np.sort(NearestNeighbors(n_neighbors=5).fit(X).kneighbors(X)[0][:, -1])
print("\n5th-neighbour distance at these percentiles of the sorted list:")
for p in (50, 80, 90, 95, 100):
print(f" {p:3d}% {np.percentile(kd, p):.3f}")
Output:
point value neighbours-within-eps core?
0 1.0 2 False
1 1.6 3 True
2 2.2 3 True
3 2.9 2 False
4 4.4 2 False
5 5.0 2 False
6 9.0 1 False
scikit-learn labels: [0, 0, 0, 0, -1, -1, -1]
core flags match scikit-learn: True
5th-neighbour distance at these percentiles of the sorted list:
50% 0.067
80% 0.094
90% 0.122
95% 0.161
100% 1.102
Reading it: the first table is the counts above (core only where the count reaches 3). The labels [0, 0, 0, 0, -1, -1, -1] put the four chained points in cluster 0 and mark the last three as noise. In the second block, the 5th-neighbour distance is below about 0.16 for 95% of the crescent data and then climbs to 1.10 at the far end, where the scattered points are, so the bend of the sorted curve sits near 0.15, which is the eps used in the main example. Plotted, the same numbers are the k-distance plot.
A computed example on crescents
Two crescent-shaped customer groups (400 points) plus 30 scattered points that belong to neither. k-means with k=2 and DBSCAN are scored by the adjusted Rand index (ARI; 1.0 means the grouping matches the truth, about 0 means chance) on the 400 crescent points. A second table varies the settings.
import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score
# two crescent-shaped customer groups plus 30 scattered points that belong to no group
X, y = make_moons(n_samples=400, noise=0.06, random_state=0)
rng = np.random.default_rng(0)
scatter = rng.uniform([-1.5, -1.0], [2.5, 1.5], size=(30, 2))
X = np.vstack([X, scatter])
y = np.r_[y, np.full(30, -1)] # -1 marks the scattered points
real = y >= 0
km = KMeans(n_clusters=2, n_init=10, random_state=0).fit(X)
print("k-means k=2: ARI on the 400 crescent points", round(adjusted_rand_score(y[real], km.labels_[real]), 3))
# core-point rule checked by hand: a point is core if at least min_samples points (itself included) lie within eps
eps, ms = 0.15, 5
db = DBSCAN(eps=eps, min_samples=ms).fit(X)
dist = np.linalg.norm(X[:, None] - X[None], axis=2)
core_by_hand = (dist <= eps).sum(axis=1) >= ms
core_sklearn = np.zeros(len(X), bool); core_sklearn[db.core_sample_indices_] = True
print("core points by hand match scikit-learn:", bool((core_by_hand == core_sklearn).all()), "| core points:", int(core_by_hand.sum()))
lab = db.labels_
print(f"DBSCAN eps={eps} min_samples={ms}: clusters {len(set(lab) - {-1})}, noise {int((lab == -1).sum())}, "
f"border {int(((lab != -1) & ~core_sklearn).sum())}")
print(" ARI on the 400 crescent points (noise counted as its own label):", round(adjusted_rand_score(y[real], lab[real]), 3))
print(" scattered points labelled noise:", int(((lab == -1) & ~real).sum()), "of 30")
print("\neps min_samples clusters noise points")
for e, m in [(0.05, 5), (0.10, 5), (0.15, 5), (0.30, 5), (0.60, 5), (0.15, 20)]:
l = DBSCAN(eps=e, min_samples=m).fit(X).labels_
print(f"{e:<5} {m:<12d} {len(set(l) - {-1}):<9d} {int((l == -1).sum())}")
Output:
k-means k=2: ARI on the 400 crescent points 0.248
core points by hand match scikit-learn: True | core points: 405
DBSCAN eps=0.15 min_samples=5: clusters 2, noise 18, border 7
ARI on the 400 crescent points (noise counted as its own label): 1.0
scattered points labelled noise: 18 of 30
eps min_samples clusters noise points
0.05 5 28 245
0.1 5 4 33
0.15 5 2 18
0.3 5 1 10
0.6 5 1 4
0.15 20 8 232
Reading it:
- k-means draws a straight boundary between two round regions and cuts both crescents (ARI 0.248). DBSCAN at
eps0.15 andmin_samples5 finds the two crescents with ARI 1.0, and labels 18 of the 30 scattered points as noise. The other 12 scattered points landed within reach of a crescent and were absorbed into it, so the noise label is not a complete outlier detector. - The settings behave as follows.
epstoo small (0.05): the crescents shatter into 28 pieces and 245 points are noise.epstoo large (0.30 here, and 0.60 in every draw of the data tried): the crescents merge into one cluster; with other random draws of the data the merge sometimes starts only at larger values, so where it starts depends on the data.min_samplesraised to 20 at the sameeps: 8 clusters and 232 noise points, because few places are crowded enough. Soepssets how far apart points may be and still be neighbours, andmin_samplessets how crowded a place must be to count as a cluster core; a largermin_samplesor a smallerepsproduces more noise.
Choosing the settings
eps: sort every point's distance to itsmin_samples-th nearest neighbour and plot it (a k-distance plot, as printed above); points inside crowded places have small values and isolated points have large ones, so the bend in the curve where values start to jump marks the distance at which "crowded" turns into "lonely", a candidate foreps. Features must be scaled first, becauseepsis one distance for every feature.min_samples: a common rule of thumb is to start around twice the number of features (and at least 3 or 4 for low-dimensional data), then check the stability of the result under small changes. The reasoning is that with more features a point has more directions in which a chance neighbour can sit, so a larger crowd is needed before a place counts as dense; treat it as a heuristic, not a theorem.
When k-means is the better choice
| Situation | Better choice |
|---|---|
| Round, similar-size groups, every customer must belong to a segment, a fixed number of segments is required | k-means |
| Crescents, rings, chains, unknown number of groups, noise should stay out | DBSCAN |
| Groups of very different density (a tight cluster beside a diffuse one) | A density method with one eps struggles: a radius that suits the diffuse group merges the tight ones. Two follow-on options, rarely needed for a first segmentation: HDBSCAN (runs DBSCAN at many radii and keeps the clusters that persist, so there is no single radius) or a Gaussian mixture model (each group is a bell-shaped blob with its own spread) |
| Many features | Distances lose meaning in high dimension, so reduce the features first |
| A stakeholder wants a centroid profile per segment | k-means; for DBSCAN, profile each cluster by its mean afterward |
Common mistakes
- Running DBSCAN on unscaled features.
- Treating "noise" as "bad customers": it means "not in a dense region" and often marks interesting rare customers.
- Tuning
epsuntil the cluster count looks right and calling it discovered.
Explain how survival analysis applies to customer churn and why it is better than a plain classifier when many customers are still active. How would you estimate time to churn and relate it to customer attributes?
Sample Answer
The idea in plain words
Churn is a question about when, and the data usually only half-answer it. Most customers are still active when you pull the data. For them you know they lasted at least this long, but not how long they will last. This kind of partial information is called right-censoring: the observation is cut off before the event happens. Survival analysis is the set of methods built for exactly this data. Each customer contributes two columns: how long they were observed (their tenure) and whether churn was seen (1) or they were still active when observation ended (0, "censored").
A plain classifier needs a yes/no label per customer. Labelling every still-active customer "did not churn" teaches the model that recent joiners are loyal, because they simply have not had time to leave. Dropping them throws away most of the data and keeps only the customers who churned quickly. Survival methods use each censored customer correctly: as evidence that churn did not happen up to their last observed day.
The mechanism, in the order it happens
1. Survival curve. S(t) is the probability a customer is still subscribed at time t after joining. The Kaplan-Meier estimator builds it step by step:
S^(t)=ti≤t∏(1−nidi)where ti are the distinct times at which at least one churn occurred, di is the number of churns at ti, and ni is the number of customers still being watched just before ti (the risk set). A censored customer stays in ni until the day they drop out of observation and then leaves it without counting as a churn. That is how censoring is handled: nobody is wrongly counted as lost or as safe.
Six customers make the steps visible. Three customers churn, at months 2, 4 and 7. The other three are still active when their observation ends, at months 3, 6 and 9, so they are censored at those months.
import numpy as np
# six customers: months observed, and 1 = churn seen, 0 = still active (censored)
tenure = np.array([2, 3, 4, 6, 7, 9])
churned = np.array([1, 0, 1, 0, 1, 0])
s = 1.0
print("time at_risk churns factor(1-d/n) S(t)")
for t in np.unique(tenure[churned == 1]):
n_i = np.sum(tenure >= t) # still being watched just before t
d_i = np.sum((tenure == t) & (churned == 1))
s *= 1 - d_i / n_i
print("%4d %7d %6d %13.3f %.3f" % (t, n_i, d_i, 1 - d_i / n_i, s))
# naive: call every customer who has not churned a survivor, whatever their tenure
print("naive share surviving past month 7: %.3f" % (1 - np.mean((churned == 1) & (tenure <= 7))))
time at_risk churns factor(1-d/n) S(t)
2 6 1 0.833 0.833
4 4 1 0.750 0.625
7 2 1 0.500 0.312
naive share surviving past month 7: 0.500
Read each row left to right. At month 2 all six are being watched and one leaves, so the factor is 5/6 = 0.833. The customer censored at month 3 then leaves the risk set without being counted as a churn, so at month 4 only four remain and the factor is 3/4. Multiplying the factors gives 0.833×0.750=0.625, then 0.625×0.5=0.312 at month 7, where two customers remain. The naive count (one minus the share who churned by month 7, treating the censored customers as survivors) says 0.500, because it counts the customers censored at months 3 and 6 as having survived to month 7 when they were only watched for 3 and 6 months. The same mistake is what the large simulation below measures.
2. Comparing groups. Plot one Kaplan-Meier curve per group (annual versus monthly plan, acquisition channel) and test whether they differ with the log-rank test: at each churn time it compares the churns seen in a group with the number expected if both groups had the same risk, and combines the differences into a chi-squared statistic. If the two groups truly had the same risk, that statistic would average 1 (its average when there is 1 degree of freedom, which here means one comparison between two groups), and values above 3.84 occur by chance only 5% of the time. A large value therefore means the observed churns differ from the equal-risk expectation by far more than chance, and the p-value is the chance of a gap that large if the groups were equal.
3. Relating time to attributes. The Cox proportional hazards model models the hazard, the instantaneous churn rate at time t among customers still active:
h(t∣x)=h0(t)exp(β⊤x)Here h0(t) is a baseline hazard that is never specified, x is the vector of customer attributes and β the coefficients. exp(βj) is the hazard ratio: how much the churn rate is multiplied by when attribute j rises by one unit, holding the others fixed. The coefficients are estimated by maximising the partial likelihood, which at each churn time asks how likely it was that that customer, among those at risk, was the one to leave. The baseline h0(t) cancels out of that comparison, which is why the model can estimate β without ever specifying the shape of the baseline.
4. Time to churn. Read it from the curve: the median survival time is the first time S^(t)≤0.5 (undefined if the curve never reaches 0.5, which is common in a young cohort; report survival at 6 or 12 months instead). A Cox model gives a curve per customer profile as well.
A worked example you can run
Synthetic data: 4,000 customers join at random times over a 24-month window and are watched until the end of that window. Each has an annual-plan flag and a standardised ticket count; the true hazard ratios are e−0.7=0.50 for the annual plan and e0.4=1.49 per standard deviation of tickets. The simulation also keeps each customer's full true lifetime as an oracle, which real data never has. The program implements Kaplan-Meier, the log-rank test and the Cox partial likelihood in numpy and scipy; its assert checks that, with no censoring, Kaplan-Meier equals the plain empirical survival fraction.
import numpy as np
from scipy.optimize import minimize
from scipy.stats import chi2
rng = np.random.default_rng(0)
n = 4000
annual = (rng.random(n) < 0.4).astype(float) # 1 = annual plan
tickets = rng.normal(size=n) # standardised support tickets
rate = 0.05 * np.exp(-0.7 * annual + 0.4 * tickets) # churn hazard per month (constant over time per customer)
T = rng.exponential(1 / rate) # true months until churn (oracle, never observable in full)
join = rng.uniform(0, 24, n) # joined at a random month in a 24-month window
C = 24 - join # months we can watch each customer
time = np.minimum(T, C)
event = (T <= C).astype(int) # 1 = churn seen, 0 = censored (still active)
print("customers %d, churn observed for %.0f%%, censored (still active) %.0f%%" % (n, 100 * event.mean(), 100 * (1 - event.mean())))
def km(time, event):
"""Kaplan-Meier: returns event times and S(t) = product over event times of (1 - d_i / n_i)."""
ts = np.unique(time[event == 1])
s, out = 1.0, []
for t in ts:
n_i = np.sum(time >= t)
d_i = np.sum((time == t) & (event == 1))
s *= 1 - d_i / n_i
out.append(s)
return ts, np.array(out)
def surv_at(ts, s, t):
k = np.searchsorted(ts, t, side="right")
return 1.0 if k == 0 else s[k - 1]
# sanity check that can fail: with no censoring KM must equal the plain empirical survival fraction
ts0, s0 = km(T[:500], np.ones(500, int))
assert abs(surv_at(ts0, s0, 12) - np.mean(T[:500] > 12)) < 1e-12
ts, s = km(time, event)
oracle = np.mean(T > 12)
naive_active = 1 - np.mean((event == 1) & (time <= 12)) # treats every non-churner as a survivor
print("share still subscribed at 12 months:")
print(" oracle (uncensored lifetimes) %.3f" % oracle)
print(" Kaplan-Meier on censored data %.3f" % surv_at(ts, s, 12))
print(" naive: 1 - churned within 12 / all %.3f" % naive_active)
print("mean months watched, customers seen to churn only %.1f; true mean lifetime, all customers %.1f" % (time[event == 1].mean(), T.mean()))
# log-rank test: annual vs monthly plan
def logrank(time, event, g):
ts = np.unique(time[event == 1]); O = E = V = 0.0
for t in ts:
r = time >= t; n_all = r.sum(); n1 = (r & (g == 1)).sum()
d = ((time == t) & (event == 1)).sum(); d1 = ((time == t) & (event == 1) & (g == 1)).sum()
O += d1; E += d * n1 / n_all
if n_all > 1: V += d * (n1 / n_all) * (1 - n1 / n_all) * (n_all - d) / (n_all - 1)
return (O - E) ** 2 / V
lr = logrank(time, event, annual)
print("log-rank chi2 (1 df) annual vs monthly: %.1f, p = %.1e" % (lr, chi2.sf(lr, 1)))
# Cox model: maximise the partial likelihood (no tied times here, so no tie correction is needed)
def neg_pl(beta, X, time, event):
o = np.argsort(-time) # latest first, so cumulative sums give each risk set
eta = X[o] @ beta
log_risk = np.log(np.cumsum(np.exp(eta)))
return -np.sum((eta - log_risk)[event[o] == 1])
X = np.column_stack([annual, tickets])
fit = minimize(neg_pl, np.zeros(2), args=(X, time, event), method="BFGS")
print("Cox coefficients (true -0.70, 0.40):", np.round(fit.x, 2))
print("hazard ratios (true %.2f, %.2f): %s" % (np.exp(-0.7), np.exp(0.4), np.round(np.exp(fit.x), 2)))
boots = []
for b in range(30):
i = np.random.default_rng(100 + b).integers(0, n, n)
boots.append(minimize(neg_pl, np.zeros(2), args=(X[i], time[i], event[i]), method="BFGS").x)
lo, hi = np.percentile(np.exp(np.array(boots)), [2.5, 97.5], axis=0)
print("bootstrap 95%% interval for hazard ratios: annual %.2f-%.2f, tickets %.2f-%.2f" % (lo[0], hi[0], lo[1], hi[1]))
customers 4000, churn observed for 35%, censored (still active) 65%
share still subscribed at 12 months:
oracle (uncensored lifetimes) 0.613
Kaplan-Meier on censored data 0.620
naive: 1 - churned within 12 / all 0.705
mean months watched, customers seen to churn only 6.5; true mean lifetime, all customers 30.6
log-rank chi2 (1 df) annual vs monthly: 121.5, p = 2.9e-28
Cox coefficients (true -0.70, 0.40): [-0.65 0.44]
hazard ratios (true 0.50, 1.49): [0.52 1.55]
bootstrap 95% interval for hazard ratios: annual 0.49-0.58, tickets 1.49-1.62
- Censoring is most of the data. Churn is observed for 35% of customers; 65% are still active.
- The naive estimate is biased; Kaplan-Meier is not. The oracle share still subscribed at 12 months is 0.613. Kaplan-Meier on the censored data gives 0.620. Treating every non-churner as a survivor gives 0.705, because customers watched for under 12 months are counted as having survived 12 months. The same trap hits averages, though across different groups: the customers seen to churn were watched for 6.5 months on average, while the true mean lifetime across all customers in this simulation is 30.6 months, because most customers have not churned yet. Averaging the tenure of churners alone describes who left early, not how long customers last.
- Group comparison. The log-rank statistic for annual versus monthly is 121.5 on 1 degree of freedom, against an average of 1 and a 5% cut-off of 3.84 if the plans had equal risk. The p-value of about 3e-28 says such a gap would almost never appear by chance, so the plans differ; how much they differ is the hazard ratio below.
- Attributes. The Cox fit gives hazard ratios of 0.52 (annual plan) and 1.55 (per standard deviation of tickets) against true 0.50 and 1.49. Bootstrap intervals (30 refits on resampled customers) are 0.49-0.58 and 1.49-1.62; one simulation shows the method recovers the effects, not the long-run coverage of that interval.
Trade-offs, assumptions and failure modes
The first two items below are the ones to be ready to explain in an interview; the rest are the next layer of depth.
- Independent censoring. Censoring must not be related to the risk of churn. If customers on the verge of leaving are moved to a different system, or accounts are closed by fraud checks and logged as censored, the curves are biased.
- Proportional hazards. The Cox model assumes the ratio between two customers' hazards is constant over time. Check it by plotting log-log survival curves or testing scaled Schoenfeld residuals (a correlation of each covariate's residuals with time). When a plan has a strong effect only in the first three months, stratify by it, add a time interaction, or use an accelerated failure time or discrete-time model.
- Time-varying attributes. Usage in month 3 is only known after month 3. Put it in the model as a time-varying covariate with counting-process rows; using final usage leaks the future.
- Competing events. Involuntary churn from a failed payment differs from a voluntary cancel. Model them as separate event types.
- Association, not cause. A hazard ratio on tickets is a statement about who leaves, not what would keep them.
When a classifier is the right tool. When the decision is a fixed horizon ("will this customer cancel in the next 30 days?") and every training customer has been watched for at least that long, a classifier with a properly built label is simpler and often as good. Survival methods earn their cost when follow-up varies a lot, when the timing of churn drives the action, or when you need expected lifetime, for example as an input to lifetime value.
Common mistakes
- Labelling active customers 0 in a classifier.
- Comparing average tenure of churned versus active customers, which conflates "left early" with "joined recently".
- Reading a hazard ratio as a probability ratio.
- Reporting a median survival time from a curve that never crosses 0.5.
Unlock Full Question Bank
Get access to all 18 Predictive Modeling and Machine Learning Fundamentals interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.