Approach: use the Misra–Gries (aka Frequent) algorithm to track top-k frequent items with limited memory. It maintains up to k counters; each stream element is processed in O(1) time. Misra–Gries guarantees that for any returned item i, estimated_count(i) ≤ true_count(i) and true_count(i) ≥ estimated_count(i) − n/(k+1), where n is total items seen. (If you need probabilistic bounds and point queries with overestimates, Count–Min Sketch is an alternative.)Java implementation (simple, single-threaded):java
import java.util.*;
public class MisraGriesTopK<T> {
private final int k;
private final Map<T, Integer> counters = new HashMap<>();
private long total = 0;
public MisraGriesTopK(int k) {
if (k < 1) throw new IllegalArgumentException("k>=1");
this.k = k;
}
// Process one stream item
public void add(T item) {
total++;
if (counters.containsKey(item)) {
counters.put(item, counters.get(item) + 1);
return;
}
if (counters.size() < k) {
counters.put(item, 1);
return;
}
// decrement all counters by 1 and remove zeros
List<T> toRemove = null;
for (Iterator<Map.Entry<T, Integer>> it = counters.entrySet().iterator(); it.hasNext();) {
Map.Entry<T, Integer> e = it.next();
int v = e.getValue() - 1;
if (v == 0) it.remove();
else e.setValue(v);
}
}
// Get candidate top-k with their current counters (lower bounds)
public Map<T, Integer> getCandidates() {
return new HashMap<>(counters);
}
public long totalSeen() { return total; }
}
Why this works:- Each update is O(1) amortized (hash operations + at most k decrements if implemented naively).- Space: O(k) counters.- Guarantee: For any item x, estimatedCount(x) ≤ trueCount(x), and trueCount(x) − estimatedCount(x) ≤ total/(k+1). So heavy hitters (frequency >> total/(k+1)) will appear in candidates.Complexity:- Time per update: O(1) average (HashMap get/put) plus O(k) for the decrement step if implemented as above; optimized implementations maintain a shared decrement offset to make per-update strictly O(1).- Space: O(k).Error bounds and failure modes:- Misra–Gries underestimates counts; an item’s estimated count can be lower by up to n/(k+1). If many items have similar frequencies near the error threshold, algorithm may miss some true top-k.- If k too small relative to distinct items or required accuracy, results will be poor.- For distributed streams, you must merge summaries: counters can be merged by summing and reapplying MG trimming.- Count–Min Sketch alternative: uses w x d table; point query overestimates by ≤ εN with probability 1−δ (w=⌈e/ε⌉, d=⌈ln(1/δ)⌉). CMS gives provable probabilistic bounds and smaller memory for some settings, but lacks exact decrements and can overcount due to collisions.When to choose which:- Misra–Gries: best when you need actual candidate identities and deterministic L1-style guarantee, small k.- Count–Min Sketch: best when you need compact point estimates with tunable ε,δ and can tolerate overestimates.