Approach (heuristics): combine multiple weak signals into a score rather than a single rule. Use features like: event-rate per user/ip, burstiness (many events in short window), missing or generic user_agent, identical event_time across many events, many distinct user_ids from same ip, uncommon event_type distribution, impossible travel (same user from distant IPs in short time), repeated identical payloads.Example SQL to compute features per user_id and ip_address (Postgres syntax):sql
WITH base AS (
SELECT
event_id, user_id, ip_address, user_agent, event_time, event_type,
DATE_TRUNC('minute', event_time) AS minute_bucket
FROM events
WHERE event_time >= now() - interval '7 days'
),
agg AS (
SELECT
user_id,
ip_address,
COUNT(*) AS events_last_7d,
COUNT(DISTINCT minute_bucket) AS active_minutes,
SUM(CASE WHEN user_agent IS NULL OR user_agent = '' THEN 1 ELSE 0 END) AS missing_ua,
MAX(events_per_min) AS peak_events_per_min,
COUNT(DISTINCT user_id) OVER (PARTITION BY ip_address) AS distinct_users_on_ip
FROM (
SELECT *, COUNT(*) OVER (PARTITION BY user_id, DATE_TRUNC('minute', event_time)) AS events_per_min
FROM base
) t
GROUP BY user_id, ip_address
)
SELECT *,
(events_last_7d::numeric / NULLIF(active_minutes,1)) AS avg_events_per_min_active,
(missing_ua::numeric / events_last_7d) AS frac_missing_ua
FROM agg
ORDER BY events_last_7d DESC
LIMIT 100;
Simple filter example (flag likely bots):- avg_events_per_min_active > 10- OR peak_events_per_min >= 50- OR frac_missing_ua > 0.5- OR distinct_users_on_ip > 20- OR many events sharing the exact event_time (same second) — detect by COUNT(*) GROUP BY event_time > thresholdFalse positive mitigation:- Combine signals into a weighted score and set threshold after validation.- Whitelist known crawlers (trusted user_agents, verified IP ranges).- Exclude service accounts or internal IPs.- Use smoothing (require behaviors persist for >1 day) to avoid flagging legitimate spikes.- Inspect distributions and set thresholds at extreme percentiles (e.g., 99.9th).Validation with labeled samples:- Assemble ground truth: manual review, server access logs, CAPTCHA results, or known bot lists.- Split into train/validation. Compute precision, recall, ROC/AUC for different score thresholds.- Use confusion matrix to tune weights to maximize precision if business cares about low false positives.- Run A/B: apply detection to a subset and measure downstream impact (e.g., conversion rates, support tickets).- Continuously retrain/adjust as bot behavior evolves and monitor drift metrics.Key metrics to report: precision at chosen threshold, recall, false positive rate, coverage (fraction of total events flagged), and business impact (reduced fraud, improved analytics accuracy).