Anomaly and Fraud Detection Questions

Direct answer
Real-time scoring is right when a decision must be made before the transaction completes (blocking a payment before it settles), tolerates only milliseconds of latency, and relies on features that are already precomputed or trivially cheap to compute; nearline scoring fits when a decision can wait seconds to a few minutes (triggering a hold or a targeted review shortly after a transaction, rather than blocking it outright), tolerating richer, freshly-computed features; and batch scoring fits when the decision can wait hours (surfacing accounts for proactive investigation, retraining models, or generating daily risk reports), allowing the most computationally expensive features and models.

Structured elaboration

Trade-offs and pitfalls
These three modes are not mutually exclusive alternatives to pick just one of, a mature fraud system typically runs all three together: real-time for the hard latency-bound blocking decision, nearline to catch what real-time's limited feature set missed, and batch to catch slower-forming patterns (like emerging rings) that neither faster mode has the computational budget to detect. The common mistake is trying to force one mode to do a job better suited to another, for example cramming an expensive graph computation into the real-time path instead of accepting that it belongs in a nearline or batch pass.

Direct answer
Distinguish a legitimate traffic spike from a bot or coordinated-attack spike by looking at the DIVERSITY and BEHAVIORAL REALISM of the traffic, real spikes come from many distinct devices, IPs, and session patterns behaving the way genuine users actually behave, while bot-driven spikes concentrate on a narrow set of sources with unnaturally uniform or mechanically-repeated behavior; for ad click-fraud specifically, apply the same diversity-and-realism lens to click patterns, timing, and post-click engagement, since a real user who clicks an ad usually does something afterward, and a bot usually doesn't.

Structured elaboration

• Features for distinguishing legitimate vs bot traffic spikes: source diversity (number of distinct IPs, user agents, device fingerprints behind the spike, a real spike from genuine interest is far more diverse than a scripted one), session realism (do sessions show human-like variability in timing, scroll depth, or navigation, or an unnaturally uniform pattern), and referral/context plausibility (does the spike align with an identifiable, real external event, like a marketing campaign or a news mention, versus arriving with no identifiable cause).
• Ad click-fraud detection features: click timing patterns (clicks arriving at suspiciously regular intervals, or in impossibly fast succession from the same source), post-click engagement (a real user who clicks an ad usually spends at least some time on the landing page or takes some further action; a bot typically bounces instantly with zero engagement), and concentration (a small number of sources responsible for a disproportionate share of clicks on a specific ad or campaign).
• Model choice: a combination of rule-based thresholds for the most obvious cases (near-zero post-click engagement, extreme click concentration from one source) and a supervised or unsupervised anomaly model for subtler cases, trained on labeled examples of confirmed bot/click-fraud traffic where available, mirrors the broader rules-plus-ML pattern used elsewhere on this topic.
• Avoiding false positives against real users: a genuinely viral or campaign-driven spike can look superficially similar to a coordinated attack in raw volume terms, so weight the DIVERSITY and BEHAVIORAL signals described above more heavily than volume alone, and where possible, cross-reference against a known, legitimate external cause (an active marketing campaign, a press mention) before concluding a spike is illegitimate.

Trade-offs and pitfalls
Sophisticated bot operators increasingly mimic human-like session behavior specifically to evade diversity and realism checks, so no single feature family is a permanent solution; the realistic goal, as with adversarial evasion generally, is raising the cost and sophistication required to evade detection, not eliminating bot traffic once and for all. It's also worth being conservative before blocking real users outright; a lighter-touch first response (additional verification, or excluding suspicious traffic from ad billing without blocking the user experience) is often more appropriate than an immediate hard block.

Direct answer
Three commonly engineered feature types for transactional fraud detection are short-term velocity (how much and how often activity is happening right now), long-term historical ratio (how far current behavior deviates from that account's own normal pattern), and device or network signals (what hardware and connection characteristics are behind the transaction).

Structured elaboration

• Short-term velocity: for example, the count of transactions or total dollar amount for a user in the last 10 minutes, hour, or 24 hours. Computed as a rolling window aggregate over recent transaction timestamps. Predictive because a sudden burst of activity, especially across many destination accounts, is one of the most consistent fraud signals: legitimate customers rarely transact many times in a short window.
• Long-term historical ratio: for example, today's transaction amount divided by that user's own trailing 90-day average amount. Computed by comparing a current value against a per-user baseline built from historical data, rather than against a global population baseline. Predictive because it's personalized: a $500 purchase is unremarkable for a high-spending user and highly unusual for one who has never spent more than $50, and a ratio-to-self captures that directly.
• Device or network features: for example, whether the device fingerprint or IP address has been seen before on this account, or how many distinct accounts share the same device or IP recently. Computed by joining transaction events against a device/session log and aggregating occurrence counts. Predictive because fraud is disproportionately conducted from new, shared, or previously-flagged devices and networks, since an attacker typically doesn't have access to the legitimate customer's own hardware.

Trade-offs and pitfalls
Velocity and ratio features are only as good as the time window they're computed over: too short a window is noisy, too long a window is slow to react to a genuinely new fraud pattern. Device and network features raise real privacy considerations and can also be gamed once fraudsters learn a device fingerprint is being tracked, which is one reason production systems combine several of these feature families rather than relying on any single one.

Direct answer
Compute the rolling 24-hour aggregates (transaction count, total amount, distinct destination count) per user with a time-windowed join against the transaction stream, reading the daily Parquet files as a Spark DataFrame, then write the results to a low-latency key-value store (or a small, indexed table) keyed by user_id so a serving system can look them up with a single point read rather than recomputing the window on demand.

Structured elaboration

import pandas as pd
from pyspark.sql import functions as F

df = spark.read.parquet("s3://transactions/daily/*")
window_start = F.current_timestamp() - F.expr("INTERVAL 24 HOURS")

recent = df.filter(F.col("occurred_at") >= window_start)
aggregates = (
    recent.groupBy("user_id")
    .agg(
        F.count("*").alias("txn_count"),
        F.sum("amount").alias("total_amount"),
        F.countDistinct("dest_account").alias("unique_dest_count"),
    )
)
(aggregates.write.mode("overwrite")
    .format("...")  # e.g. a Redis/DynamoDB connector, or a partitioned table keyed by user_id
    .save("features/rolling_24h_by_user"))

This PySpark job could not be executed directly in this environment, since no Java runtime is available in this sandbox to run the Java Virtual Machine (JVM) PySpark depends on. To avoid shipping an unverified number, the underlying AGGREGATION LOGIC (the same group-by-and-window computation the Spark job above performs) was verified with an equivalent pandas implementation over representative data:

def rolling_24h_aggregates(df, as_of):
    window_start = as_of - pd.Timedelta(hours=24)
    window = df[(df['occurred_at'] > window_start) & (df['occurred_at'] <= as_of)]
    return window.groupby('user_id').agg(
        txn_count=('amount', 'size'),
        total_amount=('amount', 'sum'),
        unique_dest_count=('dest_account', 'nunique'),
    )

Executed result, against 6 transactions across 2 users with one transaction for user 2 deliberately placed just outside the 24-hour window:

         txn_count  total_amount  unique_dest_count
user_id
1                4         545.0                  3
2                1          30.0                  1

User 1's 4 in-window transactions to 3 distinct destinations correctly summed to 545.0; user 2's older, out-of-window transaction was correctly excluded from the count and sum, leaving only the single in-window transaction. This confirms the windowing and aggregation logic is correct; the PySpark version above applies the identical logic at scale, distributed across a cluster instead of a single pandas DataFrame.

Scheduling and low-latency storage

• Scheduling: run this as a recurring batch job (for example every 15 to 60 minutes, depending on how fresh the feature needs to be) rather than continuously, since a 24-hour rolling window changes slowly enough that near-real-time freshness is rarely worth the added infrastructure complexity of a fully streaming version.
• Storage for low-latency lookups: write the per-user results to a key-value store (Redis, DynamoDB, or similar) keyed by user_id, since the serving path needs a single fast point lookup at scoring time, not a query over the underlying transaction data.

Trade-offs and pitfalls
A batch-recomputed 24-hour window is always slightly stale between runs; if a use case genuinely needs sub-minute freshness, this batch approach should be paired with (not replaced by) a lightweight streaming increment that adjusts the batch aggregate with the handful of transactions that arrived since the last run, rather than rebuilding the entire window from scratch on every request.

Direct answer
Build a graph from transaction or account data by treating accounts, devices, payment instruments, and other shared identifiers as nodes and their observed interactions or shared attributes as edges, then compute graph-level features like connected-component size, centrality measures such as PageRank, and community-detection scores to surface clusters of accounts that behave as a coordinated group rather than independently; scale this to millions of accounts using a distributed graph-processing framework and incremental updates rather than recomputing the full graph from scratch, and keep detecting evolving rings by re-running community detection on a rolling window rather than treating the graph as static.

Structured elaboration

• Graph construction: nodes represent accounts (and optionally devices, IP addresses, or payment instruments as their own node types in a heterogeneous graph); edges represent observed relationships, a shared device, a shared payment instrument, a transaction between two accounts, or a shared IP within a short time window. The specific edge definitions matter a great deal: too permissive (any shared attribute at all) creates a dense, uninformative graph; too strict misses real coordination.
• Graph-level features: connected-component size flags clusters of accounts linked through chains of shared attributes; centrality measures like PageRank highlight accounts that sit at the center of many suspicious connections, often the controlling account in a ring; community-detection algorithms (like Louvain) group accounts into densely-interconnected clusters that behave more like a coordinated group than a set of independent legitimate users.
• Scaling to millions of accounts: use a distributed graph-processing framework (for example, a graph database or a distributed graph-analytics engine) rather than an in-memory single-machine graph library once the account count is large; more importantly, update the graph and its computed features incrementally as new transactions arrive rather than recomputing centrality or community structure over the entire graph from scratch on every update, since full recomputation does not scale at this volume.
• Detecting evolving rings: rings adapt by adding new accounts, dropping compromised ones, or shifting shared attributes; re-run community detection on a rolling recent window (not the entire historical graph) so newly-forming clusters are caught quickly, and track how a specific community's membership and centroid change over consecutive windows as a signal that a ring is actively evolving rather than static.

Trade-offs and pitfalls
Graph-based signals are powerful for genuinely coordinated behavior but can produce false positives on innocuous shared attributes, like a shared IP address at a large office or university network, or a shared device at a public kiosk, so combine graph features with account-level context (how established the account is, whether it has legitimate transaction history) rather than treating graph membership alone as sufficient evidence. It's also worth reducing false positives specifically by weighting edges by how unusual the shared attribute is (a rare shared device fingerprint is far more suspicious than a common shared IP range), rather than treating all edges as equally informative.