Python and Pandas for Data Analysis Questions

Direct answer

Localize event timestamps to a consistent timezone, bucket by ISO week (the ISO 8601 standard week-numbering scheme, weeks run Monday to Sunday and can span a year boundary), count distinct users per week for WAU (weekly active users), then fill any week with zero events before computing week-over-week percent change so a silent gap does not get skipped over.

Approach

1. Parse event_time; if naive, localize it to a chosen timezone (UTC unless you have a reason to use local time); if already timezone-aware, convert to that timezone.
2. Derive (iso_year, iso_week) with .dt.isocalendar(), which correctly handles ISO week 52/53 and year-boundary weeks.
3. Count distinct user_id per (iso_year, iso_week) for WAU.
4. Build the full continuous range of ISO weeks between the first and last observed week and left-fill any missing week with wau = 0, so "no events that week" is visible rather than silently absent from the output.
5. Compute week-over-week percent change with a shift(1), explicitly leaving it undefined (not inf) for the first week and for any week following a zero-WAU week.

Worked example

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "user_id": [1, 2, 1, 3, 1, 2, 4],
    "event_time": [
        "2024-01-02 10:00:00", "2024-01-03 11:00:00",
        "2024-01-09 09:00:00", "2024-01-10 12:00:00",
        "2024-01-25 08:00:00", "2024-01-26 09:30:00", "2024-01-27 14:00:00",
    ],
})

def compute_wau(df, ts_col="event_time", user_col="user_id", tz="UTC"):
    df = df.copy()
    df[ts_col] = pd.to_datetime(df[ts_col])
    if df[ts_col].dt.tz is None:
        df[ts_col] = df[ts_col].dt.tz_localize(tz)
    else:
        df[ts_col] = df[ts_col].dt.tz_convert(tz)

    iso = df[ts_col].dt.isocalendar()
    df["iso_year"], df["iso_week"] = iso["year"], iso["week"]

    weekly = (df.groupby(["iso_year", "iso_week"])[user_col]
                .nunique().rename("wau").reset_index())

    start = pd.Timestamp.fromisocalendar(int(weekly["iso_year"].min()), int(weekly["iso_week"].min()), 1)
    end = pd.Timestamp.fromisocalendar(int(weekly["iso_year"].max()), int(weekly["iso_week"].max()), 1)
    week_starts = pd.date_range(start, end, freq="7D")
    all_idx = [(d.isocalendar().year, d.isocalendar().week) for d in week_starts]

    full = (pd.DataFrame(all_idx, columns=["iso_year", "iso_week"])
              .merge(weekly, on=["iso_year", "iso_week"], how="left")
              .fillna({"wau": 0})
              .sort_values(["iso_year", "iso_week"])
              .reset_index(drop=True))

    full["wau_prev"] = full["wau"].shift(1)
    with np.errstate(divide="ignore", invalid="ignore"):
        full["wow_pct_change"] = ((full["wau"] - full["wau_prev"]) / full["wau_prev"]) * 100
    # undefined for the first week and for any week following a zero-WAU
    # week (division by zero would otherwise give an unusable inf)
    full.loc[(full["wau_prev"].isna()) | (full["wau_prev"] == 0), "wow_pct_change"] = None

    return full[["iso_year", "iso_week", "wau", "wow_pct_change"]]

print(compute_wau(df))

Output (verified against pandas 3.0.3):

   iso_year  iso_week  wau  wow_pct_change
0      2024         1  2.0             NaN
1      2024         2  2.0             0.0
2      2024         3  0.0          -100.0
3      2024         4  3.0             NaN

Week 3 has zero events (the gap), correctly reported as wau=0 and -100.0% change from week 2's 2. Week 4 follows a zero-WAU week, where a naive percent-change formula would divide by zero and report inf; that case is explicitly nulled out here instead, since "infinite percent growth" is not an interpretable business number.

Assumptions

• WAU is counted per ISO calendar week, not a rolling trailing-7-days window; these are two different (and both legitimate) definitions of "weekly," and the code above computes the calendar-bucket one because the question asks for week-over-week comparison, which implies fixed weeks.
• Naive timestamps are assumed to already be in the target timezone; if they are actually in a mix of timezones (e.g. server local time that changed when infrastructure moved regions), this code will silently mislabel some events' week.

Complexity

O(n) to parse and group the raw events, plus O(w) for the missing-week fill, where w is the number of distinct ISO weeks in range; w is normally tiny compared to n. Memory: the weekly aggregate is O(w), far smaller than the input, since the whole point of the aggregation is to collapse n events down to one row per week.

Edge cases

• Weeks with zero events: made explicit via the fill step rather than silently missing from the output, and handled specially in the percent-change formula (see above).
• ISO week 52/53 and year boundaries: .dt.isocalendar() returns the ISO year, which can differ from the calendar year for dates near January 1st; grouping by (iso_year, iso_week) rather than (calendar_year, iso_week) is what keeps late-December and early-January weeks from colliding.
• Duplicate events for the same user in the same week: .nunique() on user_id already collapses these correctly, no separate deduplication step needed.
• Bots or test accounts: not filtered here; if they are present in user_id, WAU includes them, and a real implementation should exclude them before this step.

Trade-offs and pitfalls

• Silently reporting inf for a percent change out of a zero base is a common and confusing mistake in growth-metric code; treat it the same way you would treat a first-period-of-history percent change, as undefined rather than a number.

Direct answer

.loc selects by label with an inclusive slice endpoint, .iloc selects by integer position with an exclusive slice endpoint, matching a Python list slice. Chained indexing, two indexing operations performed back to back such as df[df['age'] > 30]['score'] = value, is dangerous because under pandas' Copy-on-Write (CoW) engine the first indexing step always produces an independent copy, so the second step's assignment can land on that throwaway copy and leave the original df completely unchanged. The fix is to do the row selection and the column write in a single .loc call: df.loc[cond, 'col'] = value.

Approach

import pandas as pd

df = pd.DataFrame({
    'id': range(30),
    'name': [f"u{i}" for i in range(30)],
    'age': [20 + (i % 50) for i in range(30)],
    'signup_date': pd.date_range("2020-01-01", periods=30, freq='D'),
    'score': [i * 0.5 for i in range(30)],
})

# Rows 10-20
rows_loc = df.loc[10:20]        # 11 rows: labels 10 through 20, BOTH inclusive
rows_iloc = df.iloc[10:21]      # 11 rows: positions 10 through 20, stop 21 EXCLUDED

# Boolean condition
cond_rows = df.loc[df['age'] > 30]

# Columns by label and by position
cols_label = df.loc[:, ['id', 'age', 'score']]
cols_pos = df.iloc[:, [0, 2, 4]]

# DANGEROUS: chained indexing (single expression)
df[df['age'] > 30]['score'] = 999
# pandas raises the ChainedAssignmentError warning at this line, and df is untouched

# ALSO DANGEROUS, and silent: chained indexing across two statements
subset = df[df['age'] > 30]
subset['score'] = subset['score'] * 1.1
# NO warning at all here: subset is already an independent copy, so the assignment
# succeeds on subset only; df is never touched, and nothing signals that anything happened

# SAFE: row and column selection in one .loc call
df.loc[df['age'] > 30, 'score'] = df.loc[df['age'] > 30, 'score'] * 1.1

Key points

• rows_loc and rows_iloc are identical here (11 rows each) only because reaching position 20 by label and by "stop at 21" both land on the same rows; the inclusive/exclusive difference is what makes the two slice expressions (10:20 vs 10:21) necessary to get the same result.
• The single-expression chain (df[cond]['score'] = value) IS detected by pandas: it raises the ChainedAssignmentError warning at the point of assignment, and the original frame is left unchanged.
• The two-statement chain (subset = df[cond]; subset['score'] = ...) is NOT detected: by the time you mutate subset, pandas has no way to know you intended it to represent a piece of df, so the write lands cleanly on subset alone, with zero warning of any kind. This is the more dangerous shape precisely because nothing looks wrong.
• SettingWithCopyWarning was removed from pandas entirely as of 3.0. If you are reading older material describing it, that warning class no longer exists; the CoW-era failure modes are the ChainedAssignmentError warning (visible) and the silent two-statement case above (not visible).

Complexity

.loc label lookups are backed by the index's hash table, close to O(1) average per label, so selecting k rows by label is roughly O(k). .iloc positional lookups are direct offset arithmetic, O(k) for k positions. A boolean-mask selection like df['age'] > 30 is O(n) to build the mask over all n rows, plus O(k) to materialize the k matching rows. Chained assignment costs nothing extra algorithmically: the danger is purely correctness (the write silently lands nowhere useful), not performance.

Edge cases

• Duplicate index labels: df.loc[label] returns every row carrying that label, which can silently expand what you expected to be a single row; .iloc is unaffected since it always addresses exactly one position.
• No rows match the condition: .loc/boolean selection returns an empty (0-row) DataFrame, not an error, so a chained assignment onto an empty selection also fails silently with nothing to observe.
• An index that is itself integer-labeled but not a simple 0..n-1 range (for example after a prior filter without reset_index): df.loc[10:20] uses those labels, not positions, so it can select a completely different set of rows than df.iloc[10:20] would.
• Assigning into a genuinely one-off copy returned by another operation (e.g. a Series pulled out of a groupby result) via chained indexing hits the exact same footgun: the write updates the copy, not any object you can trace back to.

Trade-offs and pitfalls

Copy-on-Write makes the old "is this a view or a copy" ambiguity deterministic (df[cond] is now always a copy), but that determinism does not make chained assignment safe, it just changes how it fails: a warned-and-ineffective write for the single-expression form, and a silent-and-ineffective write for the two-statement form. Both leave df unmodified either way. The reliable pattern is always a single .loc/.iloc call that does the row and column selection together, or an explicit .copy() when you genuinely intend to work on an independent object rather than update the source.

Direct answer: A pandas MultiIndex (hierarchical index) lets a single axis (rows or columns) carry more than one level of labels, so a (country, city) pair identifies a row instead of needing two separate lookup columns. It gives you fast partial selection on the outer level(s), natural groupby-by-level aggregation, and a compact representation that avoids repeating country on every city row.

Structured elaboration

Setup used for every example below (build this fixture exactly as shown before running any block):

import pandas as pd

tuples = [('US', 'NY'), ('US', 'SF'), ('CA', 'Toronto'), ('CA', 'Vancouver')]
idx = pd.MultiIndex.from_tuples(tuples, names=['country', 'city'])
df = pd.DataFrame(
    {'year': [2020, 2020, 2020, 2020],
     'population': [8_336_000, 884_000, 2_930_000, 631_000]},
    index=idx,
)

1. Select every row for one country (partial indexing on the outer level):

print(df.loc['US'])
# equivalently, explicit about which level you're selecting on:
print(df.xs('US', level='country'))

Both drop the country level and return a frame indexed only by city.

2. Reorder and sort the levels:

df_swapped = df.swaplevel('country', 'city').sort_index()
print(df_swapped)

swaplevel only changes level order; it does not sort. Skipping sort_index after a level swap leaves the index non-monotonic, which degrades or breaks partial-label slicing (see Edge cases).

3. Turn the MultiIndex back into plain columns:

df_flat = df.reset_index()
print(df_flat)
df_flat['country_city'] = df_flat['country'] + '_' + df_flat['city']

reset_index() moves every level into a regular column and replaces the index with a default RangeIndex.

4. A related but different operation, unstack, pivots one index level into columns instead of dropping it into rows. It is the tool when you want a country x city matrix rather than flat rows:

print(df['population'].unstack('city'))

This produces one row per country and one column per city, with NaN (not-a-number, pandas' missing-value marker) where a country has no data for that city (verified: CA has NaN under NY/SF, US has NaN under Toronto/Vancouver).

Worked example (verified output, pandas 3.0.3, REPL transcript shown for readability, not meant to be pasted as one script):

>>> df.loc['US']
      year  population
city
NY    2020     8336000
SF    2020      884000

>>> df.swaplevel('country', 'city').sort_index()
                   year  population
city      country
NY        US       2020     8336000
SF        US       2020      884000
Toronto   CA       2020     2930000
Vancouver CA       2020      631000

>>> df.reset_index()
  country       city  year  population
0      US         NY  2020     8336000
1      US         SF  2020      884000
2      CA    Toronto  2020     2930000
3      CA  Vancouver  2020      631000

Complexity: Building the MultiIndex from tuples is O(n log n) (it deduplicates each level into codes and sorts internally for the factorization). Once the index is monotonic ("lexsorted"), .loc['US'] is a binary search on the outer level, O(log n) to find the block plus O(k) to return the k matching rows. Internally a MultiIndex stores each level as small integer codes referencing a deduplicated array of labels, so it is more memory-efficient than a flat index built from concatenated "country_city" strings, especially when labels repeat heavily. reset_index() and swaplevel() both copy the underlying data (O(n) space), they do not mutate in place.

Edge cases:

• Non-lexsorted index: if the levels aren't sorted, partial-label indexing still works but pandas emits PerformanceWarning: indexing past lexsort depth may impact performance (verified) because it falls back to a linear scan instead of binary search. Always sort_index() after swaplevel or after building an index from unsorted tuples.
• Duplicate index entries: df.loc[('US', 'NY')] when that tuple appears twice returns both rows, not an error, silently changing your code's assumption of "one row per key" if you didn't expect duplicates.
• Missing key: df.loc['MX'] on a country not present raises KeyError, so wrap lookups you don't control (e.g. user input) in a try/except or check 'MX' in df.index.get_level_values('country') first.
• All-NaN or empty frame: reset_index() and unstack() both work fine on an empty MultiIndex frame, they simply return an empty frame or an all-NaN matrix respectively; no special-casing is required.

Trade-offs and pitfalls: when MultiIndex helps vs. a flat index

A common mistake is reaching for MultiIndex purely to avoid repeating a column value, when a flat index with an ordinary groupby on that column would be just as fast and far more familiar to teammates and downstream tools.

Direct answer

On current pandas (verified on 3.0.3), this code neither raises an error nor modifies df. It silently succeeds on subset alone and leaves df completely unchanged, with no warning of any kind. That is different from what older pandas material teaches: pre-2.0 pandas would very likely have printed a SettingWithCopyWarning on the second line, precisely because it could not tell whether subset shared memory with df. Copy-on-Write (CoW), pandas's memory model where any object derived from indexing behaves as an independent copy until (and unless) you write back to it, is mandatory and cannot be disabled starting in pandas 3.0, and under CoW pandas.errors.SettingWithCopyWarning has been removed from the library entirely. The ambiguity that warning used to flag no longer exists, subset is now unambiguously independent, so pandas has nothing left to warn about, but it also will not tell you that your write did not reach df.

What is actually happening, line by line

import pandas as pd

df = pd.DataFrame({'A': [1, -1, 2, -2], 'B': [0, 0, 0, 0], 'C': [10, 20, 30, 40]})
subset = df[df['A'] > 0]
subset['B'] = subset['C'] * 2

• Line 1 is a single indexing operation: boolean-mask filtering. Under CoW its result is guaranteed to behave as an independent object with respect to writes, regardless of whatever it may share with df internally for reads.
• Line 2 is a plain __setitem__ call on the named variable subset. This is not chained indexing, subset is a real, already-bound DataFrame object, and setting a column on it is a completely ordinary, single operation. Because pandas has no way (and no need) to know that subset originated from filtering df, and because CoW already guarantees subset behaves independently, the write lands on subset and nowhere else.

This is genuinely different from the pattern that still produces a diagnostic on pandas 3.0, writing the filter and the assignment as one chained expression:

import pandas as pd
import warnings

df2 = pd.DataFrame({'A': [1, -1, 2, -2], 'B': [0, 0, 0, 0], 'C': [10, 20, 30, 40]})
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    df2[df2['A'] > 0]['B'] = 99   # one expression, two indexing ops chained together
    print(len(caught), caught[0].category.__name__ if caught else None)

Verified output: 1 ChainedAssignmentError, and df2['B'] is confirmed unchanged (still [0, 0, 0, 0]). Here pandas can detect, within that single expression, that a __getitem__ result is immediately being written to, and it raises pandas.errors.ChainedAssignmentError. That name is slightly misleading: it is implemented as a Warning subclass, so by default it prints a message and execution continues (it does not halt the program) unless you have configured warnings.filterwarnings('error', category=pandas.errors.ChainedAssignmentError) to escalate it. Either way, df2 is left untouched in that case too, CoW guarantees the intermediate object from df2[df2['A'] > 0] behaves as a copy, so the chained write can never propagate back.

The two-line version in the question is quietly the more dangerous case of the two: it produces no diagnostic at all, so a candidate who expects a SettingWithCopyWarning from older material, or a ChainedAssignmentError by analogy with the one-liner, will be surprised that pandas 3.0 says nothing while still not updating df.

Worked example (verified against pandas 3.0.3)

import pandas as pd

df = pd.DataFrame({'A': [1, -1, 2, -2], 'B': [0, 0, 0, 0], 'C': [10, 20, 30, 40]})
subset = df[df['A'] > 0]
subset['B'] = subset['C'] * 2

print(subset)
print(df)

Actual output:

   A   B   C
0  1  20  10
2  2  60  30
   A  B   C
0  1  0  10
1 -1  0  20
2  2  0  30
3 -2  0  40

subset['B'] was updated to [20, 60]. df['B'] stayed exactly [0, 0, 0, 0]. No warning was printed. Running this with warnings.catch_warnings(record=True) around it confirms zero warnings were emitted, this is not a case where the warning is merely suppressed by default settings, there genuinely is nothing raised.

The correct rewrites

To modify df itself, in place, unambiguously, do the filter and the write as a single .loc call directly on df:

mask = df['A'] > 0
df.loc[mask, 'B'] = df.loc[mask, 'C'] * 2

.loc[row_selector, col_selector] = value is one indexing operation on df, so there is no intermediate object for the write to get lost in.

To work on a genuinely independent object on purpose (for example, exploratory analysis you do not want to feed back into df), make the copy explicit so the intent is unambiguous to both pandas and the next reader of the code:

subset = df[df['A'] > 0].copy()
subset['B'] = subset['C'] * 2   # safe: subset is explicitly independent, no ambiguity to guess about

How to tell which one you have, before writing to it

The honest pandas-3.0 answer is that you no longer need to determine this before writing, and that is the actual point of Copy-on-Write. Pre-CoW, pandas tracked provenance internally (an undocumented, private flag set when a DataFrame was produced by slicing another one) so that it could guess whether to warn, and that guess was exactly the source of the bug class: a subset could be a view sometimes and a copy other times, depending on internal memory layout you did not control, so "check whether you have a view or a copy" was fragile advice even when it was the standard advice. Under CoW, write-time behavior is uniform regardless of what pandas is doing with memory sharing behind the scenes: any object produced by indexing behaves as independent unless you write to the source through a single .loc (or .iloc) call. So the reliable practice is not "inspect what you have," it is "decide what you intend before you write": use .loc on the original if you intend to mutate df, use an explicit .copy() if you intend to work on something separate, and treat the result of any other indexing expression as read-only in between.

Trade-offs and pitfalls

• .loc-based writes are direct and memory-efficient: no extra copy is made beyond what the write itself requires. The risk is entirely in getting the mask or column selector wrong, since there is no separate subset object to sanity-check before the write lands.
• An explicit .copy() is the safer default for exploratory or multi-step work: you avoid ever wondering whether a downstream mutation reaches df, at the cost of the extra memory the copy uses. If you later do want the changes back in df, you need an explicit step (df.loc[subset.index, 'B'] = subset['B'], or df.update(subset[['B']])), it does not happen automatically.
• The two-line pattern shown in the question is the one most likely to fool someone who learned pandas from pre-2.0 material: it looks exactly like the textbook SettingWithCopyWarning example, and it still silently fails to update df on pandas 3.0, but it no longer prints anything telling you so. Do not rely on a warning being there to catch this class of bug on current pandas; make .loc versus .copy() a deliberate choice instead of something a warning nudges you toward after the fact.
• If you are maintaining code that still targets pandas versions before Copy-on-Write was mandatory, the underlying advice does not change, .loc for in-place writes, .copy() for independent objects, but you may still see SettingWithCopyWarning on those versions, and silencing it with pd.options.mode.chained_assignment = None hides the ambiguity rather than resolving it.

Complexity and edge cases

Complexity: both rewrites are O(k) where k is the number of matched rows, no different in asymptotic cost from the original buggy version. .copy() additionally allocates O(k) new memory for the independent object; the .loc in-place write allocates no extra copy beyond what the assignment itself requires.
Edge cases:

• An empty mask (no rows satisfy the filter) makes either rewrite a safe no-op, df.loc[mask, 'B'] = ... with an all-False mask assigns to zero rows without error.
• If df has a duplicate (non-unique) index, .loc[mask, 'B'] = df.loc[mask, 'C'] * 2 still works correctly because both sides of the assignment are computed from the same boolean mask over the same frame, but constructing the right-hand side from a different object with an overlapping-but-not-identical index would silently misalign by label, so keep both sides derived from the same masked selection.
• If C were float64 while B started as int64, assigning subset['C'] * 2 into B upcasts B to float64 for that object; this is ordinary pandas dtype-promotion behavior, not specific to the view/copy issue, but it is worth checking for when a downstream step assumes integer dtypes.

Direct answer

Prefer pivot_table when you want a one-call pivot that also aggregates duplicate (index, column) pairs; prefer groupby + unstack when you want explicit control over the aggregation step or need to chain other transformations before reshaping. pivot_table is a convenience layer built on the same groupby machinery underneath, so their performance is close; the real difference is ergonomics and what happens when your keys are not unique.

Structured elaboration

pd.pivot (no _table) is the plain reshape, no aggregation at all: give it duplicate keys and it raises rather than silently picking one. pivot_table groups the duplicate rows and reduces them with aggfunc before reshaping, which is exactly why it never errors on the duplicate (store, date, product) rows described here.

Worked example

import pandas as pd

df = pd.DataFrame({
    'store': ['S1', 'S1', 'S1', 'S2', 'S2'],
    'date': ['2026-01-01', '2026-01-01', '2026-01-02', '2026-01-01', '2026-01-01'],
    'product': ['A', 'A', 'B', 'A', 'A'],
    'sales': [10, 5, 7, 3, 4],
})

sales_matrix = pd.pivot_table(
    df,
    values='sales',
    index='store',
    columns='product',
    aggfunc='sum',
    fill_value=0
)
print(sales_matrix)

Output (verified by running this exact code, and confirmed identical to df.groupby(['store','product'])['sales'].sum().unstack(fill_value=0)):

product   A  B
store
S1       15  7
S2        7  0

Store S1 has two rows for ('S1', 'A') (sales 10 and 5); pivot_table sums them to 15 rather than raising or silently dropping one. The (S2, B) combination never occurs in the source data, so fill_value=0 fills it in rather than leaving NaN.

How pivot_table handles duplicates, precisely

For every (index, column) combination that appears more than once, pivot_table collects all matching rows' values and reduces them with aggfunc before placing a single cell in the output. This is exactly groupby([index_cols, columns_cols])[values].agg(aggfunc) followed by .unstack(); pivot_table is not doing anything groupby cannot do, it is packaging that same operation into one call with a friendlier signature (aggfunc, fill_value, margins) at the cost of being slightly less flexible if you need to do something groupby's chain does not directly expose.

Trade-offs and pitfalls

• Performance is close to identical since pivot_table is a groupby under the hood; do not choose between them for speed, choose for readability and whether you need margins/fill_value for free.
• groupby + unstack gives you more control mid-pipeline: you can filter, apply multiple aggregations with .agg({...}), or reset the index before unstacking, which is awkward to bolt onto pivot_table's single call.
• pivot_table defaults aggfunc to 'mean', which silently changes your result if you forget to pass aggfunc='sum' explicitly, a common source of "my totals look too small" bugs.
• Both can leave you with a MultiIndex on the columns if you pivot on more than one values/columns combination; flatten it explicitly (e.g. columns.map('_'.join)) before handing the result to code that expects flat column names.