Python and Pandas for Data Analysis Questions

Direct answer

On current pandas (verified on 3.0.3), this code neither raises an error nor modifies df. It silently succeeds on subset alone and leaves df completely unchanged, with no warning of any kind. That is different from what older pandas material teaches: pre-2.0 pandas would very likely have printed a SettingWithCopyWarning on the second line, precisely because it could not tell whether subset shared memory with df. Copy-on-Write (CoW), pandas's memory model where any object derived from indexing behaves as an independent copy until (and unless) you write back to it, is mandatory and cannot be disabled starting in pandas 3.0, and under CoW pandas.errors.SettingWithCopyWarning has been removed from the library entirely. The ambiguity that warning used to flag no longer exists, subset is now unambiguously independent, so pandas has nothing left to warn about, but it also will not tell you that your write did not reach df.

What is actually happening, line by line

import pandas as pd

df = pd.DataFrame({'A': [1, -1, 2, -2], 'B': [0, 0, 0, 0], 'C': [10, 20, 30, 40]})
subset = df[df['A'] > 0]
subset['B'] = subset['C'] * 2

• Line 1 is a single indexing operation: boolean-mask filtering. Under CoW its result is guaranteed to behave as an independent object with respect to writes, regardless of whatever it may share with df internally for reads.
• Line 2 is a plain __setitem__ call on the named variable subset. This is not chained indexing, subset is a real, already-bound DataFrame object, and setting a column on it is a completely ordinary, single operation. Because pandas has no way (and no need) to know that subset originated from filtering df, and because CoW already guarantees subset behaves independently, the write lands on subset and nowhere else.

This is genuinely different from the pattern that still produces a diagnostic on pandas 3.0, writing the filter and the assignment as one chained expression:

import pandas as pd
import warnings

df2 = pd.DataFrame({'A': [1, -1, 2, -2], 'B': [0, 0, 0, 0], 'C': [10, 20, 30, 40]})
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    df2[df2['A'] > 0]['B'] = 99   # one expression, two indexing ops chained together
    print(len(caught), caught[0].category.__name__ if caught else None)

Verified output: 1 ChainedAssignmentError, and df2['B'] is confirmed unchanged (still [0, 0, 0, 0]). Here pandas can detect, within that single expression, that a __getitem__ result is immediately being written to, and it raises pandas.errors.ChainedAssignmentError. That name is slightly misleading: it is implemented as a Warning subclass, so by default it prints a message and execution continues (it does not halt the program) unless you have configured warnings.filterwarnings('error', category=pandas.errors.ChainedAssignmentError) to escalate it. Either way, df2 is left untouched in that case too, CoW guarantees the intermediate object from df2[df2['A'] > 0] behaves as a copy, so the chained write can never propagate back.

The two-line version in the question is quietly the more dangerous case of the two: it produces no diagnostic at all, so a candidate who expects a SettingWithCopyWarning from older material, or a ChainedAssignmentError by analogy with the one-liner, will be surprised that pandas 3.0 says nothing while still not updating df.

Worked example (verified against pandas 3.0.3)

import pandas as pd

df = pd.DataFrame({'A': [1, -1, 2, -2], 'B': [0, 0, 0, 0], 'C': [10, 20, 30, 40]})
subset = df[df['A'] > 0]
subset['B'] = subset['C'] * 2

print(subset)
print(df)

Actual output:

   A   B   C
0  1  20  10
2  2  60  30
   A  B   C
0  1  0  10
1 -1  0  20
2  2  0  30
3 -2  0  40

subset['B'] was updated to [20, 60]. df['B'] stayed exactly [0, 0, 0, 0]. No warning was printed. Running this with warnings.catch_warnings(record=True) around it confirms zero warnings were emitted, this is not a case where the warning is merely suppressed by default settings, there genuinely is nothing raised.

The correct rewrites

To modify df itself, in place, unambiguously, do the filter and the write as a single .loc call directly on df:

mask = df['A'] > 0
df.loc[mask, 'B'] = df.loc[mask, 'C'] * 2

.loc[row_selector, col_selector] = value is one indexing operation on df, so there is no intermediate object for the write to get lost in.

To work on a genuinely independent object on purpose (for example, exploratory analysis you do not want to feed back into df), make the copy explicit so the intent is unambiguous to both pandas and the next reader of the code:

subset = df[df['A'] > 0].copy()
subset['B'] = subset['C'] * 2   # safe: subset is explicitly independent, no ambiguity to guess about

How to tell which one you have, before writing to it

The honest pandas-3.0 answer is that you no longer need to determine this before writing, and that is the actual point of Copy-on-Write. Pre-CoW, pandas tracked provenance internally (an undocumented, private flag set when a DataFrame was produced by slicing another one) so that it could guess whether to warn, and that guess was exactly the source of the bug class: a subset could be a view sometimes and a copy other times, depending on internal memory layout you did not control, so "check whether you have a view or a copy" was fragile advice even when it was the standard advice. Under CoW, write-time behavior is uniform regardless of what pandas is doing with memory sharing behind the scenes: any object produced by indexing behaves as independent unless you write to the source through a single .loc (or .iloc) call. So the reliable practice is not "inspect what you have," it is "decide what you intend before you write": use .loc on the original if you intend to mutate df, use an explicit .copy() if you intend to work on something separate, and treat the result of any other indexing expression as read-only in between.

Trade-offs and pitfalls

• .loc-based writes are direct and memory-efficient: no extra copy is made beyond what the write itself requires. The risk is entirely in getting the mask or column selector wrong, since there is no separate subset object to sanity-check before the write lands.
• An explicit .copy() is the safer default for exploratory or multi-step work: you avoid ever wondering whether a downstream mutation reaches df, at the cost of the extra memory the copy uses. If you later do want the changes back in df, you need an explicit step (df.loc[subset.index, 'B'] = subset['B'], or df.update(subset[['B']])), it does not happen automatically.
• The two-line pattern shown in the question is the one most likely to fool someone who learned pandas from pre-2.0 material: it looks exactly like the textbook SettingWithCopyWarning example, and it still silently fails to update df on pandas 3.0, but it no longer prints anything telling you so. Do not rely on a warning being there to catch this class of bug on current pandas; make .loc versus .copy() a deliberate choice instead of something a warning nudges you toward after the fact.
• If you are maintaining code that still targets pandas versions before Copy-on-Write was mandatory, the underlying advice does not change, .loc for in-place writes, .copy() for independent objects, but you may still see SettingWithCopyWarning on those versions, and silencing it with pd.options.mode.chained_assignment = None hides the ambiguity rather than resolving it.

Complexity and edge cases

Complexity: both rewrites are O(k) where k is the number of matched rows, no different in asymptotic cost from the original buggy version. .copy() additionally allocates O(k) new memory for the independent object; the .loc in-place write allocates no extra copy beyond what the assignment itself requires.
Edge cases:

• An empty mask (no rows satisfy the filter) makes either rewrite a safe no-op, df.loc[mask, 'B'] = ... with an all-False mask assigns to zero rows without error.
• If df has a duplicate (non-unique) index, .loc[mask, 'B'] = df.loc[mask, 'C'] * 2 still works correctly because both sides of the assignment are computed from the same boolean mask over the same frame, but constructing the right-hand side from a different object with an overlapping-but-not-identical index would silently misalign by label, so keep both sides derived from the same masked selection.
• If C were float64 while B started as int64, assigning subset['C'] * 2 into B upcasts B to float64 for that object; this is ordinary pandas dtype-promotion behavior, not specific to the view/copy issue, but it is worth checking for when a downstream step assumes integer dtypes.

Direct answer: Treat "different results on two machines" as an environment-parity problem first and a pandas-logic problem second: confirm the pandas/numpy/dependency versions actually match, then add explicit dtype casts, an explicit sort key, and a tolerance-based comparison to the transformation itself so any remaining difference is forced to show up as a concrete, inspectable diff rather than a vague "the numbers are different."

Structured elaboration: debugging plan

1. Confirm environment parity (Python, pandas, numpy, and their native dependencies like BLAS/LAPACK).
2. Reproduce the failure on a small, concrete input you can run in both places.
3. Make the transformation deterministic: explicit dtypes, explicit sort key, stable sort algorithm.
4. Compare outputs with a tolerance-aware assertion instead of eyeballing numbers.
5. Once you have a reproducible diff, narrow the cause: dtype coercion, unstable sort/groupby ordering, or genuine floating-point difference from a different math backend.

1. Environment parity

import sys, platform
import pandas as pd, numpy as np

print("python:", sys.version.splitlines()[0])
print("platform:", platform.platform())
print("pandas:", pd.__version__)
print("numpy:", np.__version__)
pd.show_versions()  # prints pandas + every dependency's version; don't wrap it in print(), it already prints and returns None

Run pip freeze (or conda list) on both machines and diff the two outputs directly; a mismatched numpy, pyarrow, or BLAS library is the single most common cause of "same code, different numbers."

2-4. A concrete, reproducible transformation with built-in invariants (verified, pandas 3.0.3):

from pandas.testing import assert_frame_equal

def normalize_df(df):
    df = df.copy()
    df = df.reset_index(drop=True)
    if 'id' in df: df['id'] = df['id'].astype('int64')
    if 'cat' in df: df['cat'] = df['cat'].astype('category')
    sort_keys = ['id'] if 'id' in df.columns else list(df.columns)
    df = df.sort_values(sort_keys, kind='mergesort')  # stable sort: ties keep input order
    df = df.reset_index(drop=True)
    return df

def transform(df):
    df2 = df.assign(value=df['a'] / df['b'])
    grouped = df2.groupby(['cat', 'id'], sort=True, as_index=False).agg({'value': 'mean'})
    return grouped

a = pd.DataFrame({
    'id':  [3, 1, 2, 1],
    'cat': ['x', 'y', 'x', 'y'],
    'a':   [10.0, 20.0, 30.0, 5.0],
    'b':   [2.0, 4.0, 3.0, 2.0],
})

left = normalize_df(transform(a))   # stand-in for "developer laptop output"
right = normalize_df(transform(a))  # stand-in for "production output"

try:
    assert_frame_equal(left, right, check_dtype=True, check_exact=False, rtol=1e-9, atol=1e-12)
    print("Frames equal within tolerance")
except AssertionError as e:
    print("Assertion failed:", e)
    diff = left.compare(right, align_axis=0)
    print(diff.head(50))

Verified output on this fixture: Frames equal within tolerance, and left is

  cat  id  value
0   y   1   3.75
1   x   2  10.00
2   x   3   5.00

In a real "diverges between machines" scenario, left would be the developer laptop's saved output and right the production output loaded from wherever it was captured; the point of normalize_df is to remove sources of incidental difference (row order, index values, dtype) before you ask whether the values differ.

Checks and invariants worth adding

• df.dtypes.value_counts() and, for categoricals, df['cat'].cat.categories, a dtype or category-set mismatch is the most common silent cause.
• df.isna().sum() and np.isfinite(df.select_dtypes(float)).all() to catch unexpected nulls or infinities (division by zero, overflow) before they propagate into an aggregation.
• Force a stable sort (kind='mergesort') before any groupby whose downstream code assumes row order, the default sort is not guaranteed stable across all inputs.
• If you use Parquet/pyarrow anywhere in the pipeline, confirm pyarrow.__version__ parity too and pin engine='pyarrow' explicitly rather than letting pandas auto-select an engine that may differ by environment.
• Add an automated CI check that fails the build when pip freeze on the CI runner diverges from a locked requirements/lockfile, catching the root cause before it ever reaches production.

Complexity: the checks above are dominated by the sort (O(n log n)) and the groupby (O(n) average with a hash-based grouping), the same order of work the original transform already does, so adding them does not change the pipeline's asymptotic cost. The assert_frame_equal comparison is O(n * m) in the size of the two frames being compared.

Edge cases

• Empty input on one machine: assert_frame_equal on two empty frames with matching dtypes passes trivially; verify dtypes still match on an empty frame; an empty groupby result can silently have a different dtype (e.g. object instead of float64) than a non-empty one.
• All-NaN column: df['a'] / df['b'] where b is 0 produces inf, not NaN (not-a-number, pandas' missing-value marker), catch it explicitly with the isfinite check above rather than assuming a null check is sufficient.
• Duplicate index or duplicate group keys: reset_index(drop=True) before comparing sidesteps index-value mismatches entirely, always drop the index in the comparison unless the index itself is part of what you're validating.
• Timezone-aware datetimes: a tz-naive timestamp on one machine and a tz-aware one on the other compare as different dtypes entirely (assert_frame_equal will fail on dtype, not just value), normalize both to a single, explicit timezone before comparing.

Direct answer

Set a DatetimeIndex (not a (user, timestamp) MultiIndex) for the rolling call itself, grouped by the user column, then join the result back onto a frame keyed by the full (user, timestamp) MultiIndex. Rolling with a calendar window like '7D' requires a DatetimeIndex; putting the datetime inside a MultiIndex and calling .groupby(level=0).rolling('7D') on the whole frame raises ValueError: window must be an integer 0 or greater, because pandas cannot resolve a calendar window against a MultiIndex level directly.

Approach

1. Sort by (user, timestamp). This is a hard requirement: rolling windows trust row order.
2. For the rolling computation only, index the frame by timestamp alone (a plain DatetimeIndex) and group by the user column.
3. Call .rolling('7D', min_periods=1) on the single value column and aggregate mean/std/min/max in one .agg([...]) call, avoiding four separate rolling passes.
4. Join the result back onto a frame indexed by the full (user, timestamp) MultiIndex, which matches the grouped rolling result's own (user, timestamp) index exactly, so timestamps that repeat across different users cannot fan out into duplicate rows during the join.

Worked example

import pandas as pd
pd.set_option("display.width", 200)
pd.set_option("display.max_columns", None)

df = pd.DataFrame({
    "user": ["a", "a", "a", "b", "b"],
    "timestamp": ["2024-01-01", "2024-01-03", "2024-01-06", "2024-01-01", "2024-01-02"],
    "value": [10.0, 20.0, 30.0, 5.0, 15.0],
})
df["timestamp"] = pd.to_datetime(df["timestamp"])
df = df.sort_values(["user", "timestamp"])

# time-based rolling needs a DatetimeIndex, so index by timestamp only for
# this call (grouped by the 'user' column, not by an index level)
by_time = df.set_index("timestamp")
rolling_obj = by_time.groupby("user")["value"].rolling("7D", closed="both", min_periods=1)
agg = rolling_obj.agg(["mean", "std", "min", "max"]).rename(columns={
    "mean": "value_7d_mean", "std": "value_7d_std", "min": "value_7d_min", "max": "value_7d_max",
})

# agg's index is already (user, timestamp), matching df's own MultiIndex, so
# no droplevel is needed and the join cannot duplicate rows even though
# 2024-01-01 repeats across both users
result = df.set_index(["user", "timestamp"]).join(agg).reset_index()
print(result)

Output (verified against pandas 3.0.3):

  user  timestamp  value  value_7d_mean  value_7d_std  value_7d_min  value_7d_max
0    a 2024-01-01   10.0           10.0           NaN          10.0          10.0
1    a 2024-01-03   20.0           15.0      7.071068          10.0          20.0
2    a 2024-01-06   30.0           20.0     10.000000          10.0          30.0
3    b 2024-01-01    5.0            5.0           NaN           5.0           5.0
4    b 2024-01-02   15.0           10.0      7.071068           5.0          15.0

User a's 01-06 row averages all three of their observations (all fall within a 7-day trailing window of 01-06): mean 20.0, std 10.0 over [10, 20, 30]. User b's rows never touch user a's despite both having a 2024-01-01 row, because the join matches on the full (user, timestamp) pair.

Indexing requirements this approach depends on

• A calendar-based .rolling('7D') call requires the object it is called on to have a DatetimeIndex (directly, or accessed through groupby(...)[col].rolling(...) where the frame being grouped has one). A MultiIndex containing a datetime level is not sufficient by itself.
• Data must be sorted by (user, timestamp) before rolling; an out-of-order timestamp within a user silently produces a wrong window rather than raising.
• Because the grouped rolling result's index is (user, timestamp), whatever you join it back onto must use that exact same MultiIndex (not a de-duplicated or level-dropped one) to avoid an accidental many-to-many join when timestamps repeat across groups.

Complexity

O(n log n) for the initial sort, then O(n) for the grouped rolling pass, since each user's window work is bounded by that user's own row count. Memory: the single .agg(['mean','std','min','max']) call computes all four statistics from one pass over each window rather than four separate rolling objects, avoiding four redundant intermediate Series; the final join adds one more full-size copy for the merged result (pandas 3.0's Copy-on-Write means the original df is never mutated by any of this).

Edge cases

• Duplicate timestamps within the same user: both rows fall in each other's windows correctly; std on a window of size 1 is NaN (there is no variance with a single observation), which is expected, not a bug.
• Duplicate timestamps across different users (as in the example, both users have a 2024-01-01 row): this is exactly the case that breaks a naive droplevel-then-join approach into a row-multiplying join; keeping the full MultiIndex on both sides avoids it.
• A user with only one observation: the rolling stats for that single row are just that row's own value for mean/min/max and NaN for std.

Trade-offs and pitfalls

• Grouping and rolling on a MultiIndex level directly (df.set_index(['user','timestamp']).groupby(level=0).rolling('7D')) looks like the natural extension of the single-index pattern but raises ValueError in current pandas; the DatetimeIndex has to be the object's only index for the calendar-window logic to resolve, so keep the datetime as the sole index during the rolling call and reintroduce the grouping key as a join afterward rather than as an index level.

Direct answer

Use pivot_table when you need a wide layout, one column per category, and possibly an aggregation, and use melt to go the other direction, back to the original long ("tidy") format. Plain pivot (without _table) only works when the index/columns pair is already unique per value; the moment two rows can land in the same cell, you need pivot_table with an explicit aggfunc.

Approach

import pandas as pd

sales = pd.DataFrame({
    'date': ['2024-01-01', '2024-01-01', '2024-01-02', '2024-01-02'],
    'store_id': [1, 1, 2, 2],
    'product': ['A', 'B', 'A', 'B'],
    'units_sold': [10, 5, 3, 8],
})

# 1) Pivot to wide format, summing units_sold, filling missing combos with 0
wide = sales.pivot_table(
    index='store_id',
    columns='product',
    values='units_sold',
    aggfunc='sum',
    fill_value=0,
).reset_index()
print('wide:')
print(wide)

# 2) Melt back to long format
long = pd.melt(
    wide,
    id_vars=['store_id'],
    var_name='product',
    value_name='units_sold',
)
print('long:')
print(long)

Output:

wide:
product  store_id   A  B
0               1  10  5
1               2   3  8

long:
   store_id product  units_sold
0         1       A          10
1         2       A           3
2         1       B           5
3         2       B           8

Key points

• pivot_table aggregates; plain pivot does not, and raises if it finds duplicate (index, columns) pairs instead of silently picking one.
• melt(id_vars=..., var_name=..., value_name=...) reverses the shape: id_vars stays fixed per row, and the former column headers become values in one var_name column.
• Columns that were dropped by the pivot (here, date) are not restored by the round trip; you would need to carry them alongside or re-join them from the original data.

Complexity

pivot_table is O(n) to scan the input rows into buckets, plus O(r x c) to materialize the wide grid, where r is the number of distinct index values and c is the number of distinct columns values; a high-cardinality columns key can blow this up even when n itself is modest. melt is O(rows_wide x value_columns), since it stacks every value column back into its own set of rows.

Edge cases

• A missing store/product combination: pivot_table fills it with fill_value=0 here rather than leaving NaN (not-a-number); without fill_value, it would be NaN.
• Duplicate (store_id, product) pairs in the input: pivot_table aggregates them via aggfunc (sum, here); plain pivot() raises ValueError: Index contains duplicate entries, cannot reshape.
• Melting a fill_value=0-filled wide table introduces rows for combinations that never existed in the original long data, all carrying value 0, so filter those out (long[long['units_sold'] != 0]) if you need to recover exactly the original rows.
• Empty input: pivot_table on an empty frame returns an empty result with the requested index/columns structure, not an error.

Trade-offs and pitfalls

Aggregating away duplicate combinations with pivot_table is convenient but lossy: if two rows in sales legitimately represent different dates for the same store/product, summing them into one wide cell discards the per-date detail permanently. If that detail matters, keep date in the index (e.g. index=['store_id', 'date']) rather than aggregating over it.

Direct answer

.loc selects by label, row and column names, and its slice endpoint is inclusive. .iloc selects by integer position, 0-based, and its slice endpoint is exclusive, exactly like a plain Python list slice. Both accept single labels/positions, lists, boolean arrays, and callables, but what those inputs mean differs: a .loc boolean array must align with the index, while .iloc needs a purely positional array of the right length.

Structured elaboration

Label vs position, and inclusive vs exclusive slicing.

import pandas as pd

df = pd.DataFrame({'A': [10, 20, 30]}, index=[0, 1, 2])

df.loc[1, 'A']    # 20  -> the row LABELED 1
df.iloc[1, 0]     # 20  -> the row at POSITION 1
# these agree here only because the index [0, 1, 2] looks identical to positions;
# they are conceptually different lookups and diverge once the index is not 0..n-1

df.loc[0:1, :]    # rows labeled 0 and 1 (BOTH included)
df.iloc[0:1, :]   # row at position 0 only (stop at 1 is EXCLUDED)

mask = df['A'] > 15
df.loc[mask]                    # rows where A > 15, mask aligned by label
df.loc[lambda d: d['A'] > 15]   # same result via a callable
# a boolean array passed to .iloc must be purely positional (no label alignment),
# so df.iloc[mask.values] is the form that works there, not the mask Series itself
df.iloc[mask.values]

Duplicate labels. If the index has repeated labels, df.loc[0] returns every row carrying that label as a DataFrame, not a single row; .iloc[0] is unaffected by duplicate labels since it addresses by position and always returns exactly the row at that position.

Worked example

import pandas as pd

df_dup = pd.DataFrame({'A': [1, 2, 3]}, index=[0, 0, 1])
df_dup.loc[0]
#    A
# 0  1
# 0  2
df_dup.iloc[0]
# A    1
# Name: 0, dtype: int64

Trade-offs and pitfalls

Relying on integer-looking labels is the most common source of confusion: df.loc[1] and df.iloc[1] coincide only when the index happens to be a default 0..n-1 range; after any filter, sort, or concat that leaves gaps or reorders the index, they diverge silently, so .loc[1] may return a completely different row than .iloc[1], or raise a KeyError if label 1 no longer exists. When you specifically need position-based access regardless of what the labels are, use .iloc; when you need to address rows by an identifier that should stay stable across reshuffles, use .loc.