Python and Pandas for Data Analysis Questions

Direct answer: groupby.filter keeps or drops an entire group's rows based on a predicate you write over that group, so df_orders.groupby('customer_id').filter(lambda g: len(g) >= 3) removes every row belonging to a customer with fewer than 3 transactions and keeps everyone else's rows untouched.

Structured elaboration and worked example (verified, pandas 3.0.3):

import pandas as pd

df_orders = pd.DataFrame({
    'customer_id': [1, 1, 1, 2, 2, 3, 3, 3, 3],
    'order_id':    [101, 102, 103, 201, 202, 301, 302, 303, 304],
    'amount':      [10, 20, 30, 15, 25, 5, 6, 7, 8],
})
df_filtered = df_orders.groupby('customer_id').filter(lambda g: len(g) >= 3)
print(df_filtered)

Verified output: customer 2 (2 transactions) is dropped entirely; customers 1 (3 transactions) and 3 (4 transactions) come back unchanged, in their original row order, with the original index preserved. An equivalent transform-based rewrite produces byte-identical output on the same fixture:

mask = df_orders.groupby('customer_id')['order_id'].transform('count') >= 3
print(df_orders[mask])
print(df_filtered.equals(df_orders[mask]))

(df_filtered.equals(df_orders[mask]) is True.)

filter vs. transform, the actual distinction

Key points

• filter evaluates the predicate once per group on the whole group slice; transform evaluates a function per group but must return something broadcastable back to every row, it cannot change how many rows exist.
• The transform-plus-boolean-mask rewrite above achieves the same result as filter here specifically because the condition ("group size >= 3") happens to reduce to a per-row boolean once broadcast; not every filter predicate has an equally simple transform equivalent.

Complexity: both approaches are O(n) in the number of rows for a group-size predicate: filter computes len(g) per group (O(n) total across all groups) then does one boolean selection pass (O(n)); transform('count') is a single vectorized pass with no per-group Python-level callback, which is typically faster for a plain built-in aggregation like 'count' because it avoids invoking a Python lambda once per group. filter with an arbitrary Python lambda predicate pays that per-group call overhead; prefer the transform + boolean-mask form when the predicate can be expressed as one, as here.

Edge cases

• No group meets the threshold: filter returns an empty DataFrame with the original columns and dtypes preserved (verified: int64 columns stay int64 on the empty result, not coerced to object).
• A customer with exactly 3 transactions: kept, >= not >, verify the boundary condition matches what "fewer than 3" actually means before shipping (fewer than 3 means drop groups with len(g) < 3, equivalently keep len(g) >= 3).
• Missing customer_id (NaN, not-a-number, pandas' missing-value marker): by default groupby excludes NaN keys entirely, so those rows never reach the predicate at all, they are silently dropped by grouping, not by filter. If null customer ids must be evaluated as their own group, pass groupby('customer_id', dropna=False) explicitly.
• Duplicate order rows for the same customer: filter counts rows, not distinct transactions, if duplicates need to be deduplicated before the size check, do that before grouping, not after.

Direct answer

Run a short, repeatable inspection pass over any newly loaded DataFrame: dtypes and non-null counts first (df.info()), then descriptive statistics split by numeric versus categorical columns (since describe() reports very different things for each), then percent missing per column, then a sample of unique values for whichever categorical column you actually care about, usually the one you are about to group, filter, or join on.

Implementation

import pandas as pd

def profile_df(df, cat_col=None, n_unique_sample=10):
    # 1) dtypes and non-null counts
    print("Dtypes and non-null counts:")
    print(df.info())

    # 2) descriptive stats for numeric columns
    print("\nNumeric summary:")
    print(df.select_dtypes(include="number").describe().T)

    # 3) descriptive stats for categorical/text/boolean columns
    # include 'string' alongside 'object' so this also catches columns under
    # pandas 3.0's default string dtype, not just the legacy object dtype
    cat = df.select_dtypes(include=["object", "string", "category", "bool"])
    if not cat.empty:
        print("\nCategorical summary (count, unique, top, freq):")
        print(cat.describe().T)
    else:
        print("\nNo categorical columns found.")

    # 4) percent missing per column
    missing_pct = df.isna().mean() * 100
    print("\nPercent missing per column:")
    print(missing_pct.sort_values(ascending=False))

    # 5) sample unique values for a chosen categorical column
    if cat_col is None:
        cat_col = cat.columns[0] if not cat.empty else None
    if cat_col is not None:
        uniques = df[cat_col].dropna().unique()
        print(f"\nSample unique values for '{cat_col}' (up to {n_unique_sample}):")
        print(list(uniques[:n_unique_sample]))
        print(f"Total unique: {len(uniques)}")
    else:
        print("\nNo categorical column to sample uniques from.")

Worked example (verified against pandas 3.0.3, no warnings emitted)

df = pd.DataFrame({
    "id":      [1, 2, 3, 4, 5],
    "price":   [10.5, 20.1, None, 15.0, 9.9],
    "country": ["US", "UK", "US", "CA", None],
    "active":  [True, False, True, True, False],
})
profile_df(df, cat_col="country")

Actual output (trimmed to the key sections):

Numeric summary:
       count    mean       std  min    25%    50%     75%   max
id       5.0   3.000  1.581139  1.0   2.00   3.00   4.000   5.0
price    4.0  13.875  4.733128  9.9  10.35  12.75  16.275  20.1

Categorical summary (count, unique, top, freq):
        count unique   top freq
country     4      3    US    2
active      5      2  True    3

Percent missing per column:
price      20.0
country    20.0
id          0.0
active      0.0

Sample unique values for 'country' (up to 10):
['US', 'UK', 'CA']
Total unique: 3

price correctly shows count=4 (one missing value excluded from the mean/std), country's describe() correctly reports US as the most frequent value with freq=2, and the missing-percent breakdown immediately tells you price and country are the two columns that need a decision (impute, drop, or carry forward as legitimately missing) before modeling or aggregation.

Key points

• df.info() is the fastest way to see dtypes and non-null counts together, catching an accidentally-all-object numeric column or an unexpectedly low non-null count in one glance.
• describe() reports different statistics depending on dtype: mean/std/quartiles for numeric columns, count/unique/top/freq for text, category, and boolean columns, calling it once on the whole DataFrame silently drops whichever family of columns it was not built for unless you explicitly split by select_dtypes first, as this function does.
• df.isna().mean() * 100 is a fast, vectorized way to get percent-missing per column without writing a loop, and sorting it descending immediately surfaces the columns that need attention first.
• Sampling unique values (rather than printing all of them) is deliberate: it is meant to catch typos and inconsistent category labels ("US" vs "U.S." vs "United States") at a glance, not to enumerate a high-cardinality column in full.

Complexity and edge cases

Complexity: each step here is a single O(n) or O(n * columns) pass, info(), describe(), and isna().mean() are all vectorized, so profiling scales linearly with the size of the DataFrame; nunique()/unique() on a very high-cardinality column is still O(n) but with a larger constant cost than the aggregate statistics.
Edge cases: a DataFrame with no numeric columns makes select_dtypes(include="number").describe() return an empty frame, not an error, worth guarding the print with a check if you want a clean message instead of an empty table. A very wide DataFrame (hundreds of columns) makes the printed describe().T output unwieldy, consider .sample(n=30, axis=1) on the columns or splitting the profile into a few DataFrames of columns at a time. A high-cardinality categorical column (a free-text or id-like column) will make cat.describe()'s unique/top/freq less informative (often unique close to len(df)), and printing "a sample of unique values" for it is far more useful than the full describe() summary, which is exactly why the function separates those two concerns.

Direct answer

Melt the wide table to long/tidy format with one row per (month, segment), reindex each segment to a complete, continuous monthly date range so missing months become explicit NaN (not-a-number) rather than silently absent rows, then compute month-over-month growth per segment with a grouped percent-change that guards zero and null denominators explicitly.

Structured elaboration

1. Parse month as a real datetime and sort by it, so downstream date-range logic has something to work with.
2. df.melt(id_vars='month', var_name='segment', value_name='value') turns the three wide segment columns into a long segment / value pair per month, which is the shape almost every later step (grouping, plotting, joining) wants.
3. Reindex each segment onto a complete pd.date_range(..., freq='MS') (month-start) so a month that is missing from the source data becomes an explicit NaN row instead of a silent gap that a pct_change would otherwise just skip over unnoticed.
4. Compute the month-over-month growth per segment. pct_change() alone already returns NaN when either the current or prior value is null, and returns inf/-inf (not a clean NaN) when the prior value is exactly zero, so wrap it to convert a zero-denominator case into an explicit sentinel (NaN or a flag column) rather than letting inf leak into later aggregations, where it silently poisons a mean or sum.

Worked example

import pandas as pd

df = pd.DataFrame({
    'month': ['2026-01', '2026-02', '2026-04'],
    'segment_A': [100, 110, 0],
    'segment_B': [200, 0, 210],
    'segment_C': [50, 55, 60],
})  # note March is missing entirely, this is what the reindex step below must surface

df['month'] = pd.to_datetime(df['month'])
df = df.sort_values('month')

long = df.melt(id_vars='month', var_name='segment', value_name='value')

min_m, max_m = long['month'].min(), long['month'].max()
all_months = pd.date_range(min_m, max_m, freq='MS')
segments = long['segment'].unique()
index = pd.MultiIndex.from_product([segments, all_months], names=['segment', 'month'])
long = long.set_index(['segment', 'month']).reindex(index).reset_index()

def safe_pct_change(x):
    prev = x.shift(1)
    denom_zero = prev == 0
    pct = (x - prev) / prev
    pct[denom_zero & x.notna()] = pd.NA
    return pct

long['mom_growth'] = long.groupby('segment')['value'].transform(safe_pct_change)
print(long)

Output, run against a 3-month fixture with a missing March row (verified by running this exact code):

      segment      month  value  mom_growth
0   segment_A 2026-01-01  100.0         NaN
1   segment_A 2026-02-01  110.0         0.1
2   segment_A 2026-03-01    NaN         NaN
3   segment_A 2026-04-01    0.0         NaN
4   segment_B 2026-01-01  200.0         NaN
5   segment_B 2026-02-01    0.0        -1.0
6   segment_B 2026-03-01    NaN         NaN
7   segment_B 2026-04-01  210.0         NaN
8   segment_C 2026-01-01   50.0         NaN
9   segment_C 2026-02-01   55.0         0.1
10  segment_C 2026-03-01    NaN         NaN
11  segment_C 2026-04-01   60.0         NaN

Key points and a correction to the pattern above

Use .transform(safe_pct_change), not .apply(safe_pct_change), to assign the result back onto the long frame's mom_growth column. groupby(...).apply() prepends the group key to the result's index by default, so assigning its output straight into long['mom_growth'] raises TypeError: incompatible index of inserted column with frame index under pandas 3.0.3, because the apply result carries a (segment, original_position) MultiIndex that pandas can no longer silently reconcile against long's plain index. .transform() is built exactly for this "one output row per input row, same index" shape and assigns back cleanly; this was verified directly, .apply() throws on assignment, .transform() does not, and both compute the identical per-row numbers when tested in isolation.

Trade-offs and pitfalls

• Deciding what to do with an explicit missing month is a business call, not a technical one: leaving it NaN is the honest default (an actual gap in the data), but if the business rule is "assume flat between reported months," use .ffill() after reindexing, never .fillna(method='ffill') (the method= keyword was removed from fillna in pandas 3.0 and now raises TypeError).
• pct_change() on a raw Series returns inf for a zero-to-nonzero jump and NaN only for a zero-to-zero step; if you skip the zero-guard shown above, an inf will silently pass through a later .mean() or .sum() and corrupt it without raising anything.
• Complexity: O(n) in the number of (segment x month) rows after melting and reindexing; the reindex step also allocates a full segment x month grid, so if segments are very high-cardinality relative to the number of months, that grid can be much bigger than the original data.

Direct answer

The root problem is that the tier column has dtype object/string, so pandas sorts and compares it lexicographically: Bronze, Gold, Platinum, Silver alphabetically, which has nothing to do with the real business ranking Bronze < Silver < Gold < Platinum. The fix is to cast the column to an ordered CategoricalDtype that encodes the actual business order, so sorting, comparisons, and groupby all use that order instead of the alphabet.

Structured elaboration

A pandas categorical dtype stores a column as a small, fixed list of allowed values (the categories) plus an integer code per row pointing into that list, instead of repeating the full string in every row. Setting ordered=True and listing the categories from lowest to highest tells pandas to use that sequence as the sort and comparison order, not the alphabet.

• Define the business order explicitly, lowest to highest, once: CategoricalDtype(categories=['Bronze','Silver','Gold','Platinum'], ordered=True).
• Cast the column with .astype(that_dtype).
• From that point on, sort_values, >/< comparisons, and groupby output order all follow the category order, not string order.

Worked example

import pandas as pd
from pandas.api.types import CategoricalDtype

df = pd.DataFrame({
    'account': ['A', 'B', 'C', 'D', 'E'],
    'tier': ['Gold', 'Bronze', 'Platinum', 'Silver', 'Gold'],
    'revenue': [500, 100, 900, 300, 450],
})

tier_order = CategoricalDtype(categories=['Bronze', 'Silver', 'Gold', 'Platinum'], ordered=True)
df['tier'] = df['tier'].astype(tier_order)

df.sort_values('tier')['tier'].tolist()
# ['Bronze', 'Silver', 'Gold', 'Gold', 'Platinum']   (would be Bronze, Gold, Gold, Platinum, Silver as plain strings)

df['tier'] > 'Silver'
# [True, False, True, False, True]   (True only for the Gold and Platinum rows)

df.groupby('tier', observed=True)['revenue'].mean()
# Bronze      100.0
# Silver      300.0
# Gold        475.0
# Platinum    900.0

Verified: all three outputs above were reproduced exactly, including that df['tier'] > 'Silver' correctly picks out only the Gold and Platinum accounts, something a plain-string comparison would get wrong ('Gold' > 'Silver' is False alphabetically).

Trade-offs and pitfalls

• Before the fix, df.sort_values('tier') on plain strings puts Platinum, the highest tier, ahead of Silver in the sorted output, misleading in any tier-ranked report. A filter meant to catch "above Silver" customers would silently miss the Gold accounts under plain-string comparison.
• What the ordered categorical buys beyond correctness: memory, each row stores a small integer code plus one shared lookup table of the distinct strings, instead of a separate string object per row, which matters on a column with millions of rows and only a handful of distinct values; speed, sorting, comparisons, and groupby operate on the integer codes internally rather than comparing strings row by row; a data-quality guardrail, a typo like 'platinum' (wrong case) or a stray value becomes NaN (not-a-number, pandas' missing-value marker) on cast rather than silently becoming a fifth tier that an alphabetical sort would place in some arbitrary wrong position.
• Complexity: the .astype() cast itself is O(n) (one pass to map each string to its integer code) and allocates a new column, so it briefly costs O(n) extra memory during the cast, not an in-place mutation. After the cast, sort/compare/groupby operations run over O(k) integer codes for k distinct categories instead of doing O(n) string comparisons each, so the ongoing cost of every later operation on that column drops, not just its one-time conversion cost. Memory at rest is O(k) for the category lookup table plus O(n) small integer codes, versus O(n) full string objects for the plain-string column.
• Edge cases: any value not in the categories list turns into NaN on cast (verified: casting 'platinum' lowercase against the four defined categories produces NaN, not an error), so validate or clean upstream values before casting rather than losing rows silently; adding a new tier later should use df['tier'].cat.add_categories(['Diamond']) rather than re-casting the whole column from scratch, since re-casting against a stale category list would NaN out every existing 'Diamond' value that arrived before the category was added; groupby(..., observed=True) avoids materializing every possible category combination as a row even when a category wasn't actually observed in the data, which matters once the category list grows.
• Categorical dtype is a net loss, not a win, on very high-cardinality columns (thousands-plus distinct values with few repeats), since the code-plus-lookup-table overhead can exceed plain string storage when there's little repetition to exploit.

Direct answer

.loc selects by label with an inclusive slice endpoint, .iloc selects by integer position with an exclusive slice endpoint, matching a Python list slice. Chained indexing, two indexing operations performed back to back such as df[df['age'] > 30]['score'] = value, is dangerous because under pandas' Copy-on-Write (CoW) engine the first indexing step always produces an independent copy, so the second step's assignment can land on that throwaway copy and leave the original df completely unchanged. The fix is to do the row selection and the column write in a single .loc call: df.loc[cond, 'col'] = value.

Approach

import pandas as pd

df = pd.DataFrame({
    'id': range(30),
    'name': [f"u{i}" for i in range(30)],
    'age': [20 + (i % 50) for i in range(30)],
    'signup_date': pd.date_range("2020-01-01", periods=30, freq='D'),
    'score': [i * 0.5 for i in range(30)],
})

# Rows 10-20
rows_loc = df.loc[10:20]        # 11 rows: labels 10 through 20, BOTH inclusive
rows_iloc = df.iloc[10:21]      # 11 rows: positions 10 through 20, stop 21 EXCLUDED

# Boolean condition
cond_rows = df.loc[df['age'] > 30]

# Columns by label and by position
cols_label = df.loc[:, ['id', 'age', 'score']]
cols_pos = df.iloc[:, [0, 2, 4]]

# DANGEROUS: chained indexing (single expression)
df[df['age'] > 30]['score'] = 999
# pandas raises the ChainedAssignmentError warning at this line, and df is untouched

# ALSO DANGEROUS, and silent: chained indexing across two statements
subset = df[df['age'] > 30]
subset['score'] = subset['score'] * 1.1
# NO warning at all here: subset is already an independent copy, so the assignment
# succeeds on subset only; df is never touched, and nothing signals that anything happened

# SAFE: row and column selection in one .loc call
df.loc[df['age'] > 30, 'score'] = df.loc[df['age'] > 30, 'score'] * 1.1

Key points

• rows_loc and rows_iloc are identical here (11 rows each) only because reaching position 20 by label and by "stop at 21" both land on the same rows; the inclusive/exclusive difference is what makes the two slice expressions (10:20 vs 10:21) necessary to get the same result.
• The single-expression chain (df[cond]['score'] = value) IS detected by pandas: it raises the ChainedAssignmentError warning at the point of assignment, and the original frame is left unchanged.
• The two-statement chain (subset = df[cond]; subset['score'] = ...) is NOT detected: by the time you mutate subset, pandas has no way to know you intended it to represent a piece of df, so the write lands cleanly on subset alone, with zero warning of any kind. This is the more dangerous shape precisely because nothing looks wrong.
• SettingWithCopyWarning was removed from pandas entirely as of 3.0. If you are reading older material describing it, that warning class no longer exists; the CoW-era failure modes are the ChainedAssignmentError warning (visible) and the silent two-statement case above (not visible).

Complexity

.loc label lookups are backed by the index's hash table, close to O(1) average per label, so selecting k rows by label is roughly O(k). .iloc positional lookups are direct offset arithmetic, O(k) for k positions. A boolean-mask selection like df['age'] > 30 is O(n) to build the mask over all n rows, plus O(k) to materialize the k matching rows. Chained assignment costs nothing extra algorithmically: the danger is purely correctness (the write silently lands nowhere useful), not performance.

Edge cases

• Duplicate index labels: df.loc[label] returns every row carrying that label, which can silently expand what you expected to be a single row; .iloc is unaffected since it always addresses exactly one position.
• No rows match the condition: .loc/boolean selection returns an empty (0-row) DataFrame, not an error, so a chained assignment onto an empty selection also fails silently with nothing to observe.
• An index that is itself integer-labeled but not a simple 0..n-1 range (for example after a prior filter without reset_index): df.loc[10:20] uses those labels, not positions, so it can select a completely different set of rows than df.iloc[10:20] would.
• Assigning into a genuinely one-off copy returned by another operation (e.g. a Series pulled out of a groupby result) via chained indexing hits the exact same footgun: the write updates the copy, not any object you can trace back to.

Trade-offs and pitfalls

Copy-on-Write makes the old "is this a view or a copy" ambiguity deterministic (df[cond] is now always a copy), but that determinism does not make chained assignment safe, it just changes how it fails: a warned-and-ineffective write for the single-expression form, and a silent-and-ineffective write for the two-statement form. Both leave df unmodified either way. The reliable pattern is always a single .loc/.iloc call that does the row and column selection together, or an explicit .copy() when you genuinely intend to work on an independent object rather than update the source.