Python Programming Questions

Why vectorized ops are faster:

• Vectorized NumPy/Pandas operations run in optimized C/Fortran loops avoiding Python per-element overhead.
• They leverage contiguous memory, CPU cache, SIMD (Single Instruction, Multiple Data: a CPU feature that applies one instruction to several values in a single step, instead of one value at a time), and multi-threaded BLAS (Basic Linear Algebra Subprogram, a standard library of fast, hardware-tuned matrix/vector math routines that numpy calls into) for numeric work.
• Example: adding two arrays uses a single C loop vs Python loop calling millions of Python operations.

Worked example, verified on CPython 3.12 (showing what actually gets computed, not a timing benchmark: exact speed depends on hardware and array size, so run time.perf_counter() yourself to see the real gap on your own machine rather than trusting a hardcoded number here):

import numpy as np

a = np.array([1.0, 2.0, 3.0, 4.0])
b = np.array([10.0, 20.0, 30.0, 40.0])

def add_loop(a, b):
    return [x + y for x, y in zip(a, b)]

def add_vectorized(a, b):
    return a + b

print(add_loop(a, b))
# [11.0, 22.0, 33.0, 44.0]
print(add_vectorized(a, b))
# [11. 22. 33. 44.]

Both compute the identical values; add_loop does it via four separate Python-level additions, each one dispatched, type-checked, and reference-counted by the interpreter, while add_vectorized's a + b dispatches once into a single compiled C loop over the two contiguous memory buffers, which is the mechanism, not a specific timing number, that makes the vectorized version pull ahead as array size grows.

When vectorization can be slower:

• Small arrays: overhead of creating intermediate arrays and function call overhead can dominate; a simple Python loop or in-place updates may be faster.
• Complex element-wise logic with branching: vectorization may require multiple large temporaries or complex masking, increasing memory bandwidth and runtime.

Where vectorization stops winning, made concrete: a piecewise function like "if x < 0, return 0; elif x < 10, return x; else return x**2" vectorizes via np.where, but every branch has to be computed for every element before the mask selects which result to keep:

x = np.array([-3.0, 5.0, 15.0])
result = np.where(x < 0, 0.0, np.where(x < 10, x, x ** 2))
print(result)
# [  0.   5. 225.]

Even though only one branch is "true" for any given element, x ** 2 is computed for all three elements (including -3.0 and 5.0, whose squared values are simply discarded by the outer np.where), and each np.where call allocates a full temporary array the size of x. For a genuinely small array or a function with many branches, that wasted work and temporary-array allocation can cost more than a plain Python loop that only evaluates the one branch each element actually needs; this is the concrete shape of "vectorization can be slower."

Example: computing a piecewise function with many branches can be slower vectorized due to multiple masks and temporary arrays; using numba (a library that compiles a plain Python function to machine code the first time it runs, letting a genuinely branchy per-element loop run at near-C speed without vectorizing it at all) for a compiled loop may be faster and more memory-efficient.

Vectorized mask for rows containing any NaN:

import numpy as np

# a is shape (N, M)
mask = np.isnan(a).any(axis=1)

Explanation:

• np.isnan(a) produces boolean array shape (N, M).
• .any(axis=1) reduces per row to True if any element is NaN. A numpy axis names which direction a reduction collapses across for a 2D array: axis=0 collapses DOWN each column (one result per column), axis=1 collapses ACROSS each row (one result per row); since the question asks for a per-row answer, one True/False per row of the (N, M) array, axis=1 is the one that produces exactly N results, matching the number of rows.

Worked example, verified on CPython 3.12:

a = np.array([[1.0, 2.0], [np.nan, 4.0], [5.0, 6.0]])
print(np.isnan(a))
# [[False False]
#  [ True False]
#  [False False]]
print(np.isnan(a).any(axis=1))
# [False  True False]

Row 0 ([1.0, 2.0]) has no NaN, so it reduces to False; row 1 ([nan, 4.0]) has one, so it reduces to True; row 2 ([5.0, 6.0]) has none, so False again, exactly the boolean mask a caller would use to filter out or flag the bad row.

This is fully vectorized, memory-efficient (single boolean array) and avoids Python loops. For very large arrays consider processing in chunks to limit peak memory.

Approach: build a directed graph of lock waits and detect cycles. Represent each lock as node; for each thread that holds locks A then requests B, add edges A -> B. A cycle implies possible deadlock.

def detect_deadlocks(thread_dump):
    # thread_dump: list of tuples (holds:list, wants:list)
    from collections import defaultdict, deque
    g=defaultdict(list)
    for holds, wants in thread_dump:
        for h in holds:
            for w in wants:
                g[h].append(w)
    # detect cycle via DFS
    visited=set(); stack=set()
    def dfs(u):
        if u in stack: return True
        if u in visited: return False
        visited.add(u); stack.add(u)
        for v in g.get(u, []):
            if dfs(v): return True
        stack.remove(u); return False
    return any(dfs(node) for node in g)

Note (a correction to the traversal, verified by actually running both versions): the loop reads g.get(u, []) rather than plain g[u]. Since g is a defaultdict(list), g[u] for a lock u that is only ever requested and never itself a holds key would silently insert a new empty entry for u into g the first time it is visited, mutating the dictionary while the outer for node in g is still iterating over it. Run directly, that raises RuntimeError: dictionary changed size during iteration on any thread_dump containing such a lock, for example the two-thread chain in the worked example just below. .get(u, []) reads the same list without ever creating a new key, so the traversal cannot perturb the structure it is iterating over.

Worked example, verified on CPython 3.12:

# Deadlock: A holds a lock and wants B's; B holds a lock and wants A's -> circular wait.
print(detect_deadlocks([(['A'], ['B']), (['B'], ['A'])]))
# True

# No deadlock: A wants what B holds, B wants what C holds -> a chain, not a cycle.
print(detect_deadlocks([(['A'], ['B']), (['B'], ['C'])]))
# False

For the deadlock case, the edge-building loop adds A -> B (from the first tuple) and B -> A (from the second), so g = {'A': ['B'], 'B': ['A']}. dfs('A') marks A visited and on-stack, follows the edge to B, marks B visited and on-stack, then follows B's edge back to A; since A is already on the current recursion stack, dfs returns True immediately, that revisit of an on-stack node is the cycle. For the no-deadlock case, g = {'A': ['B'], 'B': ['C']}; dfs('A') walks A -> B -> C, and C has no outgoing edges (g.get('C', []) returns [], since C never holds anything in this example) and is never back on the stack, so every branch returns False and no cycle is found.

Complexity: building edges O(E) where E = sum(hands*wants); cycle detection O(V+E).

Correctness: an edge A -> B means "a thread holding A is waiting to acquire B", i.e. that thread cannot proceed until whoever holds B releases it. Sufficiency (a cycle really does mean a stuck circular wait): if A -> B -> C -> A is a real cycle, the thread waiting on the A -> B edge cannot proceed until B frees up, but whoever holds B is itself stuck on the B -> C edge waiting for C, and whoever holds C is stuck on the C -> A edge waiting for A, which is held by the very first thread that is blocked; every thread on the cycle is waiting on the next one, forever, none of them can be the one to break the chain. Necessity (a stuck circular wait always shows up as a cycle here): if a set of threads really is deadlocked in a circular wait, then by definition each thread in that set holds one lock and is blocked wanting another lock in the same set, which is exactly one holds -> wants edge per thread; following those edges from any thread in the set must eventually revisit a thread already seen, since there are only finitely many threads in the set and every one of them has an outgoing edge to another member of the set, and a finite directed graph where every node in some subset has an outgoing edge back into that same subset necessarily contains a cycle. So "cycle in this graph" and "circular wait is possible" imply each other, given the model that a thread's current wait state is fully captured by its (holds, wants) entry.

Direct answer

Stability means elements that compare equal on the sort key keep their original relative order, and that is what makes multi-key sorting composable: sort by the least-significant key first, then by the most-significant key, and ties on the final sort automatically fall back to the order from the earlier pass. list.sort() sorts a list in place and returns None; sorted() returns a new list (and accepts any iterable, not just a list) leaving the original untouched. Use list.sort() when you own the list and no longer need the original order; use sorted() whenever you need to preserve the input or you are sorting something that is not already a list.

Structured elaboration

Why Timsort's stability matters for multi-key sorts. Timsort finds existing ascending/descending runs in the data, merges them with a stable merge (equal-key elements from different runs are never reordered relative to each other), and that stability is a load-bearing property, not an incidental detail: it is what lets you build an $n$-key sort out of $n$ single-key sorts instead of writing one comparator that juggles all keys at once.

• Multi-pass, relying on stability: sort by the least significant key first; each subsequent sort only needs to break ties left over from the previous pass, because the previous pass's relative order survives for anything the current pass considers equal.
• Composite key, single pass: sort(key=lambda r: (key1, key2, ...)) achieves the same result in one call and is usually simpler when all the keys are known up front; the multi-pass version is useful when the keys or the number of passes are decided dynamically, or when different passes need different comparators (e.g. one ascending, one descending, via separate reverse= flags rather than negating values).

list.sort() vs. sorted():

Returning None from list.sort() is a deliberate API choice (shared with other in-place mutators like list.append) to make it visually obvious at the call site that the operation is a mutation, not a value-producing expression; x = some_list.sort() is a very common and very wrong pattern, since x ends up None.

Worked example

records = [
    {"user": "A", "time": "2023-01-02", "score": 90},
    {"user": "A", "time": "2023-01-01", "score": 90},
    {"user": "B", "time": "2023-01-01", "score": 95},
]

r1 = list(records)
r1.sort(key=lambda r: r["time"])            # pass 1: least significant key
r1.sort(key=lambda r: r["score"], reverse=True)  # pass 2: most significant key
print(r1)

r2 = sorted(records, key=lambda r: (-r["score"], r["time"]))  # single composite-key pass
print(r2)

assert r1 == r2

a = [3, 1, 2]
print(a.sort(), a)          # None [1, 2, 3]

b = (3, 1, 2)
print(sorted(b), b)         # [1, 2, 3] (3, 1, 2)  -- tuple unchanged, sorted() works on it anyway

Both r1 and r2 come out identical: [B/2023-01-01/95, A/2023-01-01/90, A/2023-01-02/90]. Because the two A records tie on score, stability (in the multi-pass version) and the tuple's second element (in the composite-key version) both resolve that tie the same way, in favor of the earlier time.

Trade-offs & pitfalls

• Multi-pass only works because each pass is itself stable. If you ever swap in a sort that isn't stable partway through a multi-pass chain, the whole composability argument breaks; this is specific to Python's guarantee that both list.sort() and sorted() are always stable, not a property you get from sorting in general.
• Composite-key tuples need consistent direction per field. (-r["score"], r["time"]) negates score to get descending order while keeping time ascending; this trick only works cleanly for numeric keys. For a descending string key, or when combining ascending and descending on non-numeric fields, the multi-pass approach with separate reverse= flags is clearer than trying to invert a string key.
• Common wrong turn: calling sorted(some_list.sort()) (chaining the two), which sorts some_list in place, gets back None, and then crashes trying to call sorted(None). .sort() and sorted() solve the same underlying problem but are never meant to be composed with each other.
• sorted() on a large iterable still has to materialize it into a list internally; for a truly enormous or infinite stream, sorting isn't the right operation at all, you'd want an external/streaming approach (e.g. a k-way merge over pre-sorted chunks) instead of pulling everything into memory first.

Direct answer

pickle.load can execute arbitrary code during deserialization, since unpickling a crafted payload can invoke any importable callable the pickle stream references, with attacker-chosen arguments, not just rebuild plain data. Loading a pickle from a source that is not fully trusted is equivalent to running code you did not write. For anything crossing a trust boundary, prefer a format that cannot invoke arbitrary code by construction (ONNX, safetensors, or a plain parameters-only save) over pickle; if a pickle input must be accepted anyway, restrict what pickle.Unpickler is willing to construct with an explicit allowlist, since "don't unpickle untrusted data" is only a guideline until it is enforced in code.

Structured elaboration

Why pickle allows this: unpickling can construct any object by looking up and calling any importable callable named in the stream, with arguments also taken from the stream. That is the exploit surface: a malicious payload does not need to look like data at all, it can describe a call to something like a shell-executing function instead.

Additional risks beyond code execution: no version-compatibility guarantee across Python or library versions (a pickle referencing a custom class can fail to load once that class moves or changes shape), and no built-in integrity check, a corrupted or tampered file loads without any signal beyond a possible, unrelated exception.

Safer alternatives, named concretely:

• ONNX, an open, language-agnostic model interchange format: loading it replays a fixed graph of numeric operations, not arbitrary Python callables, so a malicious ONNX file's attack surface is limited to bugs in the runtime itself, not "call anything importable."
• safetensors, a data-only tensor storage format built specifically as a pickle replacement for model weights: it stores raw tensor buffers plus a JSON header describing shapes and dtypes, no arbitrary object graph, so there is no code-execution surface to exploit at all.
• Framework-native weights-only saves: current PyTorch versions support (and are moving toward defaulting to) a weights_only load mode that restricts unpickling to a safelist of tensor-related types, the same idea as the restricted unpickler below, built directly into the framework.
• joblib.dump/joblib.load is not a safer alternative, correcting a common assumption: joblib.load's own docstring states plainly that it "relies on the pickle module and can therefore execute arbitrary Python code" and "should therefore never be used to load files from untrusted sources" (verified directly against the installed joblib package, version 1.5.3). It is a convenience and performance layer over pickle for large NumPy (Numerical Python) arrays, not a security boundary; "we saved it with joblib instead of pickle" is not a valid security argument.

Worked example

Restricting deserialization with a custom Unpickler.find_class allowlist, verified on CPython 3.12:

import pickle
import io

class RestrictedUnpickler(pickle.Unpickler):
    '''Only construct an explicit allowlist of safe types; anything
    else raises instead of being built.'''
    ALLOWED = {
        ("builtins", "list"), ("builtins", "dict"), ("builtins", "set"),
        ("builtins", "tuple"), ("builtins", "str"), ("builtins", "int"),
        ("builtins", "float"), ("builtins", "bool"),
    }
    def find_class(self, module, name):
        if (module, name) not in self.ALLOWED:
            raise pickle.UnpicklingError(
                f"forbidden during restricted unpickling: {module}.{name}"
            )
        return super().find_class(module, name)

def restricted_loads(data: bytes):
    return RestrictedUnpickler(io.BytesIO(data)).load()

safe_payload = pickle.dumps({"a": 1, "b": [1, 2, 3]})
print(restricted_loads(safe_payload))
# {'a': 1, 'b': [1, 2, 3]}

import os
class Exploit:
    def __reduce__(self):
        return (os.system, ("echo pwned",))

dangerous_payload = pickle.dumps(Exploit())
try:
    restricted_loads(dangerous_payload)
except pickle.UnpicklingError as e:
    print("blocked:", e)
# blocked: forbidden during restricted unpickling: posix.system

Note the exact blocked identifier: posix.system, not os.system. Pickle records the callable's actual defining module, and os.system is really implemented in the posix module on this platform; an allowlist compares against that concrete recorded name, not the "friendly" import path a caller would normally type, which is exactly why it caught this payload regardless of which name was expected.

Trade-offs & pitfalls

• An allowlist unpickler still trusts the data inside those allowed types completely; a malicious payload built entirely from allowed types (an oversized nested list crafted to exhaust memory, or a string later passed unsanitized into a shell command downstream) still gets through. The allowlist stops arbitrary code execution during deserialization specifically, it is not a general untrusted-input sanitizer.
• The allowlist has to be as narrow as what is actually expected; a broad, convenient allowlist that includes common containers is also, by the same token, everything an attacker's crafted payload is free to use.
• Corrected claim: joblib does not add security over pickle (verified from joblib's own docstring warning above); it is a performance and convenience layer for large NumPy arrays, not a security boundary.
• Integrity checks (a hash, or a real cryptographic signature over the artifact) protect against tampering in transit or storage, a separate concern from restricting what a payload is allowed to construct: a signature says the bytes are the ones that were produced, the restricted unpickler says those bytes, even if trusted, cannot smuggle in an unexpected callable. The two are complementary, neither substitutes for the other.