Python Programming Questions

Direct answer

Python resolves a name by searching four scopes in order: Local (the current function body), Enclosing (any outer function that defines this one, for nested functions), Global (the module), Built-in (names like len and range). Lookup stops at the first scope that has the name. The catch is assignment, not lookup: if a name is assigned anywhere inside a function body, Python treats it as local to that function for the whole body, even before the assignment line runs, unless you declare it nonlocal (binds to the nearest enclosing function scope) or global (binds to module scope).

Structured elaboration

The four scopes, outside in:

• Local: parameters and names assigned inside the current function.
• Enclosing: names in any outer function that lexically contains this one (relevant only for nested functions); this is what makes closures possible.
• Global: names bound at module level.
• Built-in: the builtins module (len, range, print, and so on), checked last.

Why nonlocal exists: Python decides, at compile time, whether a name inside a function is local by scanning the function body for any assignment to that name. Reading an enclosing variable works fine without any keyword. The moment you assign to a name that lives in an enclosing scope, Python's default is to create a brand-new local variable with that name instead of reaching outward, which shadows the outer one and usually breaks the intended logic. nonlocal tells the compiler: this name is not local, resolve it in the nearest enclosing function scope (skipping straight past to global is a compile error if no enclosing binding exists). This differs from global, which always targets the module scope regardless of nesting depth.

The closure "late binding" gotcha: a nested function does not capture a variable's value at definition time, it captures the variable itself and reads it whenever the nested function actually runs. If several closures share a loop variable, they all read whatever value that variable holds when they are finally called, which is usually the loop's last value.

Worked example

Where nonlocal is required, verified on CPython 3.12:

def make_accumulator():
    total = 0
    def add(x):
        nonlocal total   # without this, `total += x` raises UnboundLocalError
        total += x
        return total
    return add

acc = make_accumulator()
print(acc(10))   # 10
print(acc(5))    # 15
print(acc(1))    # 16

What happens if you drop nonlocal (exact CPython 3.12 message):

def make_broken_accumulator():
    total = 0
    def add(x):
        total += x   # `total` is assigned here, so LEGB makes it local to add()
        return total
    return add

broken = make_broken_accumulator()
try:
    broken(10)
except UnboundLocalError as e:
    # expected: total is assigned inside add(), so LEGB makes it local for the
    # whole function body, and the read on the += happens before any assignment runs
    print(f'raises as expected: UnboundLocalError: {e}')

The late-binding closure gotcha, same mechanism (enclosing-scope lookup happens at call time, not definition time):

funcs = []
for i in range(4):
    funcs.append(lambda: i)
print([f() for f in funcs])   # [3, 3, 3, 3] -- all read the final value of i

# fix: bind the current value as a default argument, evaluated at definition time
funcs2 = []
for i in range(4):
    funcs2.append(lambda i=i: i)
print([f() for f in funcs2])  # [0, 1, 2, 3]

Trade-offs & pitfalls

• Reaching for nonlocal on every enclosing read is a common overcorrection: it is only needed when you assign to the name, never for reading it.
• nonlocal cannot create a new binding; if no enclosing function scope already has that name, Python raises a SyntaxError at compile time, it will not silently fall through to global.
• The late-binding gotcha bites hardest with lambda inside loops (event handlers, callbacks passed to a scheduler, list comprehensions of functions); the default-argument fix works because default values are evaluated once, at function-definition time, not at call time.
• Mutable enclosing state (a list or dict) sidesteps nonlocal entirely, since mutating the object's contents is not the same as rebinding the name, but that trades an explicit rebind for a shared mutable object, which is its own source of bugs in concurrent code.

Approach: schedule tasks, wait with timeout, cancel pending tasks after timeout, re-raise exceptions from completed tasks. (A coroutine is what you get back from calling an async def function; calling it does not run it yet, asyncio.create_task(c) is what actually schedules a coroutine to start running concurrently with everything else.)

Implementation:

import asyncio

async def gather_with_timeout(coros, timeout):
    tasks = [asyncio.create_task(c) for c in coros]
    done, pending = await asyncio.wait(tasks, timeout=timeout)
    # cancel pending
    for t in pending:
        t.cancel()
    # await cancellations to finish
    if pending:
        await asyncio.gather(*pending, return_exceptions=True)
    # propagate exceptions from done
    results = []
    for t in done:
        exc = t.exception()
        if exc:
            # re-raise first exception
            raise exc
        results.append(t.result())
    return tuple(results), tuple(pending)

# Usage example
async def main():
    async def work(n):
        await asyncio.sleep(n); return n
    done, pending = await gather_with_timeout([work(0.1), work(2)], timeout=1)
    print(done, pending)

# asyncio.run(main())

Notes: ensures pending tasks are cancelled and awaited to avoid warnings; exceptions from completed tasks propagate.

Running it: uncommenting asyncio.run(main()) and adding print(len(pending), pending[0].cancelled()) right after print(done, pending) shows what actually happens: work(0.1) finishes well inside the 1-second timeout and work(2) does not, so done prints as (0.1,) (the tuple of results from the finished work(0.1)) and the follow-up print shows 1 True, one pending task, and it reports cancelled() == True, confirming the still-running work(2) really was cancelled rather than left dangling.

Direct answer

pickle.load can execute arbitrary code during deserialization, since unpickling a crafted payload can invoke any importable callable the pickle stream references, with attacker-chosen arguments, not just rebuild plain data. Loading a pickle from a source that is not fully trusted is equivalent to running code you did not write. For anything crossing a trust boundary, prefer a format that cannot invoke arbitrary code by construction (ONNX, safetensors, or a plain parameters-only save) over pickle; if a pickle input must be accepted anyway, restrict what pickle.Unpickler is willing to construct with an explicit allowlist, since "don't unpickle untrusted data" is only a guideline until it is enforced in code.

Structured elaboration

Why pickle allows this: unpickling can construct any object by looking up and calling any importable callable named in the stream, with arguments also taken from the stream. That is the exploit surface: a malicious payload does not need to look like data at all, it can describe a call to something like a shell-executing function instead.

Additional risks beyond code execution: no version-compatibility guarantee across Python or library versions (a pickle referencing a custom class can fail to load once that class moves or changes shape), and no built-in integrity check, a corrupted or tampered file loads without any signal beyond a possible, unrelated exception.

Safer alternatives, named concretely:

• ONNX, an open, language-agnostic model interchange format: loading it replays a fixed graph of numeric operations, not arbitrary Python callables, so a malicious ONNX file's attack surface is limited to bugs in the runtime itself, not "call anything importable."
• safetensors, a data-only tensor storage format built specifically as a pickle replacement for model weights: it stores raw tensor buffers plus a JSON header describing shapes and dtypes, no arbitrary object graph, so there is no code-execution surface to exploit at all.
• Framework-native weights-only saves: current PyTorch versions support (and are moving toward defaulting to) a weights_only load mode that restricts unpickling to a safelist of tensor-related types, the same idea as the restricted unpickler below, built directly into the framework.
• joblib.dump/joblib.load is not a safer alternative, correcting a common assumption: joblib.load's own docstring states plainly that it "relies on the pickle module and can therefore execute arbitrary Python code" and "should therefore never be used to load files from untrusted sources" (verified directly against the installed joblib package, version 1.5.3). It is a convenience and performance layer over pickle for large NumPy (Numerical Python) arrays, not a security boundary; "we saved it with joblib instead of pickle" is not a valid security argument.

Worked example

Restricting deserialization with a custom Unpickler.find_class allowlist, verified on CPython 3.12:

import pickle
import io

class RestrictedUnpickler(pickle.Unpickler):
    '''Only construct an explicit allowlist of safe types; anything
    else raises instead of being built.'''
    ALLOWED = {
        ("builtins", "list"), ("builtins", "dict"), ("builtins", "set"),
        ("builtins", "tuple"), ("builtins", "str"), ("builtins", "int"),
        ("builtins", "float"), ("builtins", "bool"),
    }
    def find_class(self, module, name):
        if (module, name) not in self.ALLOWED:
            raise pickle.UnpicklingError(
                f"forbidden during restricted unpickling: {module}.{name}"
            )
        return super().find_class(module, name)

def restricted_loads(data: bytes):
    return RestrictedUnpickler(io.BytesIO(data)).load()

safe_payload = pickle.dumps({"a": 1, "b": [1, 2, 3]})
print(restricted_loads(safe_payload))
# {'a': 1, 'b': [1, 2, 3]}

import os
class Exploit:
    def __reduce__(self):
        return (os.system, ("echo pwned",))

dangerous_payload = pickle.dumps(Exploit())
try:
    restricted_loads(dangerous_payload)
except pickle.UnpicklingError as e:
    print("blocked:", e)
# blocked: forbidden during restricted unpickling: posix.system

Note the exact blocked identifier: posix.system, not os.system. Pickle records the callable's actual defining module, and os.system is really implemented in the posix module on this platform; an allowlist compares against that concrete recorded name, not the "friendly" import path a caller would normally type, which is exactly why it caught this payload regardless of which name was expected.

Trade-offs & pitfalls

• An allowlist unpickler still trusts the data inside those allowed types completely; a malicious payload built entirely from allowed types (an oversized nested list crafted to exhaust memory, or a string later passed unsanitized into a shell command downstream) still gets through. The allowlist stops arbitrary code execution during deserialization specifically, it is not a general untrusted-input sanitizer.
• The allowlist has to be as narrow as what is actually expected; a broad, convenient allowlist that includes common containers is also, by the same token, everything an attacker's crafted payload is free to use.
• Corrected claim: joblib does not add security over pickle (verified from joblib's own docstring warning above); it is a performance and convenience layer for large NumPy arrays, not a security boundary.
• Integrity checks (a hash, or a real cryptographic signature over the artifact) protect against tampering in transit or storage, a separate concern from restricting what a payload is allowed to construct: a signature says the bytes are the ones that were produced, the restricted unpickler says those bytes, even if trusted, cannot smuggle in an unexpected callable. The two are complementary, neither substitutes for the other.

Direct answer

Python has four argument-passing shapes: positional (matched by order), keyword (matched by name), default values (make a parameter optional), and the catch-alls *args (extra positionals, collected as a tuple) and **kwargs (extra keywords, collected as a dict). They combine in one fixed signature order: positional-or-keyword params, then *args, then keyword-only params (which may have defaults), then **kwargs.

Structured elaboration

Signature order and what each piece means:

def process_batch(data, transform, *args, normalize=True, batch_size=100, **kwargs):
    ...

• data, transform: positional-or-keyword, required. Callers usually pass these positionally because order is natural (the data, then what to do to it).
• *args: soaks up any extra positional arguments beyond data and transform, exposed inside the function as a tuple. Anything named after a bare *args (or a bare *) becomes keyword-only, meaning it cannot be passed positionally.
• normalize, batch_size: keyword-only parameters with defaults, so they are optional and callers must name them.
• **kwargs: soaks up any remaining keyword arguments not already named in the signature, exposed as a dict.

When each pattern is idiomatic versus overkill:

• Positional: idiomatic for the 1-2 arguments every call needs, in an order a reader can memorize. Overkill once a function takes more than 3-4 positional arguments; callers start passing things in the wrong order silently.
• Keyword + default: idiomatic for tunable, optional behavior (batch_size, normalize) where the default covers the common case and the name documents intent at the call site.
• *args: idiomatic for genuinely variadic operations (print(*values), a sum_all(*numbers)) or for forwarding positional arguments through a wrapper. Overkill as a substitute for a real, named parameter list; it hides the function's actual contract from anyone reading a call site.
• **kwargs: idiomatic for forwarding options to an underlying API you do not want to re-declare (backend-specific keys like shuffle or device), or for a plugin-style function whose exact options vary by caller. Overkill as a way to avoid deciding what a function's parameters actually are; unvalidated **kwargs swallows typos (nomralize=True) silently instead of raising TypeError.

Worked example

def process_batch(data, transform, *args, normalize=True, batch_size=100, **kwargs):
    results = []
    for item in data:
        value = transform(item, *args)
        if normalize:
            value = value / batch_size
        results.append(value)
    return results, kwargs

out, extra = process_batch(
    [10, 20, 30],
    lambda x: x * 2,
    normalize=True,
    batch_size=10,
    shuffle=True,
    device="cpu",
)
print(out)    # [2.0, 4.0, 6.0]
print(extra)  # {'shuffle': True, 'device': 'cpu'}

Here data and transform are passed positionally because every call needs them and the order is obvious; normalize and batch_size are named because they are optional tuning knobs; shuffle and device are not declared parameters at all, they arrive through **kwargs and the function forwards or ignores them as extra.

Trade-offs & pitfalls

• A bare * with no name (def f(a, *, b):) is legal and forces everything after it to be keyword-only without collecting extra positionals; use it to make a signature self-documenting even when you do not need *args.
• **kwargs that is never validated is a common production bug: transform_data(dat=my_df) (a typo of data=) silently lands in kwargs instead of raising TypeError: missing required argument. Pop and check known keys explicitly (kwargs.pop("shuffle", False)) if you accept **kwargs, or avoid it and declare the real parameter list.
• Order matters and is enforced by the interpreter, not just style: positional-or-keyword parameters must come before *args, which must come before keyword-only parameters, which must come before **kwargs; writing them out of order is a SyntaxError at definition time.
• Mutable defaults are the sharpest edge here (def f(items=[]):): a default value is evaluated once, when the def statement runs, not on each call, so a mutable default is shared and can accumulate state across unrelated calls.

Go vocabulary, translated for a Python reader: a goroutine is Go's lightweight, concurrently-running function, similar in spirit to a Python thread but far cheaper to start and typically used in much greater numbers in real Go code; a channel is a typed, thread-safe queue you send values into and receive values out of, roughly like a queue.Queue shared between Python threads, except the compiler enforces the type of what flows through it; a WaitGroup is a counter that lets the caller block until every worker goroutine has signaled it is done, similar to calling .join() on a list of Python Thread objects; ctx (short for context) carries a shared cancellation signal through the call tree, and ctx.Done() returns a channel that closes the moment that signal fires, so any goroutine can cheaply check "has someone asked everything to stop?" without polling a shared boolean, comparable to checking a Python threading.Event.

Structure

I’d use a bounded worker pool. A feeder sends files with an index into a jobs channel. Each worker validates one file, sends {index, result, err} to a results channel, and watches ctx.Done() so context cancellation, meaning a cooperative stop signal, is fast.

Ordering
A single collector keeps nextIndex and a map of out-of-order results. If result 7 arrives before 6, store it until 6 is ready, then flush in submission order.

Fatal parse error
If a worker sees an unrecoverable parse error, it sends the error and calls cancel(). That stops the feeder, makes workers exit on ctx.Done(), and prevents new work from starting.

Shutdown

• close jobs after feeding stops
• WaitGroup waits for workers
• close results after workers finish
• collector drains until closed or canceled

Worked example: files 1, 2, 3, 4 arrive. If file 3 has a fatal parse error, 1 and 2 can still be emitted, 4 is never started, and the summary reports the error immediately.

Shape of the code (a sketch of the structure, not a full compiled program):

type job struct {
    index int
    path  string
}
type result struct {
    index  int
    output string
    err    error
}

func run(ctx context.Context, paths []string, numWorkers int) []result {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()

    jobs := make(chan job)
    results := make(chan result)
    var wg sync.WaitGroup

    for w := 0; w < numWorkers; w++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for j := range jobs {
                out, err := validate(j.path)
                if err != nil && isFatal(err) {
                    cancel()
                }
                results <- result{index: j.index, output: out, err: err}
            }
        }()
    }

    go func() {
        for i, p := range paths {
            select {
            case jobs <- job{index: i, path: p}:
            case <-ctx.Done():
                close(jobs)
                return
            }
        }
        close(jobs)
    }()

    go func() {
        wg.Wait()
        close(results)
    }()

    return collectInOrder(results)
}

The jobs and results lines are the channel declarations; the three go func() { ... }() blocks are the goroutine launches, one pool of workers, one feeder, one closer. select { case jobs <- job{...}: ... case <-ctx.Done(): ... } is how the feeder stays responsive to cancellation even while trying to send a job that a worker isn't ready to receive yet, instead of blocking forever on a full channel after a fatal error.

This gives parallelism, ordered output, and fast failure without deadlocks.