Python Fundamentals and Core Syntax Questions

Comprehensive knowledge of core Python language features and syntax, including primitive and composite data types such as integer numbers, floating point numbers, strings, booleans, lists, dictionaries, sets, and tuples. Candidates should understand variable assignment and naming, operators for arithmetic, logical, and comparison operations, and control flow constructs including conditional statements and loops. Expect familiarity with function definition, invocation, parameter passing, return values, and scope rules, as well as common built in functions and idioms such as iteration utilities, list comprehensions, generator expressions, and basic functional utilities like map and filter. Candidates should demonstrate error and exception handling techniques and best practices for writing readable and maintainable code with modularization and clear naming. Practical skills include file input and output, working with common data formats such as comma separated values and JavaScript Object Notation, selecting appropriate data structures with attention to performance and memory characteristics, and applying memory efficient patterns for processing large data sets using iterators and generators. Familiarity with the standard library and common utilities for parsing and transforming data, and the ability to write small code snippets to solve algorithmic and data manipulation tasks, are expected.

MediumTechnical

86 practiced

Write Python 3 code that reads a large NDJSON file and writes a standard JSON array to an output file without loading all objects into memory. Ensure proper JSON formatting (commas between items, no trailing comma). Discuss how to make the write atomic and how to test this behavior.

Sample Answer

Approach: stream-read the NDJSON file line-by-line, parse each JSON object and write into an output file as a JSON array by writing the opening “[”, each item with commas only between items, then closing “]”. To make the write atomic, write to a temporary file on the same filesystem, fsync it, then os.replace() to atomically move it into place.

python

import json, os, tempfile

def ndjson_to_json_array(ndjson_path, out_path, encoding='utf-8'):
    dirpath = os.path.dirname(os.path.abspath(out_path)) or '.'
    with tempfile.NamedTemporaryFile('w', encoding=encoding, dir=dirpath, delete=False) as tmp:
        tmp_name = tmp.name
        tmp.write('[')
        first = True
        with open(ndjson_path, 'r', encoding=encoding) as src:
            for line in src:
                line = line.strip()
                if not line: 
                    continue
                obj = json.loads(line)  # may raise if invalid
                if not first:
                    tmp.write(',')
                else:
                    first = False
                json.dump(obj, tmp, ensure_ascii=False)
        tmp.write(']')
        tmp.flush()
        os.fsync(tmp.fileno())
    os.replace(tmp_name, out_path)  # atomic on most OSes

Key points:- Streams lines: O(1) memory.- Uses json.loads/json.dump to preserve proper formatting and escaping.- Atomicity: write to temp in same dir, fsync, then os.replace() (atomic rename). If cross-filesystem atomicity required, ensure temp file is on same filesystem.- Error handling: if parsing fails, tmp file remains — delete in exception handling in production.

Testing:- Unit test with small NDJSON inputs (including empty lines, invalid JSON, large single-line objects).- Simulate interruption by raising an exception mid-write and assert out_path isn't replaced.- Integration test: generate large NDJSON (~GB) and confirm output is valid JSON array (json.load) and memory stays low (monitor).- Test atomic replace: create a watcher that reads out_path concurrently; ensure it sees either old or complete new file, not a partial file.

MediumTechnical

82 practiced

Explain the Python Global Interpreter Lock (GIL): what it is, how it affects CPU-bound vs IO-bound workloads, and what options a data engineer has to parallelize Python code for CPU-heavy data transformations. Discuss trade-offs between threading, multiprocessing, and using native extensions.

Sample Answer

The GIL (Global Interpreter Lock) is a mutex in CPython that ensures only one native thread executes Python bytecode at a time. It simplifies memory management (e.g., refcounting) but limits parallel execution of Python-level code.

Impact on workloads- CPU-bound: GIL prevents true parallelism across threads — multiple threads will contend for the GIL so you won’t get near-linear speedup for CPU-heavy tasks.- IO-bound: Threads can help because I/O ops (file, network, blocking C calls) usually release the GIL, so while one thread waits the scheduler runs another — threading is effective here.

Options to parallelize CPU-heavy data transformations (trade-offs)1) multiprocessing (multiprocess module / concurrent.futures.ProcessPoolExecutor)- Pros: Each process has its own interpreter and GIL, so true parallel CPU utilization across cores.- Cons: Higher memory use (separate processes), inter-process communication (IPC) and serialization (pickle) overhead, higher startup latency. Good for embarrassingly parallel map-style transformations.

2) Native extensions / C/Cython / NumPy- Pros: Compute in C (NumPy, Pandas internals, Cython with nogil) bypasses Python bytecode and can release the GIL — very efficient and memory-friendly. Vectorized libraries handle large arrays with low overhead.- Cons: Extra dev complexity, native builds, careful memory and safety handling.

3) Threading- Pros: Low overhead, simple for I/O-bound tasks or when using libraries that release the GIL.- Cons: Not suitable for pure Python CPU work due to GIL.

4) Distributed/big-data frameworks (Spark, Dask, Ray)- Pros: Scale horizontally, handle data partitioning, fault tolerance; avoid GIL limits by running multiple worker processes or JVM-based execution (Spark).- Cons: Operational complexity, serialization, cluster cost.

When to choose:- Use threading for I/O-bound pipeline steps (downloads, DB calls).- Use multiprocessing or distributed frameworks for CPU-heavy, large-scale transforms.- Prefer native/vectorized libraries (NumPy/Pandas/Cython) when possible to get best performance with minimal IPC.

Practical tip: Profile first (cpu vs io), measure serialization costs, and prefer vectorized/native implementations before adding process-level or cluster complexity.

MediumTechnical

86 practiced

Write a Python 3 generator unique_everseen(iterable, key=None) that yields unique elements from the input preserving first-seen order. The function should work well for streaming sources. Explain memory trade-offs and how you would adapt the approach if keys are unhashable or if the unique key-space is too large for memory.

Sample Answer

To yield unique elements while preserving first-seen order for streaming sources, keep a set of seen keys and yield an item only if its key isn't in the set. This is O(1) amortized per item and memory grows with number of unique keys.

python

def unique_everseen(iterable, key=None):
    """
    Yield items from iterable, preserving order, skipping duplicates.
    key: optional function to compute comparison key for each item.
    Works as a generator for streaming sources.
    """
    seen = set()
    if key is None:
        for item in iterable:
            k = item
            if k not in seen:
                seen.add(k)
                yield item
    else:
        for item in iterable:
            k = key(item)
            if k not in seen:
                seen.add(k)
                yield item

Key points and trade-offs:- Time: average O(1) per item (set lookup/insert). Space: O(U) where U is unique key count.- For unhashable keys (e.g., dicts): convert to a hashable representation (tuple of items, json.dumps with sorted keys, or pickle). Example: k = tuple(sorted(d.items())). Beware cost of conversion.- If unique key-space is too large for RAM: use approximate structures (Bloom filter) to reduce memory with acceptable false positives (will drop some unique items); or use disk-backed/LMDB/set on external DB to store seen keys; or use LRU eviction if occasional duplicates are acceptable (memory bounded, may re-emit items). For distributed pipelines, coordinate via external store (Redis, RocksDB) or use partitioning by hash so each worker only tracks a subset.- Edge cases: streaming infinite iterables, key function raising exceptions, large or mutable items. Handle by sanitizing keys and monitoring memory.

MediumTechnical

114 practiced

Tell me about a time when you discovered a performance or correctness issue in a Python-based data pipeline. Use the STAR method: describe the Situation, the Task you were responsible for, the Actions you took (technical and non-technical), and the Results (quantified if possible). Emphasize how you prioritized fixes, communicated with stakeholders, and verified the improvement.

Sample Answer

Situation: At my previous company I owned a Python ETL pipeline that ingested user event logs into our Snowflake warehouse for analytics. After a product release, analysts reported that daily session counts dropped ~35% vs. expectations and some dashboards were showing inconsistent totals.

Task: I needed to identify whether this was a correctness bug or a performance issue, prioritize fixes with minimal disruption, and restore reliable daily metrics within 48 hours.

Action:- Quickly triaged logs and alerts: added verbose logging to the failing job and reran a backfill on a small date range to reproduce the discrepancy.- Discovered two issues: (1) a recent change to a streaming ingestion library introduced a subtle timezone-naive timestamp parsing bug in Python (correctness), and (2) a downstream Spark job was repeatedly retrying on small GC pauses because we were materializing large Pandas DataFrames in memory (performance).- Prioritized correctness first: rolled back the library change in staging, implemented a robust timestamp parsing utility using python-dateutil with explicit timezone normalization, and added unit tests.- Next, addressed performance: replaced large in-memory Pandas operations with Spark DataFrame transformations and windowing, added partitioning and persisted intermediate results to Parquet to avoid re-computation.- Communicated proactively: sent a status update to analytics and product every 6 hours, scheduled a 30-minute cross-team call to explain root cause and ETA, and coordinated a low-traffic window for deployment.- Added monitoring: end-to-end data quality checks (row counts, null rates) and alerting in our DAG scheduler.

Result:- Correctness fix restored session counts to within 2% of expected baselines within 12 hours. Performance changes reduced job runtime from 2.5 hours to 40 minutes (≈84% improvement) and reduced executor memory OOMs by 90%. Stakeholders regained confidence; dashboards stabilized and I documented the incident and preventive checks in our runbook.

EasyTechnical

71 practiced

Write increment_nested_counter(store, key1, key2, delta=1) in Python 3 that safely increments nested counters represented as a dict-of-dicts, leveraging collections.defaultdict to simplify initialization. Show how you would initialize such a store and explain why this pattern is useful in ETL aggregation code.

Sample Answer

To increment nested counters safely, use collections.defaultdict so inner dicts are auto-created. This avoids manual key checks and makes ETL aggregation loops concise and fast.

python

from collections import defaultdict

def make_store():
    # outer defaultdict maps key1 -> inner defaultdict(int)
    return defaultdict(lambda: defaultdict(int))

def increment_nested_counter(store, key1, key2, delta=1):
    """
    Increment store[key1][key2] by delta.
    store: defaultdict(lambda: defaultdict(int))
    """
    store[key1][key2] += delta

# Example usage
store = make_store()
increment_nested_counter(store, 'user_123', 'page_view')
increment_nested_counter(store, 'user_123', 'click', 3)
increment_nested_counter(store, 'user_456', 'page_view', 2)

# store now: {'user_123': {'page_view': 1, 'click': 3}, 'user_456': {'page_view': 2}}

Key points:- Simplicity: no if-checks for missing keys; clean in tight ETL loops.- Safety: inner counters default to 0 (int), preventing KeyError.- Time: O(1) per increment; Space: O(#distinct key pairs).- Useful in ETL when aggregating events by multiple dimensions (user × event, date × category) for fast in-memory rollups before batch writes.

Unlock Full Question Bank

Get access to hundreds of Python Fundamentals and Core Syntax interview questions and detailed answers.

Join thousands of developers preparing for their dream job.