Functional Programming Questions

Direct answer

The imperative loop keeps a Set of what it has seen and an output array, and appends an item when it is new. The functional rewrite is a fold (reduce: combine a list into one value by carrying an accumulator through each element) whose accumulator holds the same two things. The honest trade-off is that "no mutation at all" and "linear time" pull in opposite directions in JavaScript. A fold that returns a fresh array on every step ([...acc, x]) never mutates anything but is O(n squared), because it copies the accumulator each time. A fold whose accumulator is mutated only inside the function, and never the caller's array, keeps the pure interface and O(n) time. For this exact job the shortest correct answer is [...new Set(xs)], since a Set iterates in insertion order.

The versions

import assert from "node:assert/strict";

// Imperative: mutable Set plus an output array
function uniqueLoop(xs) {
  const seen = new Set();
  const out = [];
  for (const x of xs) {
    if (!seen.has(x)) { seen.add(x); out.push(x); }
  }
  return out;
}

// Functional, version 1: fold with a spread accumulator. Pure, but copies on every step.
let copied = 0;
const uniqueSpread = (xs) =>
  xs.reduce((acc, x) => {
    if (acc.includes(x)) return acc;
    copied += acc.length;                // elements copied by the spread below
    return [...acc, x];
  }, []);

// Functional, version 2: fold whose accumulator is a pair [seenSet, result], mutated only
// inside the function. The caller's data is never touched, and the cost is linear.
const uniqueFold = (xs) =>
  xs.reduce(
    (acc, x) => {
      if (!acc.seen.has(x)) { acc.seen.add(x); acc.out.push(x); }
      return acc;
    },
    { seen: new Set(), out: [] }
  ).out;

// Built-in shortcut: Set keeps insertion order
const uniqueSet = (xs) => [...new Set(xs)];

// Unique by a key (objects): the first item seen for each key wins
const uniqueBy = (xs, key) =>
  [...xs.reduce((m, x) => (m.has(key(x)) ? m : m.set(key(x), x)), new Map()).values()];

const input = [3, 1, 3, 2, 1, NaN, NaN, 0, -0];
const frozen = Object.freeze([...input]);
for (const [name, f] of [["loop", uniqueLoop], ["spread", uniqueSpread], ["fold", uniqueFold], ["set", uniqueSet]]) {
  console.log(name.padEnd(7), f(frozen));
}
assert.deepEqual(uniqueLoop(input), uniqueFold(input));
console.log("input unchanged:", input);

// Cost: count work on n distinct items (worst case for the spread version)
const n = 2000;
const distinct = Array.from({ length: n }, (_, i) => i);
copied = 0;
uniqueSpread(distinct);
console.log(`n=${n}: elements copied by spread version =`, copied, "| n(n-1)/2 =", (n * (n - 1)) / 2);

// Includes-based membership checks cost up to acc.length comparisons each; Set.has does not scan
let comparisons = 0;
const countingIncludes = (arr, x) => { for (const y of arr) { comparisons++; if (y === x) return true; } return false; };
const acc = [];
for (const x of distinct) { if (!countingIncludes(acc, x)) acc.push(x); }
console.log("linear-scan comparisons:", comparisons);

console.log(uniqueBy([{ id: 1, v: "a" }, { id: 2, v: "b" }, { id: 1, v: "c" }], (o) => o.id));

Output (Node 22, one run of exactly this file):

loop    [ 3, 1, 2, NaN, 0 ]
spread  [ 3, 1, 2, NaN, 0 ]
fold    [ 3, 1, 2, NaN, 0 ]
set     [ 3, 1, 2, NaN, 0 ]
input unchanged: [
    3,   1, 3,  2, 1,
  NaN, NaN, 0, -0
]
n=2000: elements copied by spread version = 1999000 | n(n-1)/2 = 1999000
linear-scan comparisons: 1999000
[ { id: 1, v: 'a' }, { id: 2, v: 'b' } ]

Reading the output

• All four variants return [3, 1, 2, NaN, 0] for the same input: first-seen order, with NaN kept once and 0 and -0 treated as the same value. That is because Set and Array.prototype.includes compare with SameValueZero (MDN: NaN equals NaN, 0 equals -0, objects are compared by reference). indexOf would not find NaN, so an indexOf-based version would keep every NaN.
• Each variant ran on a frozen copy of the input (Object.freeze), so any attempt to push to or reorder the caller's array would have thrown a TypeError in this strict-mode module. None did, and the original input prints unchanged afterwards.
• uniqueBy shows the same fold with a Map keyed by a function of the item, where the first item for each key wins (id: 1 keeps "a", not "c"): the usual form when items are objects, since two distinct objects with equal content are never equal by reference.

Time and space

The measured counts for n = 2000 distinct items are in the output: the spread version copied 1,999,000 elements, which equals n(n-1)/2 (0 + 1 + ... + 1999), and a linear-scan membership test did 1,999,000 comparisons for the same reason. Those are counts, not timings, so they hold on any machine. With n = 20,000 the same formula gives 199,990,000 each, a hundredfold increase for a tenfold larger input, which is what O(n squared) means.

"Average" matters for the Set rows: the language specification only requires Set access to be sublinear on average (a hash table gives O(1) on average), so say "O(n) expected", not "guaranteed".

Edge cases

• Empty input returns [].
• Items that are objects are unique by reference. Use uniqueBy with a stable key for "same id" semantics.
• Mixed 1 and "1" are different values.
• Very large arrays: the spread version also creates a garbage array per step; the loop and local-mutation fold do not.

Judgement

The fold with local mutation is "functional" at the boundary: same input gives the same output, the caller's array is untouched, and nothing escapes the function. That is the useful definition of purity for JavaScript in production. Purely persistent structures (which share structure between versions) can get a cheap non-mutating version, but they are not built in. Recommend [...new Set(xs)] for primitives, uniqueBy for objects, and the plain loop if a teammate finds the fold harder to read than the loop; the two have the same cost.

Direct answer

Use an optional (Option / Maybe, or T | undefined in TypeScript) when the only thing the caller needs to know is "there is a value or there isn't", such as a lookup by key. Use a result (Result / Either) when failure has a reason the caller can act on or display (which field, what code). Use throwing for failures the caller is not expected to handle locally: programmer errors, broken invariants, and unexpected conditions that should unwind to a boundary such as a request handler. The functional styles put the failure in the return type, so the compiler makes callers deal with it, and they compose through a function usually called andThen / flatMap / bind that runs the next step only if the previous one succeeded.

The three styles side by side

Vocabulary used in the code: <T, U, E> are generics, placeholders for types filled in at each use (T the success value going in, U the success value coming out, E the error type). Result is a discriminated union: a union whose members share a field (ok) with a different literal value in each, so checking r.ok narrows the type, meaning the compiler then knows which member r is and only allows r.value after r.ok is true. never is the type with no values; Result<T, never> therefore says "a success whose error side can never happen", which is why ok(5) can be used wherever a Result<number, ParseError> is expected.

type Result<T, E> = { ok: true; value: T } | { ok: false; error: E };
const ok = <T>(value: T): Result<T, never> => ({ ok: true, value });
const err = <E>(error: E): Result<never, E> => ({ ok: false, error });

// andThen: run the next step only if the previous one succeeded
const andThen = <T, U, E>(r: Result<T, E>, f: (t: T) => Result<U, E>): Result<U, E> => (r.ok ? f(r.value) : r);
const mapR = <T, U, E>(r: Result<T, E>, f: (t: T) => U): Result<U, E> => (r.ok ? ok(f(r.value)) : r);

type ParseError = { code: "not_a_number" | "out_of_range" | "missing"; input: string };

const parseAge = (s: string): Result<number, ParseError> => {
  const n = Number(s);
  return s.trim() === "" || Number.isNaN(n) ? err({ code: "not_a_number", input: s }) : ok(n);
};
const checkRange = (n: number): Result<number, ParseError> =>
  n >= 0 && n <= 120 ? ok(n) : err({ code: "out_of_range", input: String(n) });

// Option-style: absence only, no reason
const findUser = (db: Record<string, { age: string }>, id: string): { age: string } | undefined => db[id];
const db = { u1: { age: "34" }, u2: { age: "abc" }, u3: { age: "200" } };

// Throwing version: caller must know to try/catch, and the type does not say it can fail
function ageOrThrow(s: string): number {
  const r = andThen(parseAge(s), checkRange);
  if (!r.ok) throw new Error(r.error.code);
  return r.value;
}

// Result chain: no nested if checks
function ageOfUser(id: string): Result<number, ParseError> {
  const u = findUser(db, id);
  if (u === undefined) return err({ code: "missing", input: id });
  return andThen(parseAge(u.age), checkRange);
}
for (const id of ["u1", "u2", "u3", "u9"]) console.log(id, JSON.stringify(ageOfUser(id)));
console.log("mapR:", JSON.stringify(mapR(ageOfUser("u1"), (n) => n + 1)));

// Option-style chain with optional chaining and ??
const city: string = ({ a: { addr: { city: "Oslo" } } } as Record<string, any>)["a"]?.addr?.city ?? "unknown";
const none: string = ({ } as Record<string, any>)["b"]?.addr?.city ?? "unknown";
console.log(city, none);

// Throwing: the failure path is invisible to the type
try { ageOrThrow("200"); } catch (e) { console.log("caught:", (e as Error).message); }

// Collect all errors instead of stopping at the first
const validateAll = (ids: string[]) => {
  const rs = ids.map(ageOfUser);
  const errors = rs.flatMap((r) => (r.ok ? [] : [r.error.code]));
  return errors.length ? err(errors) : ok(rs.flatMap((r) => (r.ok ? [r.value] : [])));
};
console.log(JSON.stringify(validateAll(["u1", "u2", "u3"])));
console.log(JSON.stringify(validateAll(["u1"])));

Compiled with tsc 5.9.3 --strict --target es2022 and run in a node:22 container it prints:

u1 {"ok":true,"value":34}
u2 {"ok":false,"error":{"code":"not_a_number","input":"abc"}}
u3 {"ok":false,"error":{"code":"out_of_range","input":"200"}}
u9 {"ok":false,"error":{"code":"missing","input":"u9"}}
mapR: {"ok":true,"value":35}
Oslo unknown
caught: out_of_range
{"ok":false,"error":["not_a_number","out_of_range"]}
{"ok":true,"value":[34]}

Comparing them

When Result over Option: as soon as two different failures need different handling or different messages. In the output, u2, u3 and u9 all fail, but one is bad text, one is out of range and one is missing, and a UI shows different messages for each. An Option would give three identical None values and the reason would be lost.

Combining several calls without nested checks

The nested-null version is if (a) { if (b) { if (c) ... } }. With a result type, ageOfUser is a straight line: look up the user, andThen(parseAge(...), checkRange). The first ok: false skips the rest and is returned unchanged, so the failure arrives at the caller with no extra code. mapR transforms the success value without touching a failure (the mapR: ... 35 line). The difference between the two: mapR takes a function that returns a plain value (n => n + 1), while andThen takes a function that itself returns a Result (such as checkRange, which can fail). Using mapR with checkRange would wrap a result in a result, Result<Result<number, ParseError>, ParseError>, and callers would have to check twice; andThen flattens that into one layer. For Option-like data, optional chaining ?. (stop and give undefined as soon as something on the path is null or undefined) and nullish coalescing ?? (use the right side when the left is null or undefined) do the same job for property paths: Oslo is found, and the missing path returns the "unknown" default. In the example the object literal is cast with as Record<string, any> only so TypeScript allows indexing it by an arbitrary string key; the cast is not part of the technique.

When you want all the problems instead of the first one (form validation), you do not chain; you run each check independently and collect the failures, as validateAll does for u1, u2 and u3: the output lists not_a_number and out_of_range. This is the usual reason a validation layer is not a plain andThen chain: the chain is short-circuiting by design (it stops at the first failure and skips every later step).

Haskell contrast

Reading the Haskell below: Either String Int is a value that is either Left a String (the failure, by convention) or Right an Int (the success). readMaybe s returns Just n when s parses and Nothing otherwise, and maybe d f m means "if m is Nothing use d, otherwise apply f to what is inside", so maybe (Left ...) Right (readMaybe s) turns Nothing into a Left message and Just n into Right n. ++ joins strings, show turns a number into text, and the | lines in checkRange are guards, tried top to bottom.

import Text.Read (readMaybe)

parseAge :: String -> Either String Int
parseAge s = maybe (Left ("not a number: " ++ s)) Right (readMaybe s)

checkRange :: Int -> Either String Int
checkRange n | n >= 0 && n <= 120 = Right n
             | otherwise = Left ("out of range: " ++ show n)

main :: IO ()
main = do
  print (parseAge "34" >>= checkRange)
  print (parseAge "200" >>= checkRange)
  print (parseAge "abc" >>= checkRange)
  print (do { a <- parseAge "30"; b <- parseAge "40"; checkRange (a + b) })

Right 34
Left "out of range: 200"
Left "not a number: abc"
Right 70

(runghc in a haskell:9.6-slim container.) >>= (read "bind") is andThen for Either String: given a result and a function that returns the next result, it runs the function only on Right and passes a Left through unchanged. A type with such a chaining operation is what Haskell calls a monad; in practice that means it "supports andThen". In the do block, a <- parseAge "30" means "run parseAge; if it is Right, call its value a and continue; if Left, stop with that Left", so the block is the same chain written as sequential statements; the first Left stops it. Rust's Result with ?, Kotlin's Result, and Scala's Either are the same idea.

Trade-offs and pitfalls

• Throwing is not wrong: for an invariant violation or a failure that every caller would just re-throw, a result type adds ceremony with no benefit. Mix the styles deliberately: wrap I/O boundaries that throw (fetch, JSON.parse) into results at the edge, and let genuine bugs throw.
• A result type is only as good as its error type. Result<T, string> loses the structure that the UI needs; use a discriminated union of error codes as above.
• In plain JavaScript there is no compiler to make callers check ok, so the benefit comes from TypeScript (or from team convention). With strict, reading r.value without narrowing on r.ok is a compile error.
• Annotate an Option-style return explicitly. Indexing a Record<string, T> is typed as always present unless noUncheckedIndexedAccess is on, so without the | undefined on findUser the call findUser(db, "u9").age compiles under --strict and fails at run time; with it, the compiler reports Object is possibly 'undefined'.
• Do not swallow errors by converting every failure to undefined; that turns a Result problem back into an Option and loses the reason.

Direct answer

A functor is any container-like type (a "context") that has a map operation: map applies an ordinary function to the value or values inside the context and gives back the same kind of context, without changing its shape. Array.prototype.map is the one you already use: it transforms every element and returns an array of the same length and order. The same idea applies to an optional value (apply the function if a value is present, do nothing if it is missing), to a tree (transform every node, keep the branching), and loosely to a Promise. A well-behaved map obeys two laws: identity (map(x => x) returns something equal to the original) and composition (mapping f then g equals mapping the single function g(f(x))). If a map breaks them, refactors that look harmless change behaviour.

Mapping over a context, concretely

An Option is either Some(value) or None ("no value"). mapOption applies f only when there is a value, so callers never write the if (x != null) check.

import assert from "node:assert/strict";

// Option: Some(value) or None, the "context" is "a value might be missing"
const Some = (value) => ({ tag: "some", value });
const None = { tag: "none" };
const mapOption = (o, f) => (o.tag === "some" ? Some(f(o.value)) : None);

const id = (x) => x;
const inc = (x) => x + 1;
const dbl = (x) => x * 2;
const compose = (g, f) => (x) => g(f(x));

console.log(mapOption(Some(20), inc), mapOption(None, inc));

// law check: identity and composition, for a given map over given sample values
function checkLaws(name, map, samples) {
  const results = [];
  for (const s of samples) {
    const idOk = JSON.stringify(map(s, id)) === JSON.stringify(s);
    const compOk =
      JSON.stringify(map(map(s, inc), dbl)) === JSON.stringify(map(s, compose(dbl, inc)));
    results.push(idOk && compOk);
  }
  console.log(name.padEnd(14), results.every(Boolean) ? "laws hold" : "LAW BROKEN", results);
}

const optionSamples = [Some(0), Some(5), None];
checkLaws("mapOption", mapOption, optionSamples);

// Broken: treats a falsy result as "missing". Some(0) -> None even for the identity function.
const mapOptionBad = (o, f) => {
  if (o.tag !== "some") return None;
  const r = f(o.value);
  return r ? Some(r) : None;
};
checkLaws("mapOptionBad", mapOptionBad, optionSamples);

// List: Array.prototype.map is lawful
const listSamples = [[], [3], [1, 2, 3]];
checkLaws("Array map", (xs, f) => xs.map(f), listSamples);

// Broken: a "map" that also reverses. Changes structure, not only the contents.
checkLaws("mapReversing", (xs, f) => xs.map(f).reverse(), listSamples);

// Promise.then: flattens a returned promise, so it is not a strict functor
const p = Promise.resolve(1);
const f = (x) => Promise.resolve(x + 1);        // returns a promise
const g = (v) => (v instanceof Promise ? "got a Promise" : "got a number");
const viaTwoSteps = await p.then(f).then(g);     // g sees a number: then() unwrapped it
const viaOneStep = await p.then((x) => g(f(x))); // g sees the Promise itself
console.log("Promise composition:", viaTwoSteps, "|", viaOneStep);

Output (one run of exactly this file):

{ tag: 'some', value: 21 } { tag: 'none' }
mapOption      laws hold [ true, true, true ]
mapOptionBad   LAW BROKEN [ false, true, true ]
Array map      laws hold [ true, true, true ]
mapReversing   LAW BROKEN [ true, true, false ]
Promise composition: got a number | got a Promise

What it shows:

• mapOption(Some(20), inc) gives Some(21); the None case passes straight through. The function inc never has to know a context exists.
• The checker runs both laws on a list of samples, including the awkward Some(0), and it can fail: mapOptionBad (which treats a falsy result like 0 as "missing") and mapReversing (which reorders the array) are reported as LAW BROKEN.
• Array.prototype.map passes. Its guarantees are exactly "same length, same order, each element transformed independently".

The two laws

1. Identity: mapping the do-nothing function changes nothing. map(s, x => x) equals s.
2. Composition: map(map(s, f), g) equals map(s, x => g(f(x))). Two passes over a structure can be fused into one.

In Haskell, the language where the term comes from, the Functor class (Haskell's name for the idea) documents the same two laws. fmap is Haskell's name for map, id is the do-nothing function, and . composes two functions, so fmap id == id is the identity law and fmap (f . g) == fmap f . fmap g is the composition law (map the composed function in one pass, or map g then map f). The Haskell base documentation adds that the second law follows from the free theorem of the type of fmap (a guarantee that comes from fmap's type alone) together with the first law, so for a correctly typed fmap you only need to check identity. In JavaScript nothing enforces that, so you check both.

For comparison, here is the Haskell version over a small tree, run with runghc (GHC's run-a-script command). Read it line by line: data Tree a = Leaf | Node (Tree a) a (Tree a) declares a tree that is either empty (Leaf) or a Node holding a left subtree, a value and a right subtree; deriving (Show, Eq) asks the compiler to generate printing and == for it; instance Functor Tree where ... is where you supply fmap for Tree, applying f to the value and recursing into both subtrees; (+ 1) is a function that adds 1; Tree Int is a tree of integers.

data Tree a = Leaf | Node (Tree a) a (Tree a) deriving (Show, Eq)

instance Functor Tree where
  fmap _ Leaf = Leaf
  fmap f (Node l x r) = Node (fmap f l) (f x) (fmap f r)

main :: IO ()
main = do
  let t = Node (Node Leaf 1 Leaf) 2 Leaf :: Tree Int
  print (fmap (+ 1) t)
  print (fmap id t == t)
  print (fmap ((* 2) . (+ 1)) t == (fmap (* 2) . fmap (+ 1)) t)

Output:

Node (Node Leaf 2 Leaf) 3 Leaf
True
True

Why breaking the laws hurts real code

• Optimizers and refactors rely on the laws. Fusing xs.map(f).map(g) into xs.map(x => g(f(x))) to save a pass is only safe if composition holds. A map that also reorders, drops, or logs per call makes the fused and unfused versions differ.
• Identity is how you reason about "no-op" transforms. A normalizing step that maps over a Some(0) and turns it into None (the mapOptionBad case) silently deletes valid data, and it only happens for the falsy values you probably did not test.
• Generic code assumes it. A helper that takes any "mappable" thing and relies on map keeping the shape will break the moment one instance reorders or resizes.
• Promises are the cautionary tale. promise.then(f) behaves like map, but then also flattens: if f returns a promise, then waits for it and passes the plain value on. The run's last line shows that p.then(f).then(g) gives g a number, while p.then(x => g(f(x))) gives g the promise itself. So Promise satisfies the composition law only when the mapped functions return non-promise values. The reason is that then does map and flatten in one step.

Senior aside: functors live on type constructors

A functor is defined for a type constructor: a type that still needs another type plugged in before it is complete (Option, Array, Tree), as opposed to a complete type like number. Array on its own is a "type that takes a type" and only Array<number> is a complete type; this is what higher-kinded means (the "kind" of a type says how many types it still needs: a complete type has kind *, and Tree has kind * -> *, read "takes one type, gives a type", so that is the kind of thing Functor accepts). Haskell's instance Functor Tree above names Tree bare, without its parameter.

Pitfalls and when to use the pattern

• map must not do the container's own bookkeeping (count, filter, dedupe). That is a different operation (filter, or flatMap, which applies a function that itself returns a container and flattens the result into one).
• If the callback itself returns a context (Option of Option), you get a nested context. That is the point where you need flatMap, not map. A monad is simply a context type that supports flatMap (called bind or >>= in Haskell) on top of map.

Direct answer

Move toward a functional core, imperative shell: a core of pure functions that take facts (already-loaded data, the current time, an id) and return decisions (a total, a list of messages to send), wrapped in a thin shell that loads the facts, calls the core, and performs the decisions. Do it incrementally and behind a safety net: first pin down what the code does today with characterization tests, then extract one business rule at a time into a pure function, then push the reads to the top of the handler and the writes to the bottom. Stop when the rules are pure and the shell is small; do not add effect frameworks until a concrete pain, such as a decision that depends on the result of an earlier effect, requires them.

Step by step

1. Pin current behaviour (characterization tests). Before changing structure, run the legacy handler with fake collaborators (database, mailer, logger) over a representative set of inputs and record the outputs and the calls made. These are not "correct" tests; they are "same as before" tests. A seam is a place where behaviour can be swapped without editing the code around it; passing the collaborators (database, mailer, logger) in as parameters creates one, and it is a small change that does not alter behaviour. This passing-in is a form of dependency injection: a function receives the things it depends on instead of creating or importing them itself.
2. Find the decisions. Read each handler and mark lines that only compute from data (discount rules, thresholds, which email to send) against lines that talk to the outside (db.*, mailer.*, log, clock, random, environment).
3. Extract the pure core, one rule at a time. Copy a rule into a function whose parameters are everything it reads (customer, order) and whose return value is everything it wants done (Decision). Leave the old code calling it. Ship after each extraction.
4. Return decisions instead of performing them. The core returns notifications: [...] as data; it does not call the mailer. Time and randomness become parameters (now, newId) so the core stays repeatable.
5. Reshape the shell: read everything needed, call the core, write the results. Keep the order of effects the same as in the legacy code until the tests confirm equivalence; changing the order is a separate, deliberate change.
6. Delete the old tangled version once both agree on the recorded cases and in production.

Worked example

type Order = { id: string; customerId: string; subtotalCents: number; coupon?: string };
type Customer = { id: string; tier: "basic" | "gold" };
type Db = { getCustomer(id: string): Customer; saveOrder(o: Order & { totalCents: number }): void };
type Mailer = { send(to: string, body: string): void };

// ---- Step 0: legacy handler, rules tangled with effects
function legacyCheckout(db: Db, mailer: Mailer, log: (m: string) => void, order: Order) {
  log("checkout start " + order.id);
  const c = db.getCustomer(order.customerId);
  let total = order.subtotalCents;
  if (c.tier === "gold") total = Math.round(total * 0.9);
  if (order.coupon === "SAVE5") total = Math.max(0, total - 500);
  db.saveOrder({ ...order, totalCents: total });
  if (total > 10000) mailer.send(c.id, "big order " + order.id);
  log("checkout done " + order.id);
  return total;
}

// ---- Steps 2-4 of the list: pure core. Facts in, decision out. No I/O, no clock, no globals.
type Decision = { totalCents: number; notifications: { to: string; body: string }[] };
function decide(customer: Customer, order: Order): Decision {
  let total = order.subtotalCents;
  if (customer.tier === "gold") total = Math.round(total * 0.9);
  if (order.coupon === "SAVE5") total = Math.max(0, total - 500);
  return {
    totalCents: total,
    notifications: total > 10000 ? [{ to: customer.id, body: "big order " + order.id }] : [],
  };
}

// ---- Step 5 of the list: thin imperative shell. Load, decide, act.
function checkout(db: Db, mailer: Mailer, log: (m: string) => void, order: Order) {
  log("checkout start " + order.id);
  const customer = db.getCustomer(order.customerId);      // read
  const d = decide(customer, order);                     // pure
  db.saveOrder({ ...order, totalCents: d.totalCents });  // write
  d.notifications.forEach((n) => mailer.send(n.to, n.body));
  log("checkout done " + order.id);
  return d.totalCents;
}

// ---- Characterization test: record what the OLD code does, demand the NEW code matches
function fakes() {
  const saved: unknown[] = [], mails: string[] = [], logs: string[] = [];
  const db: Db = {
    getCustomer: (id) => ({ id, tier: id === "g" ? "gold" : "basic" }),
    saveOrder: (o) => void saved.push(o),
  };
  return { db, mailer: { send: (to: string, b: string) => void mails.push(to + ":" + b) }, log: (m: string) => void logs.push(m), saved, mails, logs };
}
const cases: Order[] = [
  { id: "o1", customerId: "b", subtotalCents: 4000 },
  { id: "o2", customerId: "g", subtotalCents: 20000 },
  { id: "o3", customerId: "b", subtotalCents: 12000, coupon: "SAVE5" },
  { id: "o4", customerId: "g", subtotalCents: 300, coupon: "SAVE5" },
];
type Impl = typeof checkout;
function agree(impl: Impl, reference: Impl): boolean {
  return cases.every((o) => {
    const a = fakes(), b = fakes();
    const ta = reference(a.db, a.mailer, a.log, o), tb = impl(b.db, b.mailer, b.log, o);
    return ta === tb && JSON.stringify([a.saved, a.mails, a.logs]) === JSON.stringify([b.saved, b.mails, b.logs]);
  });
}
console.log("refactor matches legacy:", agree(checkout, legacyCheckout));

// The harness can fail: a copy that nudges a gold customer's subtotal by one cent is caught
const broken: Impl = (db, m, l, o) => checkout(db, m, l, { ...o, subtotalCents: o.customerId === "g" ? o.subtotalCents + 1 : o.subtotalCents });
console.log("mutated refactor matches legacy:", agree(broken, legacyCheckout));

// Pure core tested with no fakes at all
const eq = (a: unknown, b: unknown) => JSON.stringify(a) === JSON.stringify(b);
console.log("gold 20000 ->", JSON.stringify(decide({ id: "g", tier: "gold" }, cases[1]!)));
console.log("coupon floors at 0:", decide({ id: "g", tier: "gold" }, cases[3]!).totalCents === 0);
console.log("pure and repeatable:", eq(decide({ id: "b", tier: "basic" }, cases[0]!), decide({ id: "b", tier: "basic" }, cases[0]!)));

Compiled with tsc 5.9.3 --strict --noUncheckedIndexedAccess --target es2022 and run in a node:22 container it prints:

refactor matches legacy: true
mutated refactor matches legacy: false
gold 20000 -> {"totalCents":18000,"notifications":[{"to":"g","body":"big order o2"}]}
coupon floors at 0: true
pure and repeatable: true

How to read the harness: fakes() builds a fresh set of recording stand-ins, whose saveOrder, send and log push into arrays instead of touching anything real. agree runs the reference and the candidate on each of the four cases, each with its own fresh fakes, and compares the returned total plus the three recorded arrays as JSON text; every is true only if all four cases match. Worked through, the four cases give totals 4000 (basic, no discount, no mail), 18000 (gold, 10 percent off 20000, mail sent), 11500 (basic, 12000 minus the 500 coupon, mail sent) and 0 (gold, 300 becomes 270, minus 500 clamped to 0, no mail). The ! in cases[1]! is a non-null assertion: the --noUncheckedIndexedAccess flag makes cases[1] typed as possibly undefined, and the ! tells the compiler the case exists.

What it demonstrates:

• legacyCheckout mixes the discount rules with reads, writes, email and logging. decide is the same rules with every effect removed; checkout is the shell that performs them in the same order (log, read, write, send, log).
• The characterization check agree(checkout, legacyCheckout) compares the returned total, the saved orders, the mails and the log lines for four inputs and reports true. The harness can fail: broken is the refactor with the gold customer's subtotal nudged by one cent, and agree reports false, so a silent change to a rule is caught.
• One limit of this harness: saved, mails and logs are three separate arrays, so it compares the order within each kind of effect but not between kinds. A shell that sent the mail before saving the order would still print true. When the order between effects matters (step 5 above), push every call into one shared calls array instead and compare that.
• The pure core is tested without any fakes: 20,000 cents for a gold customer is 18,000 after the 10 percent discount, which is over 10,000 so one notification is produced; a 300-cent gold order with the 500-cent coupon floors at 0 (270 minus 500, clamped), an edge case the legacy code also had.

How the benefits show up

When to stop short of heavier abstractions

Stop when the shell is a short, linear read, decide, act sequence and the core is covered by fast tests. Reach further only for a specific, observed need:

• A decision that depends on the result of an earlier effect (read, decide, read something else based on that, decide again). Keep the shell as a short loop or sequence of steps that calls the core several times, or have the core return a small data "plan" for the next step. This covers most real cases.
• Many handlers sharing the same effects, and you want to swap implementations (real, in-memory, recording). Pass a small interface of operations (the Db and Mailer types above) as a parameter. That is plain dependency injection and is the practical default.
• Effect-description abstractions (free monads, tagless final) are a heavier step toward the same goal of swapping interpreters. The idea in miniature: instead of calling db.getCustomer(id), the program is built as data that says "get this customer, then continue with this function", and an interpreter walks that data.

type Customer = { id: string; tier: "basic" | "gold" };
type Program<A> =
  | { tag: "done"; value: A }
  | { tag: "getCustomer"; id: string; next: (c: Customer) => Program<A> }
  | { tag: "send"; to: string; body: string; next: () => Program<A> };

const done = <A>(value: A): Program<A> => ({ tag: "done", value });

// A program is data: nothing is read or sent while it is being built.
const program: Program<number> = {
  tag: "getCustomer",
  id: "g",
  next: (c) => {
    const total = c.tier === "gold" ? 18000 : 20000;
    return { tag: "send", to: c.id, body: "big order", next: () => done(total) };
  },
};

// One interpreter: walks the data and records what it was asked to do.
function runForTest(p: Program<number>): { log: string[]; result: number } {
  const log: string[] = [];
  let cur: Program<number> = p;
  while (cur.tag !== "done") {
    if (cur.tag === "getCustomer") {
      log.push("getCustomer " + cur.id);
      cur = cur.next({ id: cur.id, tier: "gold" });
    } else {
      log.push("send " + cur.to + ": " + cur.body);
      cur = cur.next();
    }
  }
  return { log, result: cur.value };
}
console.log(JSON.stringify(runForTest(program)));

Compiled with tsc 5.9.3 --strict --target es2022 and run in a node:22 container it prints:

{"log":["getCustomer g","send g: big order"],"result":18000}

A real free monad adds a generic chaining operation so such programs are written in a normal sequential style instead of nested next functions; the walk-the-data interpreter is the same idea. In tagless final the program is instead written once against an interface and each interpreter implements it. Higher-kinded types are types that take another type constructor as a parameter (abstracting over "some container such as Promise or Array" rather than over Promise<number>); tagless final in its general form relies on them, which is why Haskell and Scala express it naturally and TypeScript does not. The trade-offs: interpreters are easy to write and many can coexist (a real one, a test one, a logging one), but building and walking program data adds overhead and indirection; type safety is strong in languages with higher-kinded types such as Haskell and Scala, but TypeScript has no direct way to write that abstraction, which weakens the types and makes the code harder to read; testability is excellent because the test interpreter is a few lines, though a plain fake Db gives most of the same value with far less machinery. For a typical TypeScript service, interface parameters are the proportionate choice.

Pitfalls

• Refactoring without characterization tests: a behaviour change hides inside a "cleanup".
• "Pure" functions that still read Date.now(), process.env or a module-level cache.
• Pulling all effects to the top when one read depends on another's result, creating a giant eager load; load lazily in the shell instead.
• Rewriting everything at once. Extract the highest-churn or highest-risk rule first and keep each step shippable.
• Changing effect order or transaction boundaries (save then mail) while moving code; keep the behaviour first, then improve it separately.

Direct answer

Build the pipeline as a chain of generators (functions that produce values one at a time, on demand, instead of building a whole list). Each stage pulls one item from the stage before it, does its work, and yields one item to the stage after it. Nothing asks for "all the records", so memory stays small and bounded at any moment (a chunk of the file, the lines readline has read ahead, the record being processed and the output buffer), whatever the file size. The final stage is a writable stream (an output sink with a bounded buffer); Node's stream.pipeline connects the stages and pauses the reader whenever the sink is full (back-pressure: a slow consumer slows the producer instead of letting data pile up between them).

Design: read, parse, filter, serialise, write

Four files make up the demo, all in one directory: three scripts and a shell runner. make-data.mjs creates the input:

// 1,000,000 lines "id,name,amount"; every 10th line is invalid (amount is not a number)
import { createWriteStream } from "node:fs";
const out = createWriteStream("records.csv");
for (let i = 1; i <= 1_000_000; i++) {
  out.write(`${i},user-${i},${i % 10 === 0 ? "n/a" : (i % 977) + ".25"}\n`);
}
out.end();

stream.mjs is the streaming pipeline. Two Node pieces are used: readline (a built-in module that reads a stream one line at a time) and yield* (inside a generator, hand over each value of another iterable one by one). An async function* is an async generator: like a generator, but it may await and is consumed with for await ... of, which waits for each item in turn.

import { createReadStream, createWriteStream } from "node:fs";
import { createInterface } from "node:readline";
import { pipeline } from "node:stream/promises";

// parse: one line -> record, or null when invalid
const parseLine = (line) => {
  const [id, name, amount] = line.split(",");
  const n = Number(amount);
  return Number.isFinite(n) && id && name ? { id: Number(id), name, amount: n } : null;
};

// each stage is a generator: it pulls one item, yields one item, and keeps no list of past items
async function* lines(path) {
  yield* createInterface({ input: createReadStream(path), crlfDelay: Infinity });
}
async function* parsed(source) {
  for await (const line of source) {
    const rec = parseLine(line);
    if (rec) yield rec; // invalid lines are dropped here
  }
}
async function* toJsonLines(source) {
  for await (const rec of source) yield JSON.stringify(rec) + "\n";
}

let peakRss = 0;
const timer = setInterval(() => (peakRss = Math.max(peakRss, process.memoryUsage().rss)), 20);

await pipeline(
  async function* () { yield* toJsonLines(parsed(lines("records.csv"))); },
  createWriteStream("out.jsonl")
);
clearInterval(timer);
console.log("streaming: done, peak RSS under 150 MB:", peakRss < 150 * 1024 * 1024);

The read-everything-first version of the same job, for comparison, is readall.mjs:

import { readFileSync, writeFileSync } from "node:fs";
const parseLine = (line) => {
  const [id, name, amount] = line.split(",");
  const n = Number(amount);
  return Number.isFinite(n) && id && name ? { id: Number(id), name, amount: n } : null;
};
const records = readFileSync("records.csv", "utf8").split("\n").map(parseLine).filter(Boolean);
writeFileSync("out-all.json", JSON.stringify(records));
console.log("read-all: done,", records.length, "records");

run.sh generates a 1,000,000-line input, then runs both jobs with the JavaScript heap (the memory the engine manages for objects and strings) capped at 32 MB (--max-old-space-size=32). The pipeline's check uses RSS (resident set size: all physical memory the process holds, which also counts the Node runtime, compiled code and stream buffers that live outside the capped heap). That is why RSS is compared with its own generous 150 MB ceiling rather than with 32 MB: the cap limits the heap only, and the ceiling confirms the process as a whole stays small. The 25 MB test file stands in for a multi-gigabyte one: what matters is the ratio of input size to memory available, and a 32 MB cap against a 25 MB file reproduces the same failure at a size that runs in seconds.

run.sh:

node make-data.mjs
sleep 1
echo "input bytes: $(stat -c %s records.csv)"
node --max-old-space-size=32 stream.mjs; echo "streaming exit code: $?"
echo "output lines: $(wc -l < out.jsonl)"
node --max-old-space-size=32 readall.mjs > readall.log 2>&1; echo "read-all exit code: $?"
grep -m1 -o "heap out of memory" readall.log

Run in a node:22 container (Linux) with sh run.sh, it prints:

input bytes: 25376416
streaming: done, peak RSS under 150 MB: true
streaming exit code: 0
output lines: 900000
read-all exit code: 134
heap out of memory

The streaming job writes all 900,000 valid records (every tenth line has n/a as its amount and is dropped) under the same cap that makes the read-all job abort with heap out of memory (exit code 134 is 128 plus signal 6, SIGABRT, the signal a process sends itself when it aborts). The RSS line is informational (it prints true or false and does not set the exit code); the evidence is the exit codes. A variant of stream.mjs whose toJsonLines first pushes every record into an array aborts the same way, with exit code 134 and heap out of memory, under the same cap.

Why it avoids the blowup

readFileSync(...).split("\n").map(...) holds the file as one string, then as an array of line strings, then as an array of objects, all alive at once, so memory grows linearly with the input and the process dies when it passes the limit. The generator chain is pull-based: for await asks the previous stage for the next item only when this stage is ready, so records flow through the stages one at a time and become garbage once written. One stage reads ahead: the readline async iterator queues lines and pauses its input when the queue reaches 1,024 lines (highWaterMark: 1024 in the Node 22 source). That is a fixed bound, so memory stays constant, but it is a queue of lines, not literally one line. The stream's buffer is bounded by its highWaterMark; Node's documentation describes that as a threshold at which the stream stops asking for more data, not a hard memory limit.

What breaks the property

1. Collecting anywhere. Array.from(stage), spreading, toArray, or await Promise.all(...) over the stream materialises everything (builds the full result in memory at once). So do operations that need all the data: global sort, group-by, and de-duplication with a Set (it grows with the number of distinct keys). Fix by an external sort (sort pieces small enough to fit in memory, write each sorted piece to disk, then merge the pieces by reading them in parallel), or by moving that step to a database.
2. Ignoring back-pressure. If you write with a manual loop and never wait when write() returns false, Node's documentation says it buffers every chunk until memory runs out and then aborts. pipeline (or await once(stream, "drain") in a loop, where drain is the event the stream emits when its buffer has emptied and it is safe to write again) is what keeps the producer paced.
3. Unbounded concurrency. Starting one async task per line (lines.map(async ...) collected in an array) holds every in-flight task in memory. Cap the concurrency (a small worker pool or a batch of N) if a stage does I/O.
4. One enormous record. The chain is constant-memory per record; a single multi-hundred-megabyte line is buffered whole by the line reader.
5. Keeping evidence. Pushing every rejected line into an error array re-creates the problem. Count the invalid lines and keep only a bounded sample.

Failure behaviour

pipeline destroys every stream in the chain when any stage throws and rejects its promise, so a bad file does not leave handles open. A malformed line is handled in parseLine (returns null, the stage drops it), so one bad record does not stop the job. The output is only complete if the promise resolves, so write to a temporary file and rename it on success if downstream readers must never see a partial file.