To detect trailing spaces/tabs recursively, read files in text mode with explicit encoding fallback, process line-by-line to handle very large files, and skip likely-binary files by sampling bytes. Below is a safe, practical implementation and notes on binary handling, large files, encodings, and CI/pre-commit integration.
python
import os
from pathlib import Path
from typing import List, Tuple
def is_likely_binary(path: Path, blocksize: int = 1024) -> bool:
# Sample first block for NUL bytes or high non-text byte ratio
try:
with path.open('rb') as f:
chunk = f.read(blocksize)
except Exception:
return True
if b'\x00' in chunk:
return True
# heuristic: many bytes outside printable ASCII/UTF-8
nontext = sum(1 for b in chunk if b < 9 or (13 < b < 32) or b > 126)
return (len(chunk) > 0) and (nontext / len(chunk) > 0.3)
def detect_trailing_whitespace(path: str) -> List[Tuple[str, int, str]]:
"""
Returns list of tuples: (file_path, line_number, offending_chars)
Recurses directories. Streams files line-by-line to keep memory low.
"""
results = []
p = Path(path)
targets = [p] if p.is_file() else list(p.rglob('*'))
for f in targets:
if not f.is_file():
continue
if is_likely_binary(f):
continue
# Try utf-8, fallback to latin-1
for enc in ('utf-8', 'utf-8-sig', 'latin-1'):
try:
with f.open('r', encoding=enc, errors='strict') as fh:
for i, raw in enumerate(fh, start=1):
# preserve only trailing whitespace characters before newline
# rstrip('\r\n') to ignore line endings, then check trailing space/tab
line = raw.rstrip('\r\n')
if line.endswith(' ') or line.endswith('\t'):
# capture type of whitespace found
tail = []
j = len(line) - 1
while j >= 0 and line[j] in (' ', '\t'):
tail.append(line[j])
j -= 1
results.append((str(f), i, ''.join(reversed(tail))))
break
except UnicodeDecodeError:
continue
except Exception:
# ignore unreadable files (permissions, etc.)
break
return results
Key points:
- Streams files line-by-line: O(n) time, O(1) memory per file; suitable for very large files.
- Binary detection: samples bytes and skips files with NULs or high non-text ratio to avoid false positives and decoding errors.
- Encoding handling: tries utf-8/utf-8-sig then latin-1; for strict environments you may fail on unknown encodings or add chardet/universal detector.
- Edge cases: files with mixed encodings, CR-only line endings, and generated minified files; adjust heuristics as needed.
Integration:
- Pre-commit: add a hook using pre-commit framework that runs this script and exits non-zero if results exist; optionally auto-fix by trimming trailing whitespace.
- CI: run as part of lint stage; publish failures with file/line details so developers can fix in PR.
Alternative: use existing linters (flake8, editorconfig-checker) for broader coverage, but this small script provides a customizable, fast check.
