Transformers and Attention Questions

The transformer architecture behind modern language models. Covers queries, keys and values, scaled dot-product and multi-head attention (including why scores are scaled and worked numeric examples), masking, positional encoding (sinusoidal, learned, relative with bucketed biases, rotary, and 2D schemes for image patches), encoder-only, decoder-only and encoder-decoder variants, residual connections, layer normalization placement and the feed-forward block, parameter, memory and compute accounting, why transformers scale better than recurrent models, efficient and long-context attention variants (sparse, local window with global tokens, hashing-based, linear, low-rank, FlashAttention, multi-query), and what attention heads learn and what raw attention weights do and do not explain. Includes implementing attention and its components from scratch. Distributed training, serving and quantization, fine-tuning, pretraining objectives, and general optimizer practice are covered elsewhere.