Interview
Chip design from the bottom up – Reiner Pope
- Fundamental Chip Primitives:
- Logic gates (AND, OR, NOT) are the smallest units, connected by physical metal traces.
- The core mathematical operation for AI chips is the Multiply-Accumulate (MAC) unit.
- AI chips often use 4-bit multiplication inputs with 8-bit accumulation outputs to handle error accumulation while maintaining efficiency.
- Matrix multiplication consists of nested loops where a MAC operation occurs at every single step ($output_{ik} += input_{ij} \times input_{jk}$).
- Circuit Implementation of MAC:
- Multiplication is generated via $P \times Q$ AND gates to produce partial products.
- Summation is performed using "full adders" (3-to-2 compressors), which take three single-bit inputs and produce a two-bit output (sum and carry).
- A systolic multiplier using full adders requires exactly $P \times Q$ full adders to reduce $P \times Q$ inputs plus $P+Q$ accumulator bits down to $P+Q$ output bits.
- The hardware cost of multiplication scales quadratically with bit width, making low-precision arithmetic (e.g., FP4 vs. FP8) significantly more area-efficient than linear scaling would suggest.
- Data Movement vs. Compute Trade-offs:
- In traditional CPU/CUDA core architectures, data movement between register files and ALUs (via Mux circuits) consumes approximately 75-80% of the circuit area, leaving only a small fraction for actual logic.
- A Multi-Input Mux ($N$ inputs, $P$ bits) requires $N \times P$ AND gates and $(N-1) \times P$ OR gates solely for selection logic.
- This communication overhead motivated the shift from scalar cores to Tensor Cores (systolic arrays).
- Systolic Arrays and Optimization:
- Systolic arrays map the matrix multiply loop directly to hardware, storing weight matrices locally within the array rather than fetching them from a register file every cycle.
- This architecture reduces input/output communication bandwidth from quadratic ($X \times Y$) to linear ($X$) relative to the matrix dimensions.
- Weights are trickle-fed into the array over time (daisy-chaining) to minimize the bandwidth required to cross the boundary from external memory.
- The trade-off involves sizing the systolic array against the register file; larger arrays amortize the fixed overhead of data movement but reduce flexibility for irregular workloads.
- Clock Cycles and Pipeline Registers:
- Chips synchronize via a global clock signal, forcing all logic to settle within a fixed nanosecond interval.
- Clock frequency is constrained by the "critical path" (the longest delay through logic clouds).
- Designers insert pipeline registers to split long logic chains, increasing clock speed at the cost of increased area and reduced throughput due to lower parallelism.
- Feedback loops in logic (e.g., running sums) complicate pipelining, as inserting registers mid-loop can alter the mathematical result.
- FPGA vs. ASIC Architecture:
- FPGAs offer deterministic latency crucial for high-frequency trading but are ~10x more expensive and less energy-efficient than ASICs due to configuration overhead.
- FPGAs use Look-Up Tables (LUTs) which are essentially large Muxes (e.g., 16:1) that can be programmed to emulate any logic gate.
- This programmability requires significant overhead (e.g., 32 gates in a LUT to emulate a 4-input AND gate) compared to the direct hard-wired logic of an ASIC.
- CPU Cache vs. TPU Scratchpad:
- Standard CPUs rely on hardware-managed caches, introducing non-deterministic latency dependent on cache hits/misses.
- TPUs and AI accelerators use "scratchpads," where software explicitly controls data movement between on-chip SRAM and off-chip memory, ensuring deterministic timing.
- GPU vs. TPU Topology:
- GPUs utilize a fine-grained architecture with thousands of identical Streaming Multiprocessors (SMs), offering high flexibility and local data reuse within an SM.
- TPUs use coarse-grained units with large systolic arrays and a central vector unit, optimizing for massive matrix operations but limiting data movement bandwidth between the vector unit and matrix units.
- Data movement across the GPU is more flexible (multiple paths) compared to the bottlenecked path in TPU vector-to-matrix transfers.
- Forward-Looking Statements & Strategic Trends:
- NVIDIA's historical 2x FP4-to-FP8 performance ratio was actually conservative; due to quadratic scaling, the ratio should approach 4x, with newer chips (B300+) acknowledging FP4 is ~3x faster than FP8.
- Maddox is developing a "splittable systolic array," a hybrid architecture capable of acting as both a large array for throughput and smaller units for flexibility.
- There is a recognized trend toward maximizing compute-to-memory-bandwidth ratios across the stack, from precision selection (FP4) to architectural design (systolic arrays).
- Energy Consumption Dynamics:
- Chip energy is primarily consumed by dynamic switching (charging/discharging capacitors), not leakage.
- Slowing clock speeds reduces transitions and energy linearly but does not improve energy efficiency per operation significantly, as the circuit remains idle.
- The brain's energy efficiency stems from massive parallelism and low clock speeds combined with sparse connectivity, whereas silicon efficiency is driven by high parallelism and optimized data movement.