Alok Puranik

Positional Encodings and Group Theory | 3Blue1Brown and Alok Puranik

3Blue1Brown, Alok Puranik, Grant Sanderson

This theoretical framework establishes that valid positional encodings in attention mechanisms are mathematically defined by a minimal set of linearity and translation variance assumptions, resulting in a general form where the transformation matrix is the exponential of a constant matrix. By analyzing the eigenvalues of this matrix, the work classifies existing methods into distinct dynamic behaviors, explicitly identifying Rotary Positional Embeddings (RoPE) as a pure rotation case while recovering ALiBi as a non-diagonalizable construction yielding linear dependence. The analysis concludes that the space of stable, useful encodings is effectively exhausted by combinations of exponential decay, pure rotation, and damped rotation, suggesting that future novel approaches would likely reside in unstable or higher-order polynomial regimes.