Conference Presentation, Lecture, Interview
Positional Encodings and Group Theory | 3Blue1Brown and Alok Puranik
- Anticipates that all positional encoding variants stem from group theory concepts and conjectures the entire universe of methods is encompassed by algebraic representations.
- Plans to derive the minimal assumptions required to generate observed algebraic structures, assuming real-world time series are modeled as continuous for physical phenomena.
- Intends to modify queries and keys via time-dependent linear maps rather than direct dot product adjustments, relying on translation variance constraints motivated by Noether's theorem.
- Expects that imposing an identity transition matrix at zero time difference and requiring argument addition to correspond to matrix multiplication will yield an exponential solution derived from a differential equation near zero.
- Predicts that diagonalizing the generator matrix decomposes the system into independent subspaces characterized by real-valued eigenvalues (growth, decay, or static) or complex eigenvalues (rotation).
- Notes that Rotary Positional Encodings (RoPE) specifically utilize pure imaginary eigenvalues to achieve rotation without decay or blow-up.
- Plans to explore non-diagonalizable Jordan blocks to investigate polynomial time behaviors, expecting specific structures to recover Alibi's linear penalty terms while viewing higher-order polynomial terms as unlikely to be practically useful.
- Aims to publish internal and potentially public blog posts concluding that the space of positional encodings is sufficiently explored under the stated mathematical constraints.