Flow-based Spectral Kernel Learning for Nonstationary Attention

What should count as similar?

Attention combines information from different tokens. A kernel perspective turns this into a question about similarity: how should two inputs relate to each other? Stationary kernels depend only on the difference between inputs. Nonstationary kernels can also respond to each input individually, allowing a richer structure of relationships.

Learn the spectrum, shape the kernel.

FSKA learns a joint spectral density over frequency pairs using a normalizing flow. Samples from that density define spectral features, providing a route to nonstationary kernelized attention. The architecture removes learned query and key projections and uses fixed orthogonal value projections. Each head learns its own kernel.

Aggregation without every pair.

The implementation applies ELU plus one to produce positive features. Feature–value products and feature sums can be accumulated without materializing every pairwise token interaction. With fixed feature and value dimensions, aggregation scales linearly with sequence length.

What the experiments show.

Classification experiments, ablations and scalability profiling report competitive or improved performance against the tested attention mechanisms. The paper reports approximately 40–47% fewer projection-related trainable parameters than most evaluated baselines. These are projection parameters, rather than every parameter in the model. The results do not guarantee quality on untested tasks, and linear aggregation does not make all other components cost-free.

My part in the collaboration.

I am a coauthor, fourth in the published author order. I also present the work in the explanation below and maintain its public project website. Specific scientific contributions are not individually itemized here.