Applied Analysis Seminar Gradient flows on probability measures and their inertial variants: from Wasserstein geometry to Transformers

Gradient flows on spaces of probability measures (“density manifolds”) provide a geometric framework for deriving and analyzing nonlinear evolution equations from an energy functional and a metric structure. I will begin by explaining the Wasserstein gradient flow, its kernelized variants, and their particle approximations. I will then detail how one can pass from first-order dissipative dynamics to inertial, damped Hamiltonian dynamics on density manifolds, where particles carry both positions and velocities.

This geometric viewpoint has recently appeared in the analysis of Transformer architectures: attention layers can be interpreted as interacting particle systems, with mean-field limits given by gradient flows of interaction energies on spaces of probability measures. In our “SympFormer” work, joint with Wuchen Li and Gabriele Steidl, we use this interpretation to construct accelerated attention blocks from inertial dynamics on density spaces. The resulting architecture can be viewed as a structure-preserving particle discretization of an accelerated probability-measure flow.

The talk is intended to be accessible without prior knowledge of Transformers. The main theme is that ideas from gradient flows, Hamiltonian mechanics, and infinite-dimensional geometry can be used not only to analyze PDEs, but also to design neural network architectures.