Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW

A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We fo...