Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias

The successful training of neural networks hinges on the use of first order optimization methods, yet the theoretical characterization of these methods remains incomplete. This is especially true in settings with mild overparameterization. In this work, we study the gradient flow dynamics of two-lay...