Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent

Training loss and accuracy are the standard signals used to monitor generalization during deep neural network training. Two well-documented phenomena complicate this picture: in grokking, train loss falls rapidly while test performance improves abruptly only after a long delay; in epoch-wise double ...