Grokking of Transformer on Modular Addition
Background Grokking is a phenomenon where a model quickly achieves near-perfect training accuracy (memorization), while validation accuracy remains near chance for a long time, and then later transitions sharply to strong generalization after extende...
Feb 8, 20263 min read8