Training AI on small datasets forces "grokking", where overtraining drives deep generalization beyond memorization. Current labs do the opposite, potentially explaining why LLMs make errors no comparably intelligent human would.

Get hand-picked daily summaries of the best, most informative AI articles from around the web.

Blogger Gwern argues that AI labs should train massive models on small datasets, forcing a process called "grokking," where prolonged overtraining pushes models beyond memorization into deep generalization. Current frontier labs do the opposite, which may explain why LLMs make errors no comparably intelligent human would make.

Overtraining as the path to human-like AI

Get the next batch of curated summaries in your inbox.