Why do OpenAI's GPT-2 weights beat mine?
SMRTR summary
A developer training a GPT-2-style language model from scratch noticed OpenAI's original GPT-2 small weights consistently outperformed their own models on instruction-following tasks, even when their models achieved better technical loss scores. The gap likely stems from data quality and "overtraining" — OpenAI trained on far more tokens than the standard Chinchilla-optimal formula, which may produce better-positioned model weights for fine-tuning.
SMRTR provides this summary for quick context. The original article belongs to Giles Thomas Blog.
Read the original article