Implementing DeepSeek R1's GRPO algorithm from scratch

GRPO:Zero is a project implementing Group Relative Policy Optimization (GRPO) for training large language models with minimal dependencies. It uses a single A40 GPU and includes improvements like token-level policy gradient loss and overlong episode filtering. The project trains Qwen2.5 models on the CountDown task, where the model generates mathematical expressions to reach a target number. Rewards are given for correct formatting and accurate answers. The implementation builds on work from DeepSeekMath, DAPO, TinyZero, and nano-aha-moment.