Reinforcement Learning for LLMs

Large language models after pre-training can generate fluent text but may produce responses that are confidently wrong or unhelpful, prompting the use of reinforcement learning to optimize for overall response quality rather than just token likelihood. The core challenge involves credit assignment—determining which specific tokens in a lengthy response contributed to the final reward score. This guide explains how algorithms like PPO solve this through critic networks that estimate value functions and compute per-token advantages, while newer approaches like GRPO achieve similar results by using group statistics as baselines instead of maintaining separate critic models.