New paper pushes back on Apple’s LLM ‘reasoning collapse’ study

Apple's research on large reasoning models' limitations has been challenged by Alex Lawsen of Open Philanthropy. He argues that experimental design flaws, not actual reasoning limitations, caused the reported failures. Lawsen cites issues with token budgets, impossible puzzles, and evaluation methods. He shows that models like Claude and Gemini can solve complex problems when asked to generate algorithms instead of exhaustive move lists. This debate underscores the importance of well-designed evaluations in assessing AI reasoning capabilities.