This Puzzle Shows Just How Far LLMs Have Progressed in a Little Over a Year

A geometric puzzle that took GPT-4o over two hours and 40 prompts to solve in June 2024 was recently solved by Claude Sonnet 4.5 in five seconds on the first try. The challenge involved writing Python code to find and plot all possible squares on a cross-shaped grid of dots, with the correct answer being 21 total squares. While Sonnet 4.5 initially gave an incorrect answer when asked directly about the number of squares, it immediately produced perfect Python code when given the same coding prompt that had stumped its predecessor for hours.