LLMs don’t do formal reasoning - and that is a HUGE problem

Recent Apple research challenges the notion that large language models (LLMs) can perform formal reasoning, finding no evidence of such ability. The study suggests LLMs rely on sophisticated pattern matching, with performance dropping by about 10% when names in problems are changed. The researchers developed GSM-NoOp, a task exposing LLM reasoning flaws when faced with distracting information. The study also shows LLM performance declining as problems become more complex, such as with larger multiplication problems or chess rules.