Can large language models figure out the real world?

A flicker of insight about our solar system's orbital mechanics came from Johannes Kepler in the 17th century, but it took Newton's later gravitational laws to transform these predictions into deeper understanding. This historical scientific leap now offers a revealing parallel to today's artificial intelligence systems.

"Humans all the time have been able to make this transition from good predictions to world models," explains Harvard postdoc Keyon Vafa, lead author of a new MIT and Harvard study examining whether AI has developed similar capabilities.

Their research suggests the answer is largely no. While modern AI excels at prediction tasks, it struggles to develop the deeper understanding that would allow it to transfer knowledge between domains.

The team developed a measurement called "inductive bias" to quantify how well AI systems approximate real-world conditions. Testing increasingly complex scenarios, they found AI performed well only in the simplest cases.

As MIT's Sendhil Mullainathan notes, "We know how to test whether an algorithm predicts well. But what we need is a way to test for whether it has understood well."

This work offers a benchmark for evaluating AI's true comprehension as researchers increasingly deploy these systems for scientific discovery in fields from chemistry to protein folding.