What are popular AI coding benchmarks actually measuring?

Popular AI coding benchmarks measure far narrower skills than their names suggest, focusing on well-defined problems rather than the messy reality of software development. While Claude scores 80% on SWE-bench, this doesn't translate to solving 80% of real coding tasks because benchmarks test surgical code edits (averaging 11 lines), small programming exercises, and competitive programming problems that pass unit tests rather than measuring code quality, maintainability, or the complex problem-solving that actual software engineering requires.