AI turns smartwatch speaker and mic into 3D hand tracking system using sonar tech

Your smartwatch is about to become a hand-tracking device without any upgrades. Researchers from Cornell University and KAIST have developed WatchHand, a system that transforms ordinary smartwatches into real-time finger and wrist trackers using nothing more than artificial intelligence and the device's existing speaker and microphone.

The technology works like digital echolocation. The smartwatch emits inaudible sound waves that bounce off your hand and return to the microphone, creating an acoustic fingerprint that machine learning algorithms decode into precise 3D hand movements.

"In the future, with this kind of hand-tracking technology, we might be able to track our typing with just our smartwatch," said Chi-Jung Lee, a doctoral student at Cornell and co-lead author of the study. "Our hands can act as an input device with computers."

Unlike bulky camera-based systems, WatchHand requires zero additional hardware and processes everything locally on the device. The researchers tested it on 40 participants across multiple smartwatch models and found it could reliably track gestures even in noisy environments.

The implications stretch from gesture-controlled computers to assistive technologies for users with limited mobility. "With just a software update, we can potentially unlock entirely new capabilities on millions of existing devices," said Cornell professor Cheng Zhang.