I Vibe Coded a Multiplayer ASL Game using MQTT!

A university student has pushed the boundaries of machine learning by creating a desktop game that recognizes American Sign Language alphabet gestures in real-time, complete with multiplayer functionality powered by MQTT messaging typically reserved for Internet of Things devices.

The ambitious project combines p5.js and ml5.js libraries to perform pose estimation on hand movements, challenging players to spell words using ASL gestures as quickly as possible. Players can compete solo or join multiplayer rooms through shared codes, with an Arduino display showing real-time statistics like letters per second.

The developer faced significant hurdles, particularly in training the machine learning model. After discovering that standard image classification captured too much background noise, they pivoted to hand pose detection and built a custom model from scratch.

A peculiar training glitch emerged where the system would forget the final two letters, X and Y, requiring the collection of double and triple the usual data points respectively to compensate. The project demonstrates how traditional IoT protocols like MQTT can be creatively repurposed for gaming applications, stretching beyond their typical use in constrained devices like Arduino boards.