How LLMs See Images, Audio, and More

Modern LLMs process multiple data types by converting everything into tokens, with each media requiring distinct tokenization strategies. Text becomes word pieces, images transform into patch embeddings or quantized visual patterns, while audio can be encoded as neural codec tokens or converted to text through speech recognition. These tokenization approaches involve critical tradeoffs between quality, efficiency, and semantic understanding, directly impacting what AI systems can comprehend and generate.