AudioX: Diffusion Transformer for Anything-to-Audio Generation

AudioX is a new AI model for generating both audio and music from various inputs like text, video, and images. It uses a unified architecture and masked training strategy to create high-quality audio across different tasks. The researchers created two large datasets to train AudioX: vggsound-caps with 190,000 audio captions and V2M-caps with 6 million music captions. In tests, AudioX matched or beat specialized models while offering more flexibility in handling different input types and generation tasks.