SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 4090 GPU with 3x Speedup

SVDQuant, a new post-training quantization method, enables 4-bit compression of large AI image generation models like FLUX.1 and PixArt-∑. It reduces memory usage by 3.6x and speeds up processing by 8.7x on consumer GPUs compared to 16-bit models. SVDQuant preserves image quality better than existing methods at low precision by using a low-rank branch for outlier values. Combined with the Nunchaku inference engine, it allows billion-parameter diffusion models to run efficiently on laptops, potentially increasing accessibility of advanced AI image generation.