I Built an Open-Source Tool to Attack-Test LLMs. Here's What Breaks

A security researcher created Augustus, an open-source vulnerability scanner that tests AI language models with over 210 adversarial attacks across 47 categories including jailbreaks, encoding bypasses, and data extraction attempts. The tool reveals critical security gaps in production AI systems, with studies showing 86% of LLM applications vulnerable to attacks and techniques achieving up to 98% bypass rates against major models like GPT-4o.