OmniParser V2: Turning Any LLM into a Computer Use Agent

OmniParser V2, an improved tool for GUI automation, enhances the ability of large language models to interact with user interfaces. It accurately identifies small interactive elements and understands screenshot semantics, achieving a 39.6% accuracy on the ScreenSpot Pro benchmark - a significant improvement over GPT-4o's original 0.8% score. The new version reduces latency by 60% compared to its predecessor. OmniTool, a related development, allows for faster experimentation with different agent settings using various state-of-the-art language models.