Optimise LLM usage costs with Semantic Cache

A developer faced massive LLM API costs after building a chatbot that processed 200,000 daily inquiries with ~2000 input tokens each, prompting the creation of a semantic cache solution. Unlike traditional caches that require exact key matches, semantic cache uses vector embeddings to find similar questions and return previously generated answers, bypassing expensive LLM calls. The system achieved optimal results with a 95% similarity threshold, balancing accuracy and cache hit rates while avoiding costly processes for repetitive queries.