SemHash – Fast Semantic Text Deduplication for Cleaner Datasets

SemHash is a tool for deduplicating datasets using semantic similarity, combining fast embedding generation and efficient similarity search. It offers single and multi-dataset deduplication, supports complex datasets like QA, and provides functions to inspect results. SemHash can process millions of records across various dataset types. Benchmarks demonstrate its effectiveness in removing duplicates, revealing significant train/test overlap in some cases. The tool is designed to be fast, scalable, flexible, and lightweight for easy data cleaning.