Variants of LoRA

Low-rank adaptation (LoRA) techniques allow for efficient training of specialized large language models on custom data. Several LoRA variants exist, each offering unique benefits. QLoRA uses 4-bit quantization to reduce memory usage during finetuning. LongLoRA adapts models to longer context lengths using sparse attention and LoRA. S-LoRA enables serving multiple LoRA modules on a single GPU by storing modules in main memory and using unified paging. These techniques aim to make LLM customization more accessible and resource-efficient for various applications.