6 Things I Learned Building LLMs From Scratch That No Tutorial Teaches You

A developer implemented GPT-2 from scratch using PyTorch and discovered six critical architectural insights that tutorials overlook. Key findings include using RsLoRA instead of standard LoRA to prevent weight updates from shrinking as rank increases, implementing RoPE positional embeddings that don't alter token information unlike traditional methods, and understanding that weight tying between embedding layers saves significant parameters in small models but becomes negligible in billion-parameter systems. The research revealed that KV-Cache dramatically speeds inference but creates memory bottlenecks, and LayerNorm layers are deliberately skipped during quantization because the minimal memory savings aren't worth the quality degradation.