Extending Raschka's GPT-2: an MoE trained from scratch on an RTX 3090

A developer extended the GPT-2 code from Sebastian Raschka's book "Build a Large Language Model from Scratch" by adding Mixture-of-Experts (MoE) support, then trained a 446-million-parameter model on a single RTX 3090 over eight days. MoE models replace each feed-forward network in a transformer block with multiple expert networks, activating only a subset per token, giving large-model quality at smaller-model inference speeds. The resulting model outperformed all the developer's previous models on test loss and instruction-following evaluations.