768GB Intel Optane DIMMs to run 1T-parameter LLM with single GPU at 4tps
SMRTR summary
A Reddit user built a workstation using discontinued Intel Optane memory modules to run a 1-trillion-parameter AI model locally at 4 tokens per second — a remarkable feat on a budget. Using 768GB of cheap second-hand Optane DIMMs paired with an RTX 3060 GPU and llama.cpp software, the system punches well above its price class.
SMRTR provides this summary for quick context. The original article belongs to Hacker News.
Read the original article