Tracking Down Mysterious ML Training Stalls

Pinterest's engineers discovered their PyTorch upgrade caused a mysterious 50% training slowdown instead of expected improvements. Through systematic debugging, they identified two culprits: a PyTorch dispatch mode that interfered with torch.compile optimization, and Ray's monitoring process that periodically collected detailed memory statistics, causing brief but frequent stalls. After disabling the problematic monitoring and fixing the compilation issue, they achieved a 20% speedup over the original version.