Is GitHub Actions suitable for running benchmarks?
SMRTR summary
GitHub Actions can effectively run relative benchmarks by comparing two commits in the same job, canceling out hardware variability. Testing identical commits over 16 days showed this method reliably detects performance regressions of 50% or more with only 3.7% false positives. Optimizations like using mamba instead of conda and implementing ccache reduced benchmark time from nearly 2 hours to 1 hour 20 minutes, while on-demand triggering via PR labels prevents resource waste.
SMRTR provides this summary for quick context. The original article belongs to Lobsters.
Read the original article