18 days ago
BenchmarkTools makes performance tracking of Julia code easy by supplying a framework for writing and running groups of benchmarks as well as comparing benchmark results.
This package is used to write and run the benchmarks found in BaseBenchmarks.jl.
The CI infrastructure for automated performance testing of the Julia language is not in this package, but can be found in Nanosoldier.jl.
To install BenchmarkTools, you can run the following:
If you're just getting started, check out the manual for a thorough explanation of BenchmarkTools.
If you want to explore the BenchmarkTools API, see the reference document.
If you want a short example of a toy benchmark suite, see the sample file in this repo (benchmark/benchmarks.jl).
If you want an extensive example of a benchmark suite being used in the real world, you can look at the source code of BaseBenchmarks.jl.
If you're benchmarking on Linux, I wrote up a series of tips and tricks to help eliminate noise during performance tests.
Our story begins with two packages, "Benchmarks" and "BenchmarkTrackers". The Benchmarks package implemented an execution strategy for collecting and summarizing individual benchmark results, while BenchmarkTrackers implemented a framework for organizing, running, and determining regressions of groups of benchmarks. Under the hood, BenchmarkTrackers relied on Benchmarks for actual benchmark execution.
For a while, the Benchmarks + BenchmarkTrackers system was used for automated performance testing of Julia's Base library. It soon became apparent that the system suffered from a variety of issues:
The BenchmarkTools package is a response to these issues, designed by examining user reports and the benchmark data generated by the old system. BenchmarkTools offers the following solutions to the corresponding issues above:
This package was authored primarily by Jarrett Revels (@jrevels). Additionally, I'd like to thank the following people: