SciMLBenchmarks.jl: Benchmarks for Scientific Machine Learning (SciML) and Differential Equation Solver Software
SciMLBenchmarks.jl holds webpages, pdfs, and notebooks showing the benchmarks
for the SciML Scientific Machine Learning Software ecosystem, including cross-language
benchmarks of differential equation solvers and methods for parameter estimation,
training universal differential equations (and subsets like neural ODEs), and more.
Static outputs in pdf, markdown, and html reside in SciMLBenchmarksOutput.
Table of Contents
- Multi-Language Wrapper Benchmarks
- Non-stiff ODEs
- Stiff ODEs
- Method of Lines PDEs
- Dynamical ODEs
- N-body problems
- Nonstiff SDEs
- Stiff SDEs
- Nonstiff DDEs
- Stiff DDEs
- Jump Equations
- Parameter Estimation
The following tests were developed for the paper Adaptive Methods for Stochastic Differential Equations via Natural Embeddings and Rejection Sampling with Memory. These notebooks track their latest developments.
The following is a quick summary of the benchmarks. These paint broad strokes
over the set of tested equations and some specific examples may differ.
- OrdinaryDiffEq.jl's methods are the most efficient by a good amount
Vern methods tend to do the best in every benchmark of this category
- At lower tolerances,
Tsit5 does well consistently.
- ARKODE and Hairer's
dop853 perform very similarly, but are both
far less efficient than the
- The multistep methods,
lsoda, tend to not do very well.
- The ODEInterface multistep method
ddeabm does not do as well as the other
- ODE.jl's methods are not able to consistently solve the problems.
- Fixed time step methods are less efficient than the adaptive methods.
- In this category, the best methods are much more problem dependent.
- For smaller problems:
TRBDF2 tend to be the most efficient at high
Rodas5 tend to be the most efficient at low tolerances.
- For larger problems (Filament PDE):
CVODE_BDF does the best at all tolerances.
- The ESDIRK methods like
KenCarp4 can come close.
radau is always the most efficient when tolerances go to the low extreme
- Fixed time step methods tend to diverge on every tested problem because the
high stiffness results in divergence of the Newton solvers.
- ARKODE is very inconsistent and requires a lot of tweaking in order to not
diverge on many of the tested problems. When it doesn't diverge, the similar
algorithms in OrdinaryDiffEq.jl (
KenCarp4) are much more efficient in most
- ODE.jl and GeometricIntegrators.jl fail to converge on any of the tested
- Higher order (generally order >=6) symplectic integrators are much more
efficient than the lower order counterparts.
- For high accuracy, using a symplectic integrator is not preferred. Their extra
cost is not necessary since the other integrators are able to not drift simply
due to having low enough error.
- In this class, the
DPRKN methods are by far the most efficient. The
methods do well for not being specific to the domain.
- For simple 1-dimensional SDEs at low accuracy, the
can do well. Beyond that, they are simply outclassed.
SRI methods both are very similar within-class on the simple
SRA3 is the most efficient when applicable and the tolerances are low.
- Generally, only low accuracy is necessary to get to sampling error of the mean.
- The adaptive method is very conservative with error estimates.
- The high order adaptive methods (
SRIW1) generally do well on stiff problems.
- The "standard" low-order implicit methods,
not do well on all stiff problems. Some exceptions apply to well-behaved
problems like the Stochastic Heat Equation.
- The efficiency ranking tends to match the ODE Tests, but the cutoff from
low to high tolerance is lower.
Tsit5 does well in a large class of problems here.
Vern methods do well in low tolerance cases.
- The Rosenbrock methods, specifically
Rodas5, perform well.
- Broadly two different approaches have been used, Bayesian Inference and Optimisation
- In general it seems that the optimisation algorithms perform more accurately but that can be
attributed to the larger number of data points being used in the optimisation cases, Bayesian
approach tends to be slower of the two and hence lesser data points are used, accuracy can
increase if proper data is used.
- Within the different available optimisation algorithms, BBO from the BlackBoxOptim package and GN_CRS2_LM
for the global case while LD_SLSQP,LN_BOBYQA and LN_NELDERMEAD for the local case from the NLopt package
perform the best.
- Another algorithm being used is the QuadDIRECT algorithm, it gives very good results in the shorter problem case
but doesn't do very well in the case of the longer problems.
- The choice of global versus local optimization make a huge difference in the timings. BBO tends to find
the correct solution for a global optimization setup. For local optimization, most methods in NLopt,
like :LN_BOBYQA, solve the problem very fast but require a good initial condition.
- The different backends options available for Bayesian method offer some tradeoffs beteween
time, accuracy and control. It is observed that sufficiently high accuracy can be observed with
any of the backends with the fine tuning of stepsize, constraints on the parameters, tightness of the
priors and number of iterations being passed.
To run the tutorials interactively via Jupyter notebooks and benchmark on your
- Run Weave for the file (or folder) you are interested in
- Activate the appropriate environment
- Open and run the notebook.
Note: Since notebooks default to looking for a Project.toml file at the same level or parent folder, you might need to move the notebook to the folder with the appropriate Project.toml.
Example (starting from the project root folder)
SciMLBenchmarks.weave_file("benchmarks/Jumps", "Diffusion_CTRW.jmd", [:notebook])
Diffusion_CTRW.ipynb to "benchmarks/Jumps" and open the notebook.
All of the files are generated from the Weave.jl files in the
benchmarks folder. To run the generation process, do for example:
]activate SciMLBenchmarks # Get all of the packages
To generate all of the files in a folder, for example, run:
To generate all of the notebooks, do:
Each of the benchmarks displays the computer characteristics at the bottom of
the benchmark. Since performance-necessary computations are normally performed on
compute clusters, the official benchmarks use a workstation with an
Intel Xeon CPU E5-2680 v4 @ 2.40GHz to match the performance characteristics of
a standard node in a high performance computing (HPC) cluster or cloud computing
Inspecting Benchmark Results
To see benchmark results before merging, click into the BuildKite, click onto
Artifacts, and then investigate the trained results.