The purpose of PackageEvaluator is to attempt to test every Julia package nightly, and to provide the information required to generate the Julia package listing.
This is currently done for Julia 0.6 and nightly, and the tests are run in Ubuntu 14.04 LTS ("Trusty Tahr") virtual machines managed with Vagrant. This allows users to debug why their tests are failing, and allows PackageEvaluator to be run almost anywhere.
The code itself, in particular
scripts/setup.sh, is heavily commented, so check that out for more information.
"My package is failing tests!"
Possible reasons include:
- Your package is out of date. PackageEvaluator tests the last released version of your package, not
master. Make sure you've tagged a version with your bug fixes included.
- You have a binary dependency that BinDeps can't handle.
- If the binary dependency is a commerical package, or does not work on Ubuntu (e.g. OSX only), then the package should be excluded from testing. Please submit a pull request adding a line to
- If the binary dependency is something that is not installable (or shouldn't be installed) through BinDeps, like a Python package or R package, then it should be added to the provisioning script. Please submit a pull request adding a line to
- You have a testing-only dependency that you haven't declared. Create (or check) your package's
- Your package only works on Windows/OSX/one particular *-nix. Your package might need to be excluded from testing. Please submit a pull request adding a line to
src/constants.jl saying your package shouldn't be run.
- Your testing process relies on random numbers. Please make sure you set a seed or use appropriate tolerances if you rely on random numbers in your tests.
- Your package relies on X running. It may be possible to get your package working through the magic of
xvfb. Please submit a pull request adding a line to
src/constants.jl that specifies that your package needs to be run with
- Your package's tests or installation take too long. There is a time limit of 30 minutes for installation, and a seperate 10 minute time limit for testing. You can either reduce your testing time, or exclude your package from testing.
- Your package requires too much memory. The VMs only have 2 GB of RAM. You can either reduce your test memory usage, or exclude your package from testing.
- Your tests aren't being found / wrong test file is being run. Your package needs a
test/runtests.jl file. PackageEvaluator will execute it with
- Something else. You'll probably need to check manually on the testing VM. See next section.
(Licenses are searched for in the files listed in
src/constants.jl. The goal is to support a variety of licenses. If your license isn't detected, please file a pull request with detection logic.)
Using Vagrant and PackageEvaluator
- Vagrant is a tool for creating and managing virtual machines.
- The configuration of the virtual machine, including the operating system to use, live in the
- When the virtual machine(s) are launched with
vagrant up, a provisioning script called
setup.sh is run.
- This script takes two arguments. The first is the version of Julia
to use (
- The second determines the mode to operate in:
setup: set up the machine with Julia and the same
dependencies that are used for a full PackageEvaluator run, but
do not do any testing.
setup and evaluate all the packages.
PZ: evaluate only packages with names beginning with those letters.
- Each combination of settings corresponds to a named virtual machine - see
scripts/Vagrantfile for the list of the VMs.