This package covers the gradient boosting paradigm: a framework that builds additive expansions based on any fitting criteria.

In machine learning parlance, this is typically referred to as gradient boosting machines, generalized boosted models and stochastic gradient boosting.

Normally, gradient boosting implementations cover a specific algorithm: gradient boosted decision trees. This package covers the framework itself, including such implementations.

References:

- Friedman, Jerome H. "Greedy function approximation: a gradient boosting machine." Annals of Statistics (2001): 1189-1232.
- Friedman, Jerome H. "Stochastic gradient boosting." Computational Statistics & Data Analysis 38.4 (2002): 367-378.
- Hastie, Trevor, et al. The elements of statistical learning. Vol. 2. No. 1. New York: Springer, 2009.
- Ridgeway, Greg. "Generalized Boosted Models: A guide to the gbm package." Update 1.1 (2007).
- Pedregosa, Fabian, et al. "Scikit-learn: Machine learning in Python." The Journal of Machine Learning Research 12 (2011): 2825-2830.
- Natekin, Alexey, and Alois Knoll. "Gradient boosting machines, a tutorial." Frontiers in neurorobotics 7 (2013).

Module `GradientBoost.ML`

is provided for users who are only interested in
using existing gradient boosting algorithms for prediction.
To get a feel for the API,
we will run a demonstration
of gradient boosted decision trees on the iris dataset.

At the moment only two-class classification is handled, so our learner will attempt to separate "setosa" from the other species.

```
using GradientBoost.ML
using RDatasets
# Obtain iris dataset
iris = dataset("datasets", "iris")
instances = array(iris[:, 1:end-1])
labels = [species == "setosa" ? 1.0 : 0.0 for species in array(iris[:, end])]
# Obtain training and test set (20% test)
num_instances = size(instances, 1)
train_ind, test_ind = GradientBoost.Util.holdout(num_instances, 0.2)
```

The gradient boosting (GB) learner comprises of a GB algorithm and what output it must produce. In this case, we shall assign a gradient boosted decision tree to output classes.

```
# Build GBLearner
gbdt = GBDT(;
loss_function = BinomialDeviance(),
sampling_rate = 0.6,
learning_rate = 0.1,
num_iterations = 100
)
gbl = GBLearner(
gbdt, # Gradient boosting algorithm
:class # Output (:class, :class_prob, :regression)
)
```

Currently `Matrix{Float64}`

instances and `Vector{Float64}`

labels are
the only handled types for training and prediction.
In this case, it is not an issue.

```
# Train
ML.fit!(gbl, instances[train_ind, :], labels[train_ind])
# Predict
predictions = ML.predict!(gbl, instances[test_ind, :])
```

If all is well, we should obtain better than baseline accuracy (67%).

```
# Obtain accuracy
accuracy = mean(predictions .== labels[test_ind]) * 100.0
println("GBDT accuracy: $(accuracy)")
```

That concludes the demonstration. Detailed below are the available GB learners.

Documented below are the currently implemented gradient boosting algorithms.

Gradient Boosted Decision Tree algorithm backed by
DecisionTree.jl
regression trees.
Current loss functions covered are:
`LeastSquares`

, `LeastAbsoluteDeviation`

and `BinomialDeviance`

.

```
gbdt = GBDT(;
loss_function = BinomialDeviance(), # Loss function
sampling_rate = 0.6, # Sampling rate
learning_rate = 0.1, # Learning rate
num_iterations = 100, # Number of iterations
tree_options = { # Tree options (DecisionTree.jl regressor)
:maxlabels => 5,
:nsubfeatures => 0
}
)
```

Gradient boosting with a given base learner.
Current loss functions covered are: `LeastSquares`

and `LeastAbsoluteDeviation`

.
In order to use this,
`ML.learner_fit`

and `ML.learner_predict`

functions must be extended.
Example provided below for linear regression found in
GLM.jl.

```
import GLM: fit, predict, LinearModel
# Extend functions
function ML.learner_fit(lf::LossFunction,
learner::Type{LinearModel}, instances, labels)
model = fit(learner, instances, labels)
end
function ML.learner_predict(lf::LossFunction,
learner::Type{LinearModel}, model, instances)
predict(model, instances)
end
```

Once this is done, the algorithm can be instantiated with the respective base learner.

```
gbl = GBBL(
LinearModel; # Base Learner
loss_function = LeastSquares(), # Loss function
sampling_rate = 0.8, # Sampling rate
learning_rate = 0.1, # Learning rate
num_iterations = 100 # Number of iterations
)
gbl = GBLearner(gbl, :regression)
```

All previously developed algorithms follow the framework
provided by `GradientBoost.GB`

.

As this package is in its preliminary stage, major changes may occur in the near future and as such we provide minimal README documentation.

All of what is required to be implemented is exampled below:

```
import GradientBoost.GB
import GradientBoost.LossFunctions: LossFunction
# Must subtype from GBAlgorithm defined in GB module.
type ExampleGB <: GB.GBAlgorithm
loss_function::LossFunction
sampling_rate::FloatingPoint
learning_rate::FloatingPoint
num_iterations::Int
end
# Model training and co-efficient optimization should be done here.
function GB.build_base_func(
gb::ExampleGB, instances, labels, prev_func_pred, psuedo)
model_const = 0.5
model_pred = (instances) -> Float64[
sum(instances[i,:]) for i = 1:size(instances, 1)
]
return (instances) -> model_const .* model_pred(instances)
end
```

A relatively light algorithm
that implements `GBAlgorithm`

is `GBBL`

, found in `src/gb_bl.jl`

.

The links provided below will only work if you are viewing this in the GitHub repository.

See CHANGELOG.yml.

See FUTUREWORK.md.

See CONTRIBUTING.md.

MIT "Expat" License. See LICENSE.md.

07/03/2014

2 months ago

35 commits