MDPs and POMDPs in Julia - An interface for defining, solving, and simulating discrete and continuous, fully and partially observable Markov decision processes.



Linux Mac OS X
Build Status Build Status

Docs Gitter

This package provides a core interface for working with Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). For examples, please see the Gallery.

Our goal is to provide a common programming vocabulary for:

  1. Expressing problems as MDPs and POMDPs.
  2. Writing solver software.
  3. Running simulations efficiently.

There are nested interfaces for expressing and interacting with (PO)MDPs: When the explicit interface is used, the transition and observation probabilities are explicitly defined using api functions or tables; when the generative interface is used, only a single step simulator (e.g. (s', o, r) = G(s,a)) needs to be defined.

For help, please post to the Google group, or on gitter. Check releases for information on changes.

POMDPs.jl and all packages in the JuliaPOMDP project are fully supported on Linux and OS X. Windows support is available for all native Julia packages*.


To install POMDPs.jl, run the following from the Julia REPL:


To install supported JuliaPOMDP packages including various solvers, first run:

using POMDPs

This installs the JuliaPOMDP registry so that the Julia package manager can find all the available solvers and support packages.

To check available JuliaPOMDP packages, run:

using POMDPs

To install a particular solver (say SARSOP.jl), having installed the Registry, run:


Quick Start

To run a simple simulation of the classic Tiger POMDP using a policy created by the QMDP solver.

using POMDPs, POMDPModels, POMDPSimulators, QMDP
pomdp = TigerPOMDP()

# initialize a solver and compute a policy
solver = QMDPSolver() # from QMDP
policy = solve(solver, pomdp)
belief_updater = updater(policy) # the default QMDP belief updater (discrete Bayesian filter)

# run a short simulation with the QMDP policy
history = simulate(HistoryRecorder(max_steps=10), pomdp, policy, belief_updater)

# look at what happened
for (s, b, a, o) in eachstep(history, "sbao")
    println("State was $s,")
    println("belief was $b,")
    println("action $a was taken,")
    println("and observation $o was received.\n")
println("Discounted reward was $(discounted_reward(history)).")

For more examples with visualization see POMDPGallery.jl.


Several tutorials are hosted in the POMDPExamples repository.


Detailed documentation can be found here.

Docs Docs

Supported Packages

Many packages use the POMDPs.jl interface, including MDP and POMDP solvers, support tools, and extensions to the POMDPs.jl interface.


POMDPs.jl itself contains only the interface for communicating about problem definitions. Most of the functionality for interacting with problems is actually contained in several support tools packages:

Package Build Coverage
POMDPModelTools Build Status Coverage Status
BeliefUpdaters Build Status Coverage Status
POMDPPolicies Build Status Coverage Status
POMDPSimulators Build Status Coverage Status
POMDPModels Build Status Coverage Status
POMDPTesting Build Status Coverage Status
ParticleFilters Build Status codecov.io
RLInterface Build Status Coverage Status

MDP solvers:

| Package | Build/Coverage | Online/
Offline | Continuous
States | Continuous
Actions | |-------------------|----------------------|----------------------|-------------------------|--| | Value Iteration | Build Status
Coverage Status | Offline | N | N | | Local Approximation Value Iteration | Build Status
Coverage Status | Offline | Y | N | | Monte Carlo Tree Search | Build Status
Coverage Status | Online | Y (DPW) | Y (DPW) |

POMDP solvers:

| Package | Build/Coverage | Online/
Offline | Continuous
States | Continuous
Actions | Continuous
Observations | |-------------------|----------------------|----------------------|-------------------------|--|--| | QMDP | Build Status
Coverage Status | Offline | N | N | N | | FIB | Build Status
Coverage Status | Offline | N | N | N | | SARSOP* | Build Status
Coverage Status | Offline | N | N | N | | BasicPOMCP | Build Status
Coverage Status | Online | Y | N | N1 | | ARDESPOT | Build Status
Coverage Status | Online | Y | N | N1 | | MCVI | Build Status
Coverage Status | Offline | Y | N | Y | | POMDPSolve* | Build Status
Coverage Status | Offline | N | N | N | | POMCPOW | Build Status
Coverage Status | Online | Y | Y2 | Y | | AEMS | Build Status
Coverage Status | Online | N | N | N |

1: Will run, but will not converge to optimal solution

2: Will run, but convergence to optimal solution is not proven, and it will likely not work well on multidimensional action spaces

Reinforcement Learning:

Package Build/Coverage Continuous
TabularTDLearning Build Status
Coverage Status
DeepQLearning Build Status
Coverage Status
Y1 N

1: For POMDPs, it will use the observation instead of the state as input to the policy. See RLInterface.jl for more details.

Packages Awaiting Update

These packages were written for POMDPs.jl in Julia 0.6 and have not been updated to 1.0 yet.

Package Build Coverage
DESPOT Build Status Coverage Status
IncrementalPruning Build Status Coverage Status

Performance Benchmarks:


*These packages require non-Julia dependencies

Citing POMDPs

If POMDPs is useful in your research and you would like to acknowledge it, please cite this paper:

  author  = {Maxim Egorov and Zachary N. Sunberg and Edward Balaban and Tim A. Wheeler and Jayesh K. Gupta and Mykel J. Kochenderfer},
  title   = {{POMDP}s.jl: A Framework for Sequential Decision Making under Uncertainty},
  journal = {Journal of Machine Learning Research},
  year    = {2017},
  volume  = {18},
  number  = {26},
  pages   = {1-5},
  url     = {http://jmlr.org/papers/v18/16-300.html}

First Commit


Last Touched

about 12 hours ago


571 commits