POMDPs

This package provides a core interface for working with Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs).

Our goal is to provide a common programming vocabulary for:

1. Expressing problems as POMDPs.
2. Writing solver software.
3. Running simulations efficiently.

For problems and solvers that only use a generative model (rather than explicit transition and observation distributions), see GenerativeModels.jl.

For help, please post to the Google group at https://groups.google.com/forum/#!forum/pomdps-users. See NEWS.md for information on changes.

Installation

To install POMDPs.jl, run the following from the Julia REPL:

Pkg.add("POMDPs")

To install a specific supported JuliaPOMDP package run:

using POMDPs
# the following command installs the SARSOP solver, you can add any supported solver this way
POMDPs.add("SARSOP") 

To install all solvers, support tools, and dependencies that are part of JuliaPOMDP, run:

using POMDPs
POMDPs.add_all() # this may take a few minutes

To only install native solvers, without any non-Julia dependecies, run:

using POMDPs
POMDPs.add_all(native_only=true)

Quick Start

Start the Julia REPL and run the following:

using POMDPs
using POMDPModels, POMDPToolbox, QMDP
# initialize problem and solver
pomdp = TigerPOMDP() # from POMDPModels
solver = QMDPSolver() # from QMDP
# compute a policy
policy = solve(solver, pomdp)
#evaluate the policy
belief_updater = updater(policy) # the default QMPD belief updater (discrete Bayesian filter)
init_dist = initial_state_distribution(pomdp) # from POMDPModels
# evaluate the policy
r = simulate(HistoryRecorder(max_steps=100), pomdp, policy, belief_updater, init_dist) # run 100 step simulation

The code above solves the Tiger POMDP using the QMDP algorithm, and evaluates its performance.

Tutorials

The following tutorials aim to get you up to speed with POMDPs.jl:

• MDP Tutorial for beginners gives an overview of using Value Iteration and Monte-Carlo Tree Search with the classic grid world problem
• POMDP Tutorial gives an overview of using SARSOP and QMDP to solve the tiger problem

Documentation

Detailed documentation can be found here.

Supported Packages

Many packages use the POMDPs.jl interface, including MDP and POMDP solvers, support tools, and extensions to the POMDPs.jl interface.

MDP solvers:

Package	Build	Coverage
Value Iteration
Monte Carlo Tree Search

POMDP solvers:

Package	Build	Coverage
QMDP
SARSOP
POMCP
DESPOT
MCVI
POMDPSolve

Support Tools:

Package	Build	Coverage
POMDPToolbox
POMDPModels

Interface Extensions:

Package	Build
GenerativeModels
POMDPBounds

Performance Benchmarks:

Package
DESPOT