pomdp example

[slwu89]

Hi @sdwfrost, I'm opening this as a draft because it is clearly a work in progress, but also because I've had some problems with package satisfiability requirements and I wonder if you could help check it out, as I'm not sure from which tutorial the offending package is coming from.

I'm on Julia 1.9.3 and when I attempt to run add POMDPs, POMDPTools, QuickPOMDPs, MCTS, I get the following error:

ERROR: Unsatisfiable requirements detected for package AdvancedPS [576499cb]:
 AdvancedPS [576499cb] log:
 ├─possible versions are: 0.1.0-0.5.1 or uninstalled
 ├─restricted by compatibility requirements with AbstractMCMC [80f14c24] to versions: [0.1.0-0.2.0, 0.2.2-0.5.1] or uninstalled
 │ └─AbstractMCMC [80f14c24] log:
 │   ├─possible versions are: 0.1.0-5.0.0 or uninstalled
 │   ├─restricted by compatibility requirements with NestedSamplers [41ceaf6f] to versions: [0.5.6-1.0.1, 3.0.2-4.4.2]
 │   │ └─NestedSamplers [41ceaf6f] log:
 │   │   ├─possible versions are: 0.1.0-0.8.3 or uninstalled
 │   │   └─restricted to versions * by an explicit requirement, leaving only versions: 0.1.0-0.8.3
 │   ├─restricted by compatibility requirements with MCMCChains [c7f686f2] to versions: 0.4.0-3.3.1, leaving only versions: [0.5.6-1.0.1, 3.0.2-3.3.1]
 │   │ └─MCMCChains [c7f686f2] log:
 │   │   ├─possible versions are: 0.2.4-6.0.4 or uninstalled
 │   │   ├─restricted to versions * by an explicit requirement, leaving only versions: 0.2.4-6.0.4
 │   │   ├─restricted by compatibility requirements with NestedSamplers [41ceaf6f] to versions: 3.0.0-6.0.4
 │   │   │ └─NestedSamplers [41ceaf6f] log: see above
 │   │   ├─restricted by compatibility requirements with MCMCDiagnosticTools [be115224] to versions: 0.2.4-4.14.1 or uninstalled, leaving only versions: 3.0.0-4.14.1
 │   │   │ └─MCMCDiagnosticTools [be115224] log:
 │   │   │   ├─possible versions are: 0.1.0-0.3.8 or uninstalled
 │   │   │   └─restricted by compatibility requirements with Distributions [31c24e10] to versions: uninstalled
 │   │   │     └─Distributions [31c24e10] log:
 │   │   │       ├─possible versions are: 0.16.0-0.25.102 or uninstalled
 │   │   │       ├─restricted to versions * by an explicit requirement, leaving only versions: 0.16.0-0.25.102
 │   │   │       ├─restricted by compatibility requirements with ApproxBayes [f5f396d3] to versions: 0.16.0-0.23.12
 │   │   │       │ └─ApproxBayes [f5f396d3] log:
 │   │   │       │   ├─possible versions are: 0.1.0-0.3.2 or uninstalled
 │   │   │       │   ├─restricted to versions * by an explicit requirement, leaving only versions: 0.1.0-0.3.2
 │   │   │       │   ├─restricted by compatibility requirements with RecipesBase [3cdcf5f2] to versions: [0.1.0, 0.3.1-0.3.2] or uninstalled, leaving only versions: [0.1.0, 0.3.1-0.3.2]
 │   │   │       │   │ └─RecipesBase [3cdcf5f2] log:
 │   │   │       │   │   ├─possible versions are: 0.4.0-1.3.4 or uninstalled
 │   │   │       │   │   ├─restricted by compatibility requirements with SDDP [f4570300] to versions: [0.4.0-0.7.0, 1.0.0-1.3.4]
 │   │   │       │   │   │ └─SDDP [f4570300] log:
 │   │   │       │   │   │   ├─possible versions are: 0.1.0-1.6.6 or uninstalled
 │   │   │       │   │   │   ├─restricted to versions * by an explicit requirement, leaving only versions: 0.1.0-1.6.6
 │   │   │       │   │   │   ├─restricted by compatibility requirements with RecipesBase [3cdcf5f2] to versions: 0.3.2-1.6.6 or uninstalled, leaving only versions: 0.3.2-1.6.6
 │   │   │       │   │   │   │ └─RecipesBase [3cdcf5f2] log: see above
 │   │   │       │   │   │   └─restricted by compatibility requirements with Reexport [189a3867] to versions: 0.3.13-1.6.6 or uninstalled, leaving only versions: 0.3.13-1.6.6
 │   │   │       │   │   │     └─Reexport [189a3867] log:
 │   │   │       │   │   │       ├─possible versions are: 0.2.0-1.2.2 or uninstalled
 │   │   │       │   │   │       └─restricted by compatibility requirements with POMDPTools [7588e00f] to versions: 1.0.0-1.2.2
 │   │   │       │   │   │         └─POMDPTools [7588e00f] log:
 │   │   │       │   │   │           ├─possible versions are: 0.1.0-0.1.6 or uninstalled
 │   │   │       │   │   │           └─restricted to versions * by an explicit requirement, leaving only versions: 0.1.0-0.1.6
 │   │   │       │   │   └─restricted by compatibility requirements with GpABC [e850a1a4] to versions: 0.8.0-1.3.4, leaving only versions: 1.0.0-1.3.4
 │   │   │       │   │     └─GpABC [e850a1a4] log:
 │   │   │       │   │       ├─possible versions are: 0.1.0-0.1.1 or uninstalled
 │   │   │       │   │       └─restricted to versions * by an explicit requirement, leaving only versions: 0.1.0-0.1.1
 │   │   │       │   └─restricted by compatibility requirements with ProgressMeter [92933f4c] to versions: 0.3.1-0.3.2 or uninstalled, leaving only versions: 0.3.1-0.3.2
 │   │   │       │     └─ProgressMeter [92933f4c] log:
 │   │   │       │       ├─possible versions are: 0.6.0-1.9.0 or uninstalled
 │   │   │       │       └─restricted by compatibility requirements with MCTS [e12ccd36] to versions: 1.0.0-1.9.0
 │   │   │       │         └─MCTS [e12ccd36] log:
 │   │   │       │           ├─possible versions are: 0.4.4-0.5.5 or uninstalled
 │   │   │       │           └─restricted to versions * by an explicit requirement, leaving only versions: 0.4.4-0.5.5
 │   │   │       ├─restricted by compatibility requirements with GpABC [e850a1a4] to versions: 0.21.0-0.25.102, leaving only versions: 0.21.0-0.23.12
 │   │   │       │ └─GpABC [e850a1a4] log: see above
 │   │   │       ├─restricted by compatibility requirements with StatsBase [2913bbd2] to versions: 0.21.1-0.25.102 or uninstalled, leaving only versions: 0.21.1-0.23.12
 │   │   │       │ └─StatsBase [2913bbd2] log:
 │   │   │       │   ├─possible versions are: 0.24.0-0.34.2 or uninstalled
 │   │   │       │   ├─restricted to versions * by an explicit requirement, leaving only versions: 0.24.0-0.34.2
 │   │   │       │   ├─restricted by compatibility requirements with ApproxBayes [f5f396d3] to versions: 0.24.0-0.33.21
 │   │   │       │   │ └─ApproxBayes [f5f396d3] log: see above
 │   │   │       │   └─restricted by compatibility requirements with GpABC [e850a1a4] to versions: 0.32.0-0.33.21
 │   │   │       │     └─GpABC [e850a1a4] log: see above
 │   │   │       └─restricted by compatibility requirements with Turing [fce5fe82] to versions: 0.23.3-0.25.102, leaving only versions: 0.23.3-0.23.12
 │   │   │         └─Turing [fce5fe82] log:
 │   │   │           ├─possible versions are: 0.5.0-0.29.3 or uninstalled
 │   │   │           ├─restricted to versions * by an explicit requirement, leaving only versions: 0.5.0-0.29.3
 │   │   │           ├─restricted by compatibility requirements with MCMCChains [c7f686f2] to versions: [0.5.0-0.6.9, 0.9.0-0.29.3] or uninstalled, leaving only versions: [0.5.0-0.6.9, 0.9.0-0.29.3]
 │   │   │           │ └─MCMCChains [c7f686f2] log: see above
 │   │   │           ├─restricted by compatibility requirements with Bijectors [76274a88] to versions: 0.5.0-0.28.1 or uninstalled, leaving only versions: [0.5.0-0.6.9, 0.9.0-0.28.1]
 │   │   │           │ └─Bijectors [76274a88] log:
 │   │   │           │   ├─possible versions are: 0.1.0-0.13.7 or uninstalled
 │   │   │           │   ├─restricted by compatibility requirements with Distributions [31c24e10] to versions: 0.1.0-0.12.3 or uninstalled
 │   │   │           │   │ └─Distributions [31c24e10] log: see above
 │   │   │           │   ├─restricted by compatibility requirements with Requires [ae029012] to versions: [0.1.0, 0.3.0-0.13.7] or uninstalled, leaving only versions: [0.1.0, 0.3.0-0.12.3] or uninstalled
 │   │   │           │   │ └─Requires [ae029012] log:
 │   │   │           │   │   ├─possible versions are: 0.5.0-1.3.0 or uninstalled
 │   │   │           │   │   └─restricted by compatibility requirements with Weave [44d3d7a6] to versions: 1.0.0-1.3.0
 │   │   │           │   │     └─Weave [44d3d7a6] log:
 │   │   │           │   │       ├─possible versions are: 0.6.0-0.10.12 or uninstalled
 │   │   │           │   │       └─restricted to versions * by an explicit requirement, leaving only versions: 0.6.0-0.10.12
 │   │   │           │   ├─restricted by compatibility requirements with Reexport [189a3867] to versions: 0.8.10-0.13.7 or uninstalled, leaving only versions: 0.8.10-0.12.3 or uninstalled
 │   │   │           │   │ └─Reexport [189a3867] log: see above
 │   │   │           │   └─restricted by compatibility requirements with Turing [fce5fe82] to versions: 0.8.0-0.9.11, leaving only versions: 0.8.10-0.9.11
 │   │   │           │     └─Turing [fce5fe82] log: see above
 │   │   │           ├─restricted by compatibility requirements with AbstractMCMC [80f14c24] to versions: [0.5.0-0.7.3, 0.9.1, 0.10.0-0.14.12, 0.15.15-0.29.3] or uninstalled, leaving only versions: [0.5.0-0.6.9, 0.9.1, 0.10.0-0.14.12, 0.15.15-0.28.1]
 │   │   │           │ └─AbstractMCMC [80f14c24] log: see above
 │   │   │           ├─restricted by compatibility requirements with DynamicPPL [366bfd00] to versions: [0.5.0-0.8.0, 0.9.0-0.25.3] or uninstalled, leaving only versions: [0.5.0-0.6.9, 0.9.1, 0.10.0-0.14.12, 0.15.15-0.25.3]
 │   │   │           │ └─DynamicPPL [366bfd00] log:
 │   │   │           │   ├─possible versions are: 0.1.0-0.23.21 or uninstalled
 │   │   │           │   ├─restricted by compatibility requirements with Compat [34da2185] to versions: 0.1.0-0.23.18 or uninstalled
 │   │   │           │   │ └─Compat [34da2185] log:
 │   │   │           │   │   ├─possible versions are: 1.0.0-4.10.0 or uninstalled
 │   │   │           │   │   ├─restricted by compatibility requirements with BlackBoxOptim [a134a8b2] to versions: 1.0.0-3.46.2
 │   │   │           │   │   │ └─BlackBoxOptim [a134a8b2] log:
 │   │   │           │   │   │   ├─possible versions are: 0.4.0-0.6.2 or uninstalled
 │   │   │           │   │   │   ├─restricted to versions * by an explicit requirement, leaving only versions: 0.4.0-0.6.2
 │   │   │           │   │   │   └─restricted by compatibility requirements with Distributions [31c24e10] to versions: 0.4.0-0.5.0 or uninstalled, leaving only versions: 0.4.0-0.5.0
 │   │   │           │   │   │     └─Distributions [31c24e10] log: see above
 │   │   │           │   │   ├─restricted by julia compatibility requirements to versions: 2.0.0-4.10.0 or uninstalled, leaving only versions: 2.0.0-3.46.2
 │   │   │           │   │   └─restricted by compatibility requirements with DataFrames [a93c6f00] to versions: 3.17.0-4.10.0, leaving only versions: 3.17.0-3.46.2
 │   │   │           │   │     └─DataFrames [a93c6f00] log:
 │   │   │           │   │       ├─possible versions are: 0.11.7-1.6.1 or uninstalled
 │   │   │           │   │       ├─restricted to versions * by an explicit requirement, leaving only versions: 0.11.7-1.6.1
 │   │   │           │   │       ├─restricted by compatibility requirements with POMDPTools [7588e00f] to versions: 0.19.0-1.6.1
 │   │   │           │   │       │ └─POMDPTools [7588e00f] log: see above
 │   │   │           │   │       ├─restricted by compatibility requirements with Compat [34da2185] to versions: 0.11.7-1.3.6 or uninstalled, leaving only versions: 0.19.0-1.3.6
 │   │   │           │   │       │ └─Compat [34da2185] log: see above
 │   │   │           │   │       ├─restricted by compatibility requirements with Reexport [189a3867] to versions: 0.22.3-1.6.1 or uninstalled, leaving only versions: 0.22.3-1.3.6
 │   │   │           │   │       │ └─Reexport [189a3867] log: see above
 │   │   │           │   │       └─restricted by compatibility requirements with PrettyTables [08abe8d2] to versions: [0.11.7-0.21.8, 0.22.4-1.6.1] or uninstalled, leaving only versions: 0.22.4-1.3.6
 │   │   │           │   │         └─PrettyTables [08abe8d2] log:
 │   │   │           │   │           ├─possible versions are: 0.1.0-2.2.8 or uninstalled
 │   │   │           │   │           ├─restricted by compatibility requirements with Reexport [189a3867] to versions: [0.1.0-0.2.1, 0.11.0-2.2.8] or uninstalled
 │   │   │           │   │           │ └─Reexport [189a3867] log: see above
 │   │   │           │   │           └─restricted by compatibility requirements with DataFrames [a93c6f00] to versions: 0.10.0-1.3.1, leaving only versions: 0.11.0-1.3.1
 │   │   │           │   │             └─DataFrames [a93c6f00] log: see above
 │   │   │           │   ├─restricted by compatibility requirements with Bijectors [76274a88] to versions: 0.1.0-0.22.4 or uninstalled
 │   │   │           │   │ └─Bijectors [76274a88] log: see above
 │   │   │           │   ├─restricted by compatibility requirements with AbstractMCMC [80f14c24] to versions: [0.4.1-0.9.8, 0.10.13-0.23.21] or uninstalled, leaving only versions: [0.4.1-0.9.8, 0.10.13-0.22.4] or uninstalled
 │   │   │           │   │ └─AbstractMCMC [80f14c24] log: see above
 │   │   │           │   └─restricted by compatibility requirements with Turing [fce5fe82] to versions: [0.10.2-0.11.4, 0.12.1-0.14.1], leaving only versions: [0.10.13-0.11.4, 0.12.1-0.14.1]
 │   │   │           │     └─Turing [fce5fe82] log: see above
 │   │   │           ├─restricted by compatibility requirements with MCMCChain [1671dc4f] to versions: 0.6.11-0.29.3 or uninstalled, leaving only versions: [0.9.1, 0.10.0-0.14.12, 0.15.15-0.25.3]
 │   │   │           │ └─MCMCChain [1671dc4f] log:
 │   │   │           │   ├─possible versions are: 0.1.0-0.2.3 or uninstalled
 │   │   │           │   └─restricted by julia compatibility requirements to versions: uninstalled
 │   │   │           ├─restricted by compatibility requirements with Reexport [189a3867] to versions: 0.15.11-0.29.3 or uninstalled, leaving only versions: 0.15.15-0.25.3
 │   │   │           │ └─Reexport [189a3867] log: see above
 │   │   │           └─restricted by compatibility requirements with MCMCChains [c7f686f2] to versions: [0.5.0-0.6.9, 0.9.0-0.17.4] or uninstalled, leaving only versions: 0.15.15-0.17.4
 │   │   │             └─MCMCChains [c7f686f2] log: see above
 │   │   ├─restricted by compatibility requirements with RecipesBase [3cdcf5f2] to versions: 3.0.11-6.0.4 or uninstalled, leaving only versions: 3.0.11-4.14.1
 │   │   │ └─RecipesBase [3cdcf5f2] log: see above
 │   │   ├─restricted by compatibility requirements with PrettyTables [08abe8d2] to versions: [0.2.4-3.0.12, 4.7.0-6.0.4] or uninstalled, leaving only versions: [3.0.11-3.0.12, 4.7.0-4.14.1]
 │   │   │ └─PrettyTables [08abe8d2] log: see above
 │   │   └─restricted by compatibility requirements with Turing [fce5fe82] to versions: 4.0.0-4.14.1, leaving only versions: 4.7.0-4.14.1
 │   │     └─Turing [fce5fe82] log: see above
 │   └─restricted by compatibility requirements with Turing [fce5fe82] to versions: 2.1.0-3.3.1, leaving only versions: 3.0.2-3.3.1
 │     └─Turing [fce5fe82] log: see above
 ├─restricted by compatibility requirements with Libtask [6f1fad26] to versions: 0.3.0-0.5.1 or uninstalled
 │ └─Libtask [6f1fad26] log:
 │   ├─possible versions are: 0.1.1-0.8.6 or uninstalled
 │   ├─restricted by compatibility requirements with Turing [fce5fe82] to versions: [0.1.1-0.5.3, 0.6.6-0.8.6]
 │   │ └─Turing [fce5fe82] log: see above
 │   ├─restricted by compatibility requirements with Libtask_jll [3ae2931a] to versions: [0.1.1-0.4.2, 0.6.0-0.8.6] or uninstalled, leaving only versions: [0.1.1-0.4.2, 0.6.6-0.8.6]
 │   │ └─Libtask_jll [3ae2931a] log:
 │   │   ├─possible versions are: 0.3.0-0.5.1 or uninstalled
 │   │   └─restricted by julia compatibility requirements to versions: [0.3.0-0.3.2, 0.5.0-0.5.1] or uninstalled
 │   └─restricted by compatibility requirements with Turing [fce5fe82] to versions: 0.4.0-0.5.3, leaving only versions: 0.4.0-0.4.2
 │     └─Turing [fce5fe82] log: see above
 └─restricted by compatibility requirements with Turing [fce5fe82] to versions: 0.1.0-0.2.4 — no versions left
   └─Turing [fce5fe82] log: see above

[sdwfrost]

Hi @slwu89, sorry for the late reply. This looks fun! I have yet to neaten up the environments within the repository; just use Pkg.activate/Pkg.add/Pkg.instantiate to use a custom environment for the tutorial, and then everything should work fine.

The output doesn't look too promising though! This framework may be better suited to the 'flattening the curve' scenario, as I've implemented (using JuMP) here.

[slwu89]

Thanks @sdwfrost, will do that.

Yeah, it does look distressingly similar to the SDDP example. I guess not too surprising, as they are both alternative ways to solve approximations of the Bellman equation for forward look ahead type problems. I'll check how it works with that alternative scenario. Perhaps I'll also modify SDDP to work on that example too.

[EDIT]: actually I think MDP and SDDP are perhaps not well suited to the "flattening the curve" scenario, at the stochastic transitions mean that constraints such as @constraint(model, [t=2:(T+1)], 0 ≤ I[t] ≤ I_max) cannot be fulfilled. If it makes sense to do so, "goal programming" could be used such that deviations from that constraint could be minimized in the objective (SDDP) or penalized in the reward function (MDP).

[sdwfrost]

You're right that 'soft' constraints may allow MDP/SDDP to work better in the flattening the curve context. If you get POMDP.jl to work in this setting, it would also be interesting to use something like https://juliareinforcementlearning.org (which also implements a MDP).

[slwu89]

I'll give those soft constraints as penalties a try when I get some time for them! Thanks for the link, I haven't seen that before.

[sdwfrost]

Just pinging @slwu89 to see if I can help unblock things...

[slwu89]

Not enough time lately, hopefully soon!

[sdwfrost]

How soon is now? :)

[slwu89]

Haha, good question. The issue is that the soft constraints and goal programming formulation would need more time/effort beyond than simple coding; I'm not sure what's the best expression to enforce such penalties on the optimizer.