reproducibility issues for planning_partial_order_planner.ipynb - prints "Probably Wrong" or "Couldn't find a solution"

[ellemcfarlane]

Problem: PartialOrderPlanner.execute() sometimes either 1. finds no plan: "Couldn't find a solution" 2. finds an incorrect one in which it prints beforehand "Probably Wrong"

According to this, the following code should have this output:

st = spare_tire()
pop = PartialOrderPlanner(st)
pop.execute()
Causal Links
(Action(PutOn(Spare, Axle)), At(Spare, Axle), Action(Finish))
(Action(Start), Tire(Spare), Action(PutOn(Spare, Axle)))
(Action(Remove(Flat, Axle)), NotAt(Flat, Axle), Action(PutOn(Spare, Axle)))
(Action(Start), At(Flat, Axle), Action(Remove(Flat, Axle)))
(Action(Remove(Spare, Trunk)), At(Spare, Ground), Action(PutOn(Spare, Axle)))
(Action(Start), At(Spare, Trunk), Action(Remove(Spare, Trunk)))
(Action(Remove(Flat, Axle)), At(Flat, Ground), Action(Finish))

Constraints
Action(Remove(Flat, Axle)) < Action(PutOn(Spare, Axle))
Action(Start) < Action(Finish)
Action(Remove(Spare, Trunk)) < Action(PutOn(Spare, Axle))
Action(Start) < Action(Remove(Spare, Trunk))
Action(Start) < Action(Remove(Flat, Axle))
Action(Remove(Flat, Axle)) < Action(Finish)
Action(PutOn(Spare, Axle)) < Action(Finish)
Action(Start) < Action(PutOn(Spare, Axle))

Partial Order Plan
[{Action(Start)}, {Action(Remove(Flat, Axle)), Action(Remove(Spare, Trunk))}, {Action(PutOn(Spare, Axle))}, {Action(Finish)}]

However, sometimes when I run it (from restarted kernel, i.e. no variables), I get:

Probably Wrong
Causal Links
(PutOn(Spare, Axle), At(Spare, Axle), Finish)
(Start, Tire(Spare), PutOn(Spare, Axle))
(Remove(Flat, Axle), NotAt(Flat, Axle), PutOn(Spare, Axle))
(Start, Tire(Flat), Remove(Flat, Axle))
(Start, At(Flat, Axle), Remove(Flat, Axle))
(Remove(Spare, Flat), At(Spare, Ground), PutOn(Spare, Axle))
(Start, Tire(Spare), Remove(Spare, Flat))
(Remove(Flat, Axle), At(Flat, Ground), Finish)

Constraints
Remove(Flat, Axle) < Finish
Start < Remove(Spare, Flat)
Start < Finish
Start < Remove(Flat, Axle)
Remove(Flat, Axle) < PutOn(Spare, Axle)
PutOn(Spare, Axle) < Finish
Remove(Spare, Flat) < PutOn(Spare, Axle)
Start < PutOn(Spare, Axle)

Partial Order Plan
[{Start}, {Remove(Spare, Flat), Remove(Flat, Axle)}, {PutOn(Spare, Axle)}, {Finish}]

or just no plan at all:
Couldn't find a solution (None, None)

I can't find any pattern that would indicate why this should happen. Also, if other plans are run in the same session (with no other code run in between) sometimes they appear to randomly fail (incorrect or not found) despite the success of other plans, e.g. I have found success for spare_tire but then a fail for simple_blocks_world:

sbw = simple_blocks_world()
pop = PartialOrderPlanner(sbw)
pop.execute()
Couldn't find a solution
(None, None)

Have also tried:

• checking out the original commit for this notebook 55cc39d and using python 3.7 but I get similarly inconsistent results.

My environment:

• Python 3.9.12
• Mac M2, Ventura 13.2.1
• packages:

absl-py==1.4.0
anyio==3.6.2
appnope==0.1.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
asgiref==3.6.0
astroid==2.14.2
asttokens==2.2.1
astunparse==1.6.3
attrs==22.2.0
backcall==0.2.0
beautifulsoup4==4.11.2
black==23.1.0
bleach==6.0.0
cachetools==5.3.0
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==3.0.1
click==8.1.3
comm==0.1.2
contourpy==1.0.7
coverage==7.2.1
cvxopt==1.3.0
cycler==0.11.0
debugpy==1.6.6
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.6
Django==4.1.7
ecos==2.0.12
exceptiongroup==1.1.0
executing==1.2.0
fastjsonschema==2.16.3
flake8==6.0.0
flatbuffers==23.1.21
fonttools==4.38.0
fqdn==1.5.1
gast==0.4.0
google-auth==2.16.2
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.51.3
h5py==3.8.0
idna==3.4
image==1.5.33
importlib-metadata==6.0.0
importlib-resources==5.12.0
iniconfig==2.0.0
ipykernel==6.21.2
ipython==8.11.0
ipython-genutils==0.2.0
ipythonblocks==1.9.0
ipywidgets==8.0.4
isoduration==20.11.0
isort==5.12.0
jedi==0.18.2
Jinja2==3.1.2
jsonpointer==2.3
jsonschema==4.17.3
jupyter==1.0.0
jupyter-console==6.6.2
jupyter-events==0.6.3
jupyter_client==8.0.3
jupyter_core==5.2.0
jupyter_server==2.3.0
jupyter_server_terminals==0.4.4
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.5
keras==2.11.0
kiwisolver==1.4.4
lazy-object-proxy==1.9.0
libclang==15.0.6.1
Markdown==3.4.1
MarkupSafe==2.1.2
matplotlib==3.7.0
matplotlib-inline==0.1.6
mccabe==0.7.0
mistune==2.0.5
mypy-extensions==1.0.0
nbclassic==0.5.2
nbclient==0.7.2
nbconvert==7.2.9
nbformat==5.7.3
nest-asyncio==1.5.6
networkx==3.0
notebook==6.5.2
notebook_shim==0.2.2
numpy==1.24.2
oauthlib==3.2.2
opencv-python==4.7.0.72
opt-einsum==3.3.0
osqp==0.6.2.post8
packaging==23.0
pandas==1.5.3
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.11.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.4.0
platformdirs==3.0.0
pluggy==1.0.0
prometheus-client==0.16.0
prompt-toolkit==3.0.38
protobuf==3.19.6
psutil==5.9.4
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.10.0
pycparser==2.21
pyflakes==3.0.1
Pygments==2.14.0
pylint==2.16.3
pyparsing==3.0.9
pyrsistent==0.19.3
pytest==7.2.1
pytest-cov==4.0.0
python-dateutil==2.8.2
python-json-logger==2.0.7
pytz==2022.7.1
PyYAML==6.0
pyzmq==25.0.0
qdldl==0.1.5.post3
qpsolvers==3.0.0
qtconsole==5.4.0
QtPy==2.3.0
quadprog==0.1.11
requests==2.28.2
requests-oauthlib==1.3.1
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rsa==4.9
scipy==1.10.1
scs==3.2.2
Send2Trash==1.8.0
six==1.16.0
sniffio==1.3.0
sortedcontainers==2.4.0
soupsieve==2.4
sqlparse==0.4.3
stack-data==0.6.2
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-estimator==2.11.0
tensorflow-io-gcs-filesystem==0.31.0
termcolor==2.2.0
terminado==0.17.1
tinycss2==1.2.1
tomli==2.0.1
tomlkit==0.11.6
tornado==6.2
traitlets==5.9.0
typing_extensions==4.5.0
uri-template==1.2.0
urllib3==1.26.14
wcwidth==0.2.6
webcolors==1.12
webencodings==0.5.1
websocket-client==1.5.1
Werkzeug==2.2.3
widgetsnbextension==4.0.5
wrapt==1.15.0
zipp==3.15.0

If issues like this could be avoided by pinning package versions in requirements.txt or at least printing and saving the versions used in the travis ci builds, that would be nice.