Albumentations is breaking with some edge cases

[talmo]

Bug description

In #1623, we switched from imaug to albumentations for our augmentation backend (see also #648).

This is mostly working, but using it in a real application resulted in some issues related to when points are close to the edge of the image or nan during training. See the logs below.

We should find out if this is a known issue with albumentation and if we should do some preprocessing to handle these cases. We should also add tests for augmentation with nans, zeros and image height/width.

This is on albumentations 1.3.1, installed via conda-forge.

Relevant error

  File "d:\sleap_develop\sleap\nn\data\augmentation.py", line 239, in py_augment
    augmented = self.augmenter(image=img, keypoints=kps)
...
ValueError: Expected x for keypoint (840.0, 93.648865, 0.0, 0.0) to be in the range [0.0, 840], got 840.0.

Also:

ValueError: Expected x for keypoint (nan, nan, 0.0, 0.0) to be in the range [0.0, 840], got nan.

Your personal set up

• OS: Windows 10

• Version(s): eb14764

• SLEAP installation method (listed here):
  • Conda from package
  • Conda from source
  • pip package
  • Apple Silicon Macs

Environment packages

# packages in environment at C:\Miniconda3\envs\sleap_develop:
#
# Name                    Version                   Build  Channel
absl-py                   2.1.0                    pypi_0    pypi
accessible-pygments       0.0.4                    pypi_0    pypi
alabaster                 0.7.13                   pypi_0    pypi
albumentations            1.3.1              pyhd8ed1ab_0    conda-forge
anyio                     3.7.1                    pypi_0    pypi
aom                       3.6.1                h63175ca_0    conda-forge
appdirs                   1.4.4                    pypi_0    pypi
argon2-cffi               23.1.0                   pypi_0    pypi
argon2-cffi-bindings      21.2.0                   pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
attrs                     21.4.0                   pypi_0    pypi
babel                     2.14.0                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
backports-zoneinfo        0.2.1                    pypi_0    pypi
beautifulsoup4            4.12.3                   pypi_0    pypi
black                     21.6b0                   pypi_0    pypi
bleach                    6.0.0                    pypi_0    pypi
blosc                     1.21.5               hdccc3a2_0    conda-forge
brotli                    1.0.9                hcfcfb64_9    conda-forge
brotli-bin                1.0.9                hcfcfb64_9    conda-forge
bzip2                     1.0.8                hcfcfb64_5    conda-forge
c-blosc2                  2.13.2               h183a6f4_0    conda-forge
ca-certificates           2024.2.2             h56e8100_0    conda-forge
cached-property           1.5.2                    pypi_0    pypi
cachetools                5.3.3                    pypi_0    pypi
cattrs                    1.1.1              pyhd8ed1ab_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1                   pypi_0    pypi
cfgv                      3.3.1                    pypi_0    pypi
cfitsio                   4.0.0                hd67004f_0    conda-forge
charls                    2.3.4                h39d44d4_0    conda-forge
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.0.4                    pypi_0    pypi
cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6                    pypi_0    pypi
comm                      0.1.4                    pypi_0    pypi
coverage                  7.2.7                    pypi_0    pypi
cryptography              42.0.5                   pypi_0    pypi
cuda-nvcc                 11.3.58              hb8d16a4_0    nvidia
cudatoolkit               11.3.1              hf2f0253_13    conda-forge
cudnn                     8.2.1.32             h754d62a_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.12.0           py37hcc03f2d_0    conda-forge
dask-core                 2022.2.0           pyhd8ed1ab_0    conda-forge
dav1d                     1.2.1                hcfcfb64_0    conda-forge
debugpy                   1.7.0                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
deprecated                1.2.14                   pypi_0    pypi
distlib                   0.3.8                    pypi_0    pypi
docutils                  0.17.1                   pypi_0    pypi
efficientnet              1.0.0                    pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
expat                     2.6.2                h63175ca_0    conda-forge
ffmpeg                    5.1.2           gpl_h8bb4bc8_112    conda-forge
filelock                  3.12.2                   pypi_0    pypi
flatbuffers               2.0.7                    pypi_0    pypi
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_1    conda-forge
fontconfig                2.14.2               hbde0cde_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.38.0           py37h51bd9d9_0    conda-forge
freeglut                  3.2.2                h63175ca_2    conda-forge
freetype                  2.12.1               hdaf720e_2    conda-forge
fsspec                    2023.1.0           pyhd8ed1ab_0    conda-forge
gast                      0.4.0                    pypi_0    pypi
gettext                   0.21.1               h5728263_0    conda-forge
giflib                    5.2.1                h64bf75a_3    conda-forge
gitdb                     4.0.11                   pypi_0    pypi
gitpython                 3.1.42                   pypi_0    pypi
glib                      2.80.0               h39d0aa6_1    conda-forge
glib-tools                2.80.0               h0a98069_1    conda-forge
google-auth               2.28.2                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
greenlet                  3.0.3                    pypi_0    pypi
grpcio                    1.62.1                   pypi_0    pypi
gst-plugins-base          1.22.9               h001b923_0    conda-forge
gstreamer                 1.22.9               hb4038d2_0    conda-forge
h5py                      3.1.0                    pypi_0    pypi
hdmf                      3.5.5                    pypi_0    pypi
icu                       70.1                 h0e60522_0    conda-forge
identify                  2.5.24                   pypi_0    pypi
idna                      3.6                      pypi_0    pypi
image-classifiers         1.0.0                    pypi_0    pypi
imagecodecs               2021.11.20       py37hfea8715_2    conda-forge
imageio                   2.34.0             pyh4b66e23_0    conda-forge
imagesize                 1.4.1                    pypi_0    pypi
imgstore                  0.2.9                    pypi_0    pypi
importlib-metadata        6.7.0                    pypi_0    pypi
importlib-resources       5.12.0                   pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
intel-openmp              2024.0.0         h57928b3_49841    conda-forge
ipykernel                 6.16.2                   pypi_0    pypi
ipython                   7.34.0                   pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                7.8.1                    pypi_0    pypi
jaraco-classes            3.2.3                    pypi_0    pypi
jasper                    2.0.33               hc2e4405_1    conda-forge
jedi                      0.17.2                   pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   hcfcfb64_3    conda-forge
jsmin                     3.0.1              pyhd8ed1ab_0    conda-forge
json5                     0.9.16                   pypi_0    pypi
jsonpickle                1.2                        py_0    conda-forge
jsonschema                4.17.3                   pypi_0    pypi
jupyter-cache             0.4.3                    pypi_0    pypi
jupyter-client            7.4.9                    pypi_0    pypi
jupyter-core              4.12.0                   pypi_0    pypi
jupyter-server            1.13.5                   pypi_0    pypi
jupyter-server-mathjax    0.2.6                    pypi_0    pypi
jupyter-sphinx            0.3.2                    pypi_0    pypi
jupyterlab                3.3.4                    pypi_0    pypi
jupyterlab-pygments       0.2.2                    pypi_0    pypi
jupyterlab-server         2.16.3                   pypi_0    pypi
jupyterlab-widgets        1.1.7                    pypi_0    pypi
jxrlib                    1.1                  hcfcfb64_3    conda-forge
keras                     2.7.0                    pypi_0    pypi
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
keyring                   24.1.1                   pypi_0    pypi
kiwisolver                1.4.4            py37h8c56517_0    conda-forge
krb5                      1.20.1               heb0366b_0    conda-forge
lcms2                     2.14                 h90d422f_0    conda-forge
lerc                      3.0                  h0e60522_0    conda-forge
libaec                    1.1.3                h63175ca_0    conda-forge
libblas                   3.9.0              21_win64_mkl    conda-forge
libbrotlicommon           1.0.9                hcfcfb64_9    conda-forge
libbrotlidec              1.0.9                hcfcfb64_9    conda-forge
libbrotlienc              1.0.9                hcfcfb64_9    conda-forge
libcblas                  3.9.0              21_win64_mkl    conda-forge
libclang                  18.1.1                   pypi_0    pypi
libclang13                15.0.7          default_h85b4d89_4    conda-forge
libcurl                   8.1.2                h68f0423_0    conda-forge
libdeflate                1.10                 h8ffe710_0    conda-forge
libexpat                  2.6.2                h63175ca_0    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libglib                   2.80.0               h39d0aa6_1    conda-forge
libhwloc                  2.9.3           default_haede6df_1009    conda-forge
libiconv                  1.17                 hcfcfb64_2    conda-forge
liblapack                 3.9.0              21_win64_mkl    conda-forge
liblapacke                3.9.0              21_win64_mkl    conda-forge
libogg                    1.3.4                h8ffe710_1    conda-forge
libopencv                 4.6.0            py37hefdbd86_5    conda-forge
libopus                   1.3.1                h8ffe710_1    conda-forge
libpng                    1.6.43               h19919ed_0    conda-forge
libprotobuf               3.21.8               h12be248_0    conda-forge
libsodium                 1.0.18               h8d14728_1    conda-forge
libsqlite                 3.45.2               hcfcfb64_0    conda-forge
libssh2                   1.11.0               h7dfc565_0    conda-forge
libtiff                   4.4.0                hc4061b1_0    conda-forge
libvorbis                 1.3.7                h0e60522_0    conda-forge
libwebp                   1.3.2                hcfcfb64_1    conda-forge
libwebp-base              1.3.2                hcfcfb64_0    conda-forge
libxcb                    1.13              hcd874cb_1004    conda-forge
libxml2                   2.12.6               hc3477c8_0    conda-forge
libxslt                   1.1.39               h3df6e99_0    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
libzopfli                 1.0.3                h0e60522_0    conda-forge
linkify-it-py             2.0.3                    pypi_0    pypi
livereload                2.6.3                    pypi_0    pypi
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lxml                      5.1.0                    pypi_0    pypi
lz4-c                     1.9.3                h8ffe710_1    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
markdown                  3.4.4                    pypi_0    pypi
markdown-it-py            1.1.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib-base           3.5.3            py37hbaab90a_2    conda-forge
matplotlib-inline         0.1.6                    pypi_0    pypi
mdit-py-plugins           0.2.8                    pypi_0    pypi
mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
mistune                   0.8.4                    pypi_0    pypi
mkl                       2024.0.0         h66d3029_49657    conda-forge
more-itertools            9.1.0                    pypi_0    pypi
msys2-conda-epoch         20160418                      1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mypy-extensions           1.0.0                    pypi_0    pypi
myst-nb                   0.13.2                   pypi_0    pypi
myst-parser               0.15.2                   pypi_0    pypi
nbclassic                 0.5.6                    pypi_0    pypi
nbclient                  0.5.13                   pypi_0    pypi
nbconvert                 6.5.4                    pypi_0    pypi
nbdime                    4.0.1                    pypi_0    pypi
nbformat                  5.1.3                    pypi_0    pypi
ndx-pose                  0.1.1                    pypi_0    pypi
nest-asyncio              1.6.0                    pypi_0    pypi
networkx                  2.7                pyhd8ed1ab_0    conda-forge
nixio                     1.5.3                    pypi_0    pypi
nodeenv                   1.8.0                    pypi_0    pypi
notebook                  6.5.6                    pypi_0    pypi
notebook-shim             0.2.4                    pypi_0    pypi
numpy                     1.21.6           py37h2830a78_0    conda-forge
oauthlib                  3.2.2                    pypi_0    pypi
opencv                    4.6.0            py37h03978a9_5    conda-forge
opencv-python-headless    4.9.0.80                 pypi_0    pypi
openh264                  2.3.1                h63175ca_2    conda-forge
openjpeg                  2.5.0                hc9384bd_1    conda-forge
openssl                   3.2.1                hcfcfb64_1    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 24.0                     pypi_0    pypi
pandas                    1.3.5            py37h9386db6_0    conda-forge
pandocfilters             1.5.1                    pypi_0    pypi
parso                     0.7.1                    pypi_0    pypi
partd                     1.4.1              pyhd8ed1ab_0    conda-forge
pathspec                  0.11.2                   pypi_0    pypi
patsy                     0.5.6              pyhd8ed1ab_0    conda-forge
pcre2                     10.43                h17e33f8_0    conda-forge
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.2.0            py37h42a8222_2    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pkginfo                   1.10.0                   pypi_0    pypi
pkgutil-resolve-name      1.3.10                   pypi_0    pypi
platformdirs              4.0.0                    pypi_0    pypi
pluggy                    1.2.0                    pypi_0    pypi
pre-commit                2.21.0                   pypi_0    pypi
prometheus-client         0.17.1                   pypi_0    pypi
prompt-toolkit            3.0.43                   pypi_0    pypi
protobuf                  3.19.6                   pypi_0    pypi
psutil                    5.9.3            py37h51bd9d9_0    conda-forge
pthread-stubs             0.4               hcd874cb_1001    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
py-opencv                 4.6.0            py37h8013278_5    conda-forge
pyasn1                    0.5.1                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pydata-sphinx-theme       0.13.3                   pypi_0    pypi
pygithub                  2.2.0                    pypi_0    pypi
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pyjwt                     2.8.0                    pypi_0    pypi
pykalman                  0.9.5                      py_1    conda-forge
pynacl                    1.5.0                    pypi_0    pypi
pynwb                     2.3.3                    pypi_0    pypi
pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
pyrsistent                0.19.3                   pypi_0    pypi
pyside2                   5.15.6           py37h7185d37_2    conda-forge
pytest                    7.4.4                    pypi_0    pypi
pytest-cov                3.0.0                    pypi_0    pypi
pytest-qt                 4.2.0                    pypi_0    pypi
pytest-xvfb               3.0.0                    pypi_0    pypi
python                    3.7.12          h900ac77_100_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-rapidjson          1.9              py37h7f67f24_0    conda-forge
python_abi                3.7                     4_cp37m    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyvirtualdisplay          3.0                      pypi_0    pypi
pywavelets                1.3.0            py37h3a130e4_1    conda-forge
pywin32                   306                      pypi_0    pypi
pywin32-ctypes            0.2.2                    pypi_0    pypi
pywinpty                  1.1.6                    pypi_0    pypi
pyyaml                    6.0              py37hcc03f2d_4    conda-forge
pyzmq                     24.0.1           py37h7347f05_0    conda-forge
qimage2ndarray            1.10.0                   pypi_0    pypi
qt-main                   5.15.8               h720456b_6    conda-forge
qt-webengine              5.15.8               h5b1ea0b_0    conda-forge
qtpy                      2.4.1              pyhd8ed1ab_0    conda-forge
qudida                    0.0.4              pyhd8ed1ab_0    conda-forge
readme-renderer           37.3                     pypi_0    pypi
regex                     2023.12.25               pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
requests-oauthlib         1.4.0                    pypi_0    pypi
requests-toolbelt         1.0.0                    pypi_0    pypi
rfc3986                   2.0.0                    pypi_0    pypi
rich                      13.7.1             pyhd8ed1ab_0    conda-forge
ruamel-yaml               0.18.6                   pypi_0    pypi
ruamel-yaml-clib          0.2.8                    pypi_0    pypi
scikit-image              0.19.2           py37h9386db6_0    conda-forge
scikit-learn              1.0              py37ha78be43_1    conda-forge
scikit-video              1.1.11             pyh24bf2e0_0    conda-forge
scipy                     1.7.3            py37hb6553fb_0    conda-forge
seaborn                   0.12.2               hd8ed1ab_0    conda-forge
seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
segmentation-models       1.0.1                    pypi_0    pypi
send2trash                1.8.2                    pypi_0    pypi
setuptools                59.8.0           py37h03978a9_1    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleap                     1.3.3                     dev_0    <develop>
smmap                     5.0.1                    pypi_0    pypi
snappy                    1.1.10               hfb803bf_0    conda-forge
sniffio                   1.3.1                    pypi_0    pypi
snowballstemmer           2.2.0                    pypi_0    pypi
soupsieve                 2.4.1                    pypi_0    pypi
sphinx                    4.5.0                    pypi_0    pypi
sphinx-autobuild          2021.3.14                pypi_0    pypi
sphinx-book-theme         1.0.1                    pypi_0    pypi
sphinx-copybutton         0.5.2                    pypi_0    pypi
sphinx-togglebutton       0.3.2                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.0                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi
sqlalchemy                1.4.52                   pypi_0    pypi
sqlite                    3.45.2               hcfcfb64_0    conda-forge
statsmodels               0.13.2           py37h3a130e4_0    conda-forge
svt-av1                   1.4.1                h63175ca_0    conda-forge
tbb                       2021.11.0            h91493d7_1    conda-forge
tensorboard               2.10.1                   pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.7.0                    pypi_0    pypi
tensorflow-estimator      2.7.0                    pypi_0    pypi
tensorflow-hub            0.12.0             pyhca92ed8_0    conda-forge
tensorflow-io-gcs-filesystem 0.31.0                   pypi_0    pypi
termcolor                 2.3.0                    pypi_0    pypi
terminado                 0.17.1                   pypi_0    pypi
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tifffile                  2021.11.2          pyhd8ed1ab_0    conda-forge
tinycss2                  1.2.1                    pypi_0    pypi
tk                        8.6.13               h5226925_1    conda-forge
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
toolz                     0.12.1             pyhd8ed1ab_0    conda-forge
tornado                   6.2                      pypi_0    pypi
tqdm                      4.66.2                   pypi_0    pypi
traitlets                 5.9.0                    pypi_0    pypi
twine                     3.3.0                    pypi_0    pypi
typed-ast                 1.5.5                    pypi_0    pypi
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzdata                    2024.1                   pypi_0    pypi
tzlocal                   5.1                      pypi_0    pypi
uc-micro-py               1.0.3                    pypi_0    pypi
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
unicodedata2              14.0.0           py37hcc03f2d_1    conda-forge
urllib3                   2.0.7                    pypi_0    pypi
vc                        14.3                hcf57466_18    conda-forge
vc14_runtime              14.38.33130         h82b7239_18    conda-forge
virtualenv                20.25.1                  pypi_0    pypi
vs2015_runtime            14.38.33130         hcb4865c_18    conda-forge
wcwidth                   0.2.13                   pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.6.1                    pypi_0    pypi
werkzeug                  2.2.3                    pypi_0    pypi
wheel                     0.43.0             pyhd8ed1ab_0    conda-forge
widgetsnbextension        3.6.6                    pypi_0    pypi
wrapt                     1.16.0                   pypi_0    pypi
x264                      1!164.3095           h8ffe710_2    conda-forge
x265                      3.5                  h2d74725_3    conda-forge
xorg-libxau               1.0.11               hcd874cb_0    conda-forge
xorg-libxdmcp             1.1.3                hcd874cb_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
yaml                      0.2.5                h8ffe710_2    conda-forge
zeromq                    4.3.4                h0e60522_1    conda-forge
zfp                       0.5.5                h0e60522_8    conda-forge
zipp                      3.15.0                   pypi_0    pypi
zlib-ng                   2.0.7                hcfcfb64_0    conda-forge
zstd                      1.5.5                h12be248_0    conda-forge

Logs

2024-03-22 17:20:12.626901: W tensorflow/core/framework/op_kernel.cc:1733] INVALID_ARGUMENT: ValueError: Expected x for keypoint (840.0, 93.648865, 0.0, 0.0) to be in the range [0.0, 840], got 840.0.
Traceback (most recent call last):

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 273, in __call__
    return func(device, token, args)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 151, in __call__
    outputs = self._call(device, args)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 158, in _call
    ret = self._func(*args)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 649, in wrapper
    return func(*args, **kwargs)

  File "d:\sleap_develop\sleap\nn\data\augmentation.py", line 239, in py_augment
    augmented = self.augmenter(image=img, keypoints=kps)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\composition.py", line 207, in __call__
    p.preprocess(data)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\utils.py", line 83, in preprocess
    data[data_name] = self.check_and_convert(data[data_name], rows, cols, direction="to")

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\utils.py", line 91, in check_and_convert
    return self.convert_to_albumentations(data, rows, cols)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 146, in convert_to_albumentations
    angle_in_degrees=params.angle_in_degrees,

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 271, in convert_keypoints_to_albumentations
    for kp in keypoints

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 271, in <listcomp>
    for kp in keypoints

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 220, in convert_keypoint_to_albumentations
    check_keypoint(keypoint, rows, cols)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 156, in check_keypoint
    "to be in the range [0.0, {size}], got {value}.".format(kp=kp, name=name, value=value, size=size)

ValueError: Expected x for keypoint (840.0, 93.648865, 0.0, 0.0) to be in the range [0.0, 840], got 840.0.
...
Traceback (most recent call last):

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 273, in __call__
    return func(device, token, args)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 151, in __call__
    outputs = self._call(device, args)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 158, in _call
    ret = self._func(*args)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 649, in wrapper
    return func(*args, **kwargs)

  File "d:\sleap_develop\sleap\nn\data\augmentation.py", line 239, in py_augment
    augmented = self.augmenter(image=img, keypoints=kps)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\composition.py", line 207, in __call__
    p.preprocess(data)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\utils.py", line 83, in preprocess
    data[data_name] = self.check_and_convert(data[data_name], rows, cols, direction="to")

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\utils.py", line 91, in check_and_convert
    return self.convert_to_albumentations(data, rows, cols)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 146, in convert_to_albumentations
    angle_in_degrees=params.angle_in_degrees,

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 271, in convert_keypoints_to_albumentations
    for kp in keypoints

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 271, in <listcomp>
    for kp in keypoints

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 220, in convert_keypoint_to_albumentations
    check_keypoint(keypoint, rows, cols)

  File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 156, in check_keypoint
    "to be in the range [0.0, {size}], got {value}.".format(kp=kp, name=name, value=value, size=size)

ValueError: Expected x for keypoint (nan, nan, 0.0, 0.0) to be in the range [0.0, 840], got nan.


         [[{{node EagerPyFunc}}]] [Op:IteratorGetNext]

[talmo]

Update: The fix should actually be simple -- on this line:

sleap/sleap/nn/data/augmentation.py

Lines 197 to 199 in eb14764

	augmenter=A.Compose(
	aug_stack, keypoint_params=A.KeypointParams(format="xy")
	),

We can just add an optional arg to not discard out-of-bounds keypoints:

            augmenter=A.Compose(
                aug_stack, keypoint_params=A.KeypointParams(
                    format="xy",
                    remove_invisible=False
                )
            ),

In quick testing this seems to bypass the error, but I suppose we should double check for correctness by placing a keypoint out of bounds, rotating, then making sure it's still there.