Training Model plot not updating on Mac

[roomrys]

Bug description

When training on a Mac (using a .pkg.slp), the centroid Training Model plot of losses at each epoch is not updated past "Preparing to run training". However, the heat-map visualization is updated and the training seems to be running fine. Also strange is that the centered-instance Training Plot is updated as expected.

Expected behaviour

The Training Model plot is updated.

Your personal set up

• OS: macOS 13.5 M2

• Version(s): SLEAP v1.3.3, python 3.9

• SLEAP installation method (listed here):
  • Conda from package
  • Conda from source
  • pip package
  • Apple Silicon Macs

micromamba list

List of packages in environment: "/Users/liezlmaree/micromamba/envs/sa0"

  Name                       Version       Build                   Channel    
────────────────────────────────────────────────────────────────────────────────
  abseil-cpp                 20211102.0    he4e09e4_3              conda-forge
  absl-py                    2.0.0         pyhd8ed1ab_0            conda-forge
  aiohttp                    3.8.5         py39h0f82c59_0          conda-forge
  aiosignal                  1.3.1         pyhd8ed1ab_0            conda-forge
  aom                        3.5.0         h7ea286d_0              conda-forge
  astunparse                 1.6.3         pyhd8ed1ab_0            conda-forge
  async-timeout              4.0.3         pyhd8ed1ab_0            conda-forge
  attrs                      23.1.0        pyh71513ae_1            conda-forge
  blas                       2.118         openblas                conda-forge
  blas-devel                 3.9.0         18_osxarm64_openblas    conda-forge
  blinker                    1.6.2         pyhd8ed1ab_0            conda-forge
  blosc                      1.21.5        hc338f07_0              conda-forge
  brotli                     1.0.9         h1a8c8d9_9              conda-forge
  brotli-bin                 1.0.9         h1a8c8d9_9              conda-forge
  brotli-python              1.0.9         py39h23fbdae_9          conda-forge
  brunsli                    0.1           h9f76cd9_0              conda-forge
  bzip2                      1.0.8         h3422bc3_4              conda-forge
  c-ares                     1.19.1        hb547adb_0              conda-forge
  c-blosc2                   2.10.2        h068da5f_0              conda-forge
  ca-certificates            2023.7.22     hf0a4a13_0              conda-forge
  cached-property            1.5.2         hd8ed1ab_1              conda-forge
  cached_property            1.5.2         pyha770c72_1            conda-forge
  cachetools                 5.3.1         pyhd8ed1ab_0            conda-forge
  cairo                      1.16.0        h73a0509_1014           conda-forge
  cattrs                     1.1.1         pyhd8ed1ab_0            conda-forge
  certifi                    2023.7.22     pyhd8ed1ab_0            conda-forge
  cffi                       1.15.1        py39he153c15_5          conda-forge
  cfitsio                    4.2.0         h2f961c4_0              conda-forge
  charls                     2.3.4         hbdafb3b_0              conda-forge
  charset-normalizer         3.2.0         pyhd8ed1ab_0            conda-forge
  click                      8.1.7         unix_pyh707e725_0       conda-forge
  cloudpickle                2.2.1         pyhd8ed1ab_0            conda-forge
  contourpy                  1.1.1         py39hbd775c9_1          conda-forge
  cryptography               39.0.0        py39haa0b8cc_0          conda-forge
  cycler                     0.11.0        pyhd8ed1ab_0            conda-forge
  cytoolz                    0.12.2        py39h0f82c59_1          conda-forge
  dask-core                  2023.9.2      pyhd8ed1ab_0            conda-forge
  dav1d                      1.2.1         hb547adb_0              conda-forge
  expat                      2.5.0         hb7217d7_1              conda-forge
  ffmpeg                     4.4.2         gpl_hf318d42_112        conda-forge
  font-ttf-dejavu-sans-mono  2.37          hab24e00_0              conda-forge
  font-ttf-inconsolata       3.000         h77eed37_0              conda-forge
  font-ttf-source-code-pro   2.038         h77eed37_0              conda-forge
  font-ttf-ubuntu            0.83          hab24e00_0              conda-forge
  fontconfig                 2.14.2        h82840c6_0              conda-forge
  fonts-conda-ecosystem      1             0                       conda-forge
  fonts-conda-forge          1             0                       conda-forge
  fonttools                  4.42.1        py39h0f82c59_0          conda-forge
  freetype                   2.12.1        hadb7bae_2              conda-forge
  frozenlist                 1.4.0         py39h0f82c59_1          conda-forge
  fsspec                     2023.9.2      pyh1a96a4e_0            conda-forge
  gast                       0.4.0         pyh9f0ad1d_0            conda-forge
  geos                       3.11.2        hb7217d7_0              conda-forge
  gettext                    0.21.1        h0186832_0              conda-forge
  giflib                     5.2.1         h1a8c8d9_3              conda-forge
  glib                       2.78.0        ha614eb4_0              conda-forge
  glib-tools                 2.78.0        ha614eb4_0              conda-forge
  gmp                        6.2.1         h9f76cd9_0              conda-forge
  gnutls                     3.7.8         h9f1a10d_0              conda-forge
  google-auth                2.23.0        pyh1a96a4e_1            conda-forge
  google-auth-oauthlib       0.4.6         pyhd8ed1ab_0            conda-forge
  google-pasta               0.2.0         pyh8c360ce_0            conda-forge
  graphite2                  1.3.13        h9f76cd9_1001           conda-forge
  grpc-cpp                   1.46.4        hcaf9be7_3              conda-forge
  grpcio                     1.46.4        py39hb9dff5f_3          conda-forge
  gst-plugins-base           1.21.3        h8b7775e_1              conda-forge
  gstreamer                  1.21.3        hcb7b3dd_1              conda-forge
  h5py                       3.8.0         nompi_py39hc9149d8_100  conda-forge
  harfbuzz                   5.3.0         hddbc195_0              conda-forge
  hdf5                       1.12.2        nompi_h8968d4b_100      conda-forge
  icu                        70.1          h6b3803e_0              conda-forge
  idna                       3.4           pyhd8ed1ab_0            conda-forge
  imagecodecs                2022.9.26     py39hd7f743f_4          conda-forge
  imageio                    2.31.1        pyh24c5eb1_0            conda-forge
  imgaug                     0.4.0         pyhd8ed1ab_1            conda-forge
  importlib-metadata         6.8.0         pyha770c72_0            conda-forge
  importlib-resources        6.1.0         pyhd8ed1ab_0            conda-forge
  importlib_metadata         6.8.0         hd8ed1ab_0              conda-forge
  importlib_resources        6.1.0         pyhd8ed1ab_0            conda-forge
  jasper                     2.0.33        hc3cd1e9_1              conda-forge
  joblib                     1.3.2         pyhd8ed1ab_0            conda-forge
  jpeg                       9e            h1a8c8d9_3              conda-forge
  jsmin                      3.0.1         pyhd8ed1ab_0            conda-forge
  jsonpickle                 1.2           py_0                    conda-forge
  jxrlib                     1.1           h27ca646_2              conda-forge
  keras                      2.9.0         pyhd8ed1ab_0            conda-forge
  keras-preprocessing        1.1.2         pyhd8ed1ab_0            conda-forge
  kiwisolver                 1.4.5         py39hbd775c9_1          conda-forge
  krb5                       1.19.3        hf9b2bbe_0              conda-forge
  lame                       3.100         h1a8c8d9_1003           conda-forge
  lazy_loader                0.3           pyhd8ed1ab_0            conda-forge
  lcms2                      2.14          h8193b64_0              conda-forge
  lerc                       4.0.0         h9a09cb3_0              conda-forge
  libabseil                  20211102.0    cxx17_h28b99d4_3        conda-forge
  libaec                     1.0.6         hb7217d7_1              conda-forge
  libavif                    0.11.1        h9f83d30_2              conda-forge
  libblas                    3.9.0         18_osxarm64_openblas    conda-forge
  libbrotlicommon            1.0.9         h1a8c8d9_9              conda-forge
  libbrotlidec               1.0.9         h1a8c8d9_9              conda-forge
  libbrotlienc               1.0.9         h1a8c8d9_9              conda-forge
  libcblas                   3.9.0         18_osxarm64_openblas    conda-forge
  libclang                   15.0.7        default_h5dc8d65_3      conda-forge
  libclang13                 15.0.7        default_hc7183e1_3      conda-forge
  libcurl                    7.86.0        hd538317_1              conda-forge
  libcxx                     16.0.6        h4653b0c_0              conda-forge
  libdeflate                 1.14          h1a8c8d9_0              conda-forge
  libedit                    3.1.20191231  hc8eb9b7_2              conda-forge
  libev                      4.33          h642e427_1              conda-forge
  libexpat                   2.5.0         hb7217d7_1              conda-forge
  libffi                     3.4.2         h3422bc3_5              conda-forge
  libgfortran                5.0.0         13_2_0_hd922786_1       conda-forge
  libgfortran5               13.2.0        hf226fd6_1              conda-forge
  libglib                    2.78.0        h24e9cb9_0              conda-forge
  libiconv                   1.17          he4db4b2_0              conda-forge
  libidn2                    2.3.4         h1a8c8d9_0              conda-forge
  liblapack                  3.9.0         18_osxarm64_openblas    conda-forge
  liblapacke                 3.9.0         18_osxarm64_openblas    conda-forge
  libllvm15                  15.0.7        h62b9111_1              conda-forge
  libnghttp2                 1.51.0        hd184df1_0              conda-forge
  libogg                     1.3.4         h27ca646_1              conda-forge
  libopenblas                0.3.24        openmp_hd76b1f2_0       conda-forge
  libopencv                  4.6.0         py39he1c1adf_3          conda-forge
  libopus                    1.3.1         h27ca646_1              conda-forge
  libpng                     1.6.39        h76d750c_0              conda-forge
  libpq                      15.1          hb650857_1              conda-forge
  libprotobuf                3.20.3        hb5ab8b9_0              conda-forge
  libsodium                  1.0.18        h27ca646_1              conda-forge
  libsqlite                  3.43.0        hb31c410_0              conda-forge
  libssh2                    1.10.0        hb80f160_3              conda-forge
  libtasn1                   4.19.0        h1a8c8d9_0              conda-forge
  libtiff                    4.4.0         heb92581_5              conda-forge
  libunistring               0.9.10        h3422bc3_0              conda-forge
  libvorbis                  1.3.7         h9f76cd9_0              conda-forge
  libvpx                     1.11.0        hc470f4d_3              conda-forge
  libwebp-base               1.3.2         hb547adb_0              conda-forge
  libxcb                     1.13          h9b22ae9_1004           conda-forge
  libxml2                    2.10.3        h67585b2_4              conda-forge
  libxslt                    1.1.37        h1bd8bc4_0              conda-forge
  libzlib                    1.2.13        h53f4e23_5              conda-forge
  libzopfli                  1.0.3         h9f76cd9_0              conda-forge
  llvm-openmp                16.0.6        h1c12783_0              conda-forge
  locket                     1.0.0         pyhd8ed1ab_0            conda-forge
  lz4-c                      1.9.4         hb7217d7_0              conda-forge
  markdown                   3.4.4         pyhd8ed1ab_0            conda-forge
  markdown-it-py             3.0.0         pyhd8ed1ab_0            conda-forge
  markupsafe                 2.1.3         py39h0f82c59_1          conda-forge
  matplotlib-base            3.7.2         py39h70c6c4c_0          conda-forge
  mdurl                      0.1.0         pyhd8ed1ab_0            conda-forge
  multidict                  6.0.4         py39h02fc5c5_0          conda-forge
  munkres                    1.1.4         pyh9f0ad1d_0            conda-forge
  mysql-common               8.0.32        hab468bb_0              conda-forge
  mysql-libs                 8.0.32        hea58576_0              conda-forge
  ncurses                    6.4           h7ea286d_0              conda-forge
  nettle                     3.8.1         h63371fa_1              conda-forge
  networkx                   3.1           pyhd8ed1ab_0            conda-forge
  nspr                       4.35          hb7217d7_0              conda-forge
  nss                        3.92          hc6b9969_0              conda-forge
  numpy                      1.22.3        py39h25ab29e_0          anaconda   
  numpy-base                 1.22.3        py39h974a1f5_0          anaconda   
  oauthlib                   3.2.2         pyhd8ed1ab_0            conda-forge
  openblas                   0.3.24        openmp_hce3e5ba_0       conda-forge
  opencv                     4.6.0         py39hdf13c20_3          conda-forge
  openh264                   2.3.1         hb7217d7_2              conda-forge
  openjpeg                   2.5.0         h5d4e404_1              conda-forge
  openssl                    1.1.1w        h53f4e23_0              conda-forge
  opt_einsum                 3.3.0         pyhd8ed1ab_1            conda-forge
  p11-kit                    0.24.1        h29577a5_0              conda-forge
  packaging                  23.1          pyhd8ed1ab_0            conda-forge
  pandas                     2.0.3         py39h6b13a34_1          conda-forge
  partd                      1.4.1         pyhd8ed1ab_0            conda-forge
  patsy                      0.5.3         pyhd8ed1ab_0            conda-forge
  pcre2                      10.40         hb34f9b4_0              conda-forge
  pillow                     9.2.0         py39h139752e_3          conda-forge
  pip                        23.2.1        pyhd8ed1ab_0            conda-forge
  pixman                     0.40.0        h27ca646_0              conda-forge
  protobuf                   3.20.3        py39h23fbdae_1          conda-forge
  psutil                     5.9.5         py39h0f82c59_1          conda-forge
  pthread-stubs              0.4           h27ca646_1001           conda-forge
  py-opencv                  4.6.0         py39hfa6204d_3          conda-forge
  pyasn1                     0.5.0         pyhd8ed1ab_0            conda-forge
  pyasn1-modules             0.3.0         pyhd8ed1ab_0            conda-forge
  pycparser                  2.21          pyhd8ed1ab_0            conda-forge
  pygments                   2.16.1        pyhd8ed1ab_0            conda-forge
  pyjwt                      2.8.0         pyhd8ed1ab_0            conda-forge
  pykalman                   0.9.5         py_1                    conda-forge
  pyopenssl                  23.2.0        pyhd8ed1ab_1            conda-forge
  pyparsing                  3.0.9         pyhd8ed1ab_0            conda-forge
  pyside2                    5.15.8        py39h0adaba8_2          conda-forge
  pysocks                    1.7.1         pyha2e5f31_6            conda-forge
  python                     3.9.15        h2d96c93_0_cpython      conda-forge
  python-dateutil            2.8.2         pyhd8ed1ab_0            conda-forge
  python-flatbuffers         1.12          pyhd8ed1ab_1            conda-forge
  python-rapidjson           1.11          py39hb198ff7_1          conda-forge
  python-tzdata              2023.3        pyhd8ed1ab_0            conda-forge
  python_abi                 3.9           4_cp39                  conda-forge
  pytz                       2023.3.post1  pyhd8ed1ab_0            conda-forge
  pyu2f                      0.1.5         pyhd8ed1ab_0            conda-forge
  pywavelets                 1.4.1         py39h4d8bf0d_0          conda-forge
  pyyaml                     6.0.1         py39h0f82c59_1          conda-forge
  pyzmq                      25.1.1        py39h1e134f0_1          conda-forge
  qt-main                    5.15.6        ha0e8b26_4              conda-forge
  qtpy                       2.4.0         pyhd8ed1ab_0            conda-forge
  re2                        2022.06.01    h9a09cb3_1              conda-forge
  readline                   8.2           h92ec313_1              conda-forge
  requests                   2.31.0        pyhd8ed1ab_0            conda-forge
  requests-oauthlib          1.3.1         pyhd8ed1ab_0            conda-forge
  rich                       13.5.3        pyhd8ed1ab_0            conda-forge
  rsa                        4.9           pyhd8ed1ab_0            conda-forge
  scikit-image               0.20.0        py39hd28f0be_1          conda-forge
  scikit-learn               1.0           py39h12ba089_1          conda-forge
  scikit-video               1.1.11        pyh24bf2e0_0            conda-forge
  scipy                      1.9.0         py39h14896cb_0          conda-forge
  seaborn                    0.12.2        hd8ed1ab_0              conda-forge
  seaborn-base               0.12.2        pyhd8ed1ab_0            conda-forge
  setuptools                 68.2.2        pyhd8ed1ab_0            conda-forge
  shapely                    2.0.1         py39h3010c4d_1          conda-forge
  six                        1.16.0        pyh6c4a22f_0            conda-forge
  snappy                     1.1.10        h17c5cce_0              conda-forge
  sqlite                     3.43.0        h203b68d_0              conda-forge
  statsmodels                0.14.0        py39h8a366b7_1          conda-forge
  svt-av1                    1.4.1         h7ea286d_0              conda-forge
  tensorboard                2.9.0         pyhd8ed1ab_0            conda-forge
  tensorboard-data-server    0.6.1         py39haa0b8cc_4          conda-forge
  tensorboard-plugin-wit     1.8.1         pyhd8ed1ab_0            conda-forge
  tensorflow                 2.9.1         cpu_py39h2839aeb_0      conda-forge
  tensorflow-base            2.9.1         cpu_py39ha1ad4ae_0      conda-forge
  tensorflow-estimator       2.9.1         cpu_py39h7b621ec_0      conda-forge
  tensorflow-hub             0.12.0        pyhca92ed8_0            conda-forge
  termcolor                  2.3.0         pyhd8ed1ab_0            conda-forge
  threadpoolctl              3.2.0         pyha21a80b_0            conda-forge
  tifffile                   2022.10.10    pyhd8ed1ab_0            conda-forge
  tk                         8.6.13        hb31c410_0              conda-forge
  toolz                      0.12.0        pyhd8ed1ab_0            conda-forge
  typing-extensions          4.8.0         hd8ed1ab_0              conda-forge
  typing_extensions          4.8.0         pyha770c72_0            conda-forge
  tzdata                     2023c         h71feb2d_0              conda-forge
  unicodedata2               15.0.0        py39h0f82c59_1          conda-forge
  urllib3                    1.26.16       pyhd8ed1ab_0            conda-forge
  werkzeug                   2.3.7         pyhd8ed1ab_0            conda-forge
  wheel                      0.41.2        pyhd8ed1ab_0            conda-forge
  wrapt                      1.15.0        py39h0f82c59_1          conda-forge
  x264                       1!164.3095    h57fd34a_2              conda-forge
  x265                       3.5           hbc6ce65_3              conda-forge
  xorg-libxau                1.0.11        hb547adb_0              conda-forge
  xorg-libxdmcp              1.1.3         h27ca646_0              conda-forge
  xz                         5.2.6         h57fd34a_0              conda-forge
  yaml                       0.2.5         h3422bc3_2              conda-forge
  yarl                       1.9.2         py39h0f82c59_0          conda-forge
  zeromq                     4.3.4         hbdafb3b_1              conda-forge
  zfp                        1.0.0         hb6e4faa_3              conda-forge
  zipp                       3.17.0        pyhd8ed1ab_0            conda-forge
  zlib                       1.2.13        h53f4e23_5              conda-forge
  zlib-ng                    2.0.7         h1a8c8d9_0              conda-forge
  zstd                       1.5.5         h4f39d0f_0              conda-forge

pip freeze

absl-py @ file:///home/conda/feedstock_root/build_artifacts/absl-py_1695154477682/work
accessible-pygments==0.0.4
aiohttp @ file:///Users/runner/miniforge3/conda-bld/aiohttp_1689805247820/work
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work
alabaster==0.7.13
aniposelib==0.4.3
anyio==3.7.1
appdirs==1.4.4
appnope==0.1.3
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
asttokens==2.4.0
astunparse @ file:///home/conda/feedstock_root/build_artifacts/astunparse_1610696312422/work
async-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1691763562544/work
attrs==21.4.0
Babel==2.12.1
backcall==0.2.0
beautifulsoup4==4.12.2
black==21.6b0
bleach==6.0.0
blinker @ file:///home/conda/feedstock_root/build_artifacts/blinker_1681349778161/work
Brotli @ file:///Users/runner/miniforge3/conda-bld/brotli-split_1687884292427/work
cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
cachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1688227447901/work
cattrs @ file:///home/conda/feedstock_root/build_artifacts/cattrs_1604136207372/work
certifi==2023.7.22
cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1695367189446/work
cfgv==3.4.0
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1688813409104/work
click==8.0.4
cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1674202310934/work
colorama==0.4.6
comm==0.1.4
contourpy @ file:///Users/runner/miniforge3/conda-bld/contourpy_1695554288975/work
coverage==7.3.1
cryptography @ file:///Users/runner/miniforge3/conda-bld/cryptography-split_1672673285537/work
cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1635519461629/work
cytoolz @ file:///Users/runner/miniforge3/conda-bld/cytoolz_1695545219492/work
dask @ file:///home/conda/feedstock_root/build_artifacts/dask-core_1694811846293/work
debugpy==1.8.0
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.14
distlib==0.3.7
docutils==0.17.1
efficientnet==1.0.0
entrypoints==0.4
exceptiongroup==1.1.3
executing==1.2.0
filelock==3.12.4
flatbuffers @ file:///home/conda/feedstock_root/build_artifacts/python-flatbuffers_1617723079010/work
fonttools @ file:///Users/runner/miniforge3/conda-bld/fonttools_1692542726085/work
frozenlist @ file:///Users/runner/miniforge3/conda-bld/frozenlist_1695377682617/work
fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1695417469930/work
gast @ file:///home/conda/feedstock_root/build_artifacts/gast_1596839682936/work
gitdb==4.0.10
GitPython==3.1.37
google-auth @ file:///home/conda/feedstock_root/build_artifacts/google-auth_1694862673644/work
google-auth-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/google-auth-oauthlib_1630497468950/work
google-pasta==0.2.0
grpcio @ file:///Users/runner/miniforge3/conda-bld/grpc-split_1660663644663/work
h5py @ file:///Users/runner/miniforge3/conda-bld/h5py_1674499003063/work
hdmf==3.9.0
identify==2.5.29
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1663625384323/work
image-classifiers==1.0.0
imagecodecs @ file:///Users/runner/miniforge3/conda-bld/imagecodecs_1668867616820/work
imageio @ file:///home/conda/feedstock_root/build_artifacts/imageio_1686552404404/work
imageio-ffmpeg==0.4.9
imagesize==1.4.1
imgaug @ file:///home/conda/feedstock_root/build_artifacts/imgaug_1640909786103/work
imgstore==0.2.9
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1688754491823/work
importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1695414790617/work
iniconfig==2.0.0
ipykernel==6.25.2
ipython==8.15.0
ipython-genutils==0.2.0
ipywidgets==7.8.1
jaraco.classes==3.3.0
jedi==0.17.2
Jinja2==3.1.2
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1691577114857/work
jsmin @ file:///home/conda/feedstock_root/build_artifacts/jsmin_1642532731678/work
json5==0.9.14
jsonpickle==1.2
jsonschema==4.17.3
jupyter-cache==0.4.3
jupyter-server==1.13.5
jupyter-server-mathjax==0.2.6
jupyter-sphinx==0.3.2
jupyter_client==7.4.9
jupyter_core==5.3.1
jupyterlab==3.3.4
jupyterlab-pygments==0.2.2
jupyterlab-widgets==1.1.7
jupyterlab_server==2.16.3
keras @ file:///home/conda/feedstock_root/build_artifacts/keras_1652925255303/work/keras-2.9.0-py2.py3-none-any.whl
Keras-Applications==1.0.8
Keras-Preprocessing @ file:///home/conda/feedstock_root/build_artifacts/keras-preprocessing_1610713559828/work
keyring==24.2.0
kiwisolver @ file:///Users/runner/miniforge3/conda-bld/kiwisolver_1695380068083/work
lazy_loader @ file:///home/conda/feedstock_root/build_artifacts/lazy_loader_1692295373316/work
libclang==16.0.6
linkify-it-py==2.0.2
livereload==2.6.3
llvmlite==0.41.0
locket @ file:///home/conda/feedstock_root/build_artifacts/locket_1650660393415/work
lxml==4.9.3
Markdown @ file:///home/conda/feedstock_root/build_artifacts/markdown_1690307387991/work
markdown-it-py==1.1.0
MarkupSafe @ file:///Users/runner/miniforge3/conda-bld/markupsafe_1695367493730/work
matplotlib @ file:///Users/runner/miniforge3/conda-bld/matplotlib-suite_1688685044808/work
matplotlib-inline==0.1.6
mdit-py-plugins==0.2.8
mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1639515908913/work
mistune==0.8.4
more-itertools==10.1.0
multidict @ file:///Users/runner/miniforge3/conda-bld/multidict_1672339583149/work
munkres==1.1.4
mypy-extensions==1.0.0
myst-nb==0.13.2
myst-parser==0.15.2
nbclassic==0.5.6
nbclient==0.5.13
nbconvert==6.5.4
nbdime==3.2.1
nbformat==5.1.3
ndx-pose==0.1.1
nest-asyncio==1.5.8
networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1680692919326/work
nh3==0.2.14
nixio==1.5.3
nodeenv==1.8.0
notebook==6.5.6
notebook_shim==0.2.3
numba==0.58.0
numpy @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_ba5c3eaa-be93-4f42-ae24-fa720a3c6627m_sp7nbj/croots/recipe/numpy_and_numpy_base_1652801688716/work
oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1666056362788/work
opencv-contrib-python==4.6.0.66
opencv-python==4.6.0
opt-einsum @ file:///home/conda/feedstock_root/build_artifacts/opt_einsum_1617859230218/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1681337016113/work
pandas @ file:///Users/runner/miniforge3/conda-bld/pandas_1688740575873/work
pandocfilters==1.5.0
parso==0.7.1
partd @ file:///home/conda/feedstock_root/build_artifacts/partd_1695667515973/work
pathspec==0.11.2
patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1665356157073/work
pexpect==4.8.0
pickleshare==0.7.5
Pillow @ file:///Users/runner/miniforge3/conda-bld/pillow_1666920692505/work
pkginfo==1.9.6
platformdirs==3.10.0
pluggy==1.3.0
pre-commit==3.4.0
prometheus-client==0.17.1
prompt-toolkit==3.0.39
protobuf==3.19.6
psutil @ file:///Users/runner/miniforge3/conda-bld/psutil_1695367129654/work
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1 @ file:///home/conda/feedstock_root/build_artifacts/pyasn1_1694615621498/work
pyasn1-modules @ file:///home/conda/feedstock_root/build_artifacts/pyasn1-modules_1695107857548/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
pydata-sphinx-theme==0.13.3
PyGithub==1.59.1
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1691408637400/work
PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1689721553971/work
pykalman==0.9.5
PyNaCl==1.5.0
pynwb==2.5.0
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1685514481738/work
pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1652235407899/work
pyrsistent==0.19.3
PySide2==5.15.8
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
pytest==7.4.2
pytest-cov==3.0.0
pytest-qt==4.2.0
pytest-xvfb==3.0.0
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
python-rapidjson @ file:///Users/runner/miniforge3/conda-bld/python-rapidjson_1695615758828/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1693930252784/work
pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work
PyVirtualDisplay==3.0
PyWavelets @ file:///Users/runner/miniforge3/conda-bld/pywavelets_1673082473214/work
PyYAML @ file:///Users/runner/miniforge3/conda-bld/pyyaml_1695373497981/work
pyzmq==24.0.1
qimage2ndarray==1.10.0
QtPy @ file:///home/conda/feedstock_root/build_artifacts/qtpy_1693347765905/work
readme-renderer==42.0
regex==2023.8.8
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1684774241324/work
requests-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/requests-oauthlib_1643557462909/work
requests-toolbelt==1.0.0
rfc3986==2.0.0
rich @ file:///home/conda/feedstock_root/build_artifacts/rich-split_1694974539951/work/dist
rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1658328885051/work
ruamel.yaml==0.17.32
ruamel.yaml.clib==0.2.7
scikit-image @ file:///Users/runner/miniforge3/conda-bld/scikit-image_1681313573054/work/dist/scikit_image-0.20.0-cp39-cp39-macosx_11_0_arm64.whl
scikit-learn @ file:///Users/runner/miniforge3/conda-bld/scikit-learn_1632611447519/work
scikit-video==1.1.11
scipy==1.9.0
seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1672497695270/work
segmentation-models==1.0.1
Send2Trash==1.8.2
shapely @ file:///Users/runner/miniforge3/conda-bld/shapely_1679082009014/work
shiboken2==5.15.8
six==1.15.0
-e git+https://github.com/talmolab/sleap@6b14bcab93b8c1953913dcfbabd6508f78f77e8a#egg=sleap
sleap-anipose==0.1.7
smmap==5.0.1
sniffio==1.3.0
snowballstemmer==2.2.0
soupsieve==2.5
Sphinx==4.5.0
sphinx-autobuild==2021.3.14
sphinx-book-theme==1.0.1
sphinx-copybutton==0.5.2
sphinx-togglebutton==0.3.2
sphinxcontrib-applehelp==1.0.4
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
SQLAlchemy==1.4.49
stack-data==0.6.2
statsmodels @ file:///Users/runner/miniforge3/conda-bld/statsmodels_1683305774790/work
tensorboard @ file:///home/conda/feedstock_root/build_artifacts/tensorboard_1651519171059/work/tensorboard-2.9.0-py3-none-any.whl
tensorboard-data-server @ file:///Users/runner/miniforge3/conda-bld/tensorboard-data-server_1670043861058/work/tensorboard_data_server-0.6.1-py3-none-macosx_11_0_arm64.whl
tensorboard-plugin-wit @ file:///home/conda/feedstock_root/build_artifacts/tensorboard-plugin-wit_1641458951060/work/tensorboard_plugin_wit-1.8.1-py3-none-any.whl
tensorflow @ file:///Users/ngam/Repos/tensorflow-feedstock/miniforge3/conda-bld/tensorflow-split_1658102176686/work/tensorflow_pkg/tensorflow-2.9.1-cp39-cp39-macosx_11_0_arm64.whl
tensorflow-estimator @ file:///Users/ngam/Repos/tensorflow-feedstock/miniforge3/conda-bld/tensorflow-split_1658102176686/work/tensorflow-estimator/wheel_dir/tensorflow_estimator-2.9.0-py2.py3-none-any.whl
tensorflow-hub @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-hub_1618768305670/work/wheel_dir/tensorflow_hub-0.12.0-py2.py3-none-any.whl
tensorflow-macos==2.9.2
tensorflow-metal==0.5.0
termcolor @ file:///home/conda/feedstock_root/build_artifacts/termcolor_1682317048417/work
terminado==0.17.1
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1689261241048/work
tifffile @ file:///home/conda/feedstock_root/build_artifacts/tifffile_1665588749940/work
tinycss2==1.2.1
toml==0.10.2
tomli==2.0.1
toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1657485559105/work
tornado==6.3.3
tqdm==4.66.1
traitlets==5.10.1
twine==3.3.0
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1695040754690/work
tzdata @ file:///home/conda/feedstock_root/build_artifacts/python-tzdata_1680081134351/work
tzlocal==5.0.1
uc-micro-py==1.0.2
unicodedata2 @ file:///Users/runner/miniforge3/conda-bld/unicodedata2_1695405755547/work
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1695649846578/work
virtualenv==20.24.5
wcwidth==0.2.6
webencodings==0.5.1
websocket-client==1.6.3
Werkzeug @ file:///home/conda/feedstock_root/build_artifacts/werkzeug_1692105915497/work
widgetsnbextension==3.6.6
wrapt @ file:///Users/runner/miniforge3/conda-bld/wrapt_1695409130439/work
yarl @ file:///Users/runner/miniforge3/conda-bld/yarl_1685191803109/work
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1695255097490/work

Logs

GUI shenanigans

(sa0) liezlmaree:~$sleap-label $dsdmc
Saving config: /Users/liezlmaree/.sleap/1.3.3/preferences.yaml
Restoring GUI state...

Software versions:
SLEAP: 1.3.3
TensorFlow: 2.9.2
Numpy: 1.22.3
Python: 3.9.15
OS: macOS-13.5-arm64-arm-64bit

Happy SLEAPing! :)
2023-10-16 06:29:32.212 python[30835:1868817] +[CATransaction synchronize] called within transaction
2023-10-16 06:29:55.283 python[30835:1868817] +[CATransaction synchronize] called within transaction
Restoring GUI state...
2023-10-16 06:30:17.615 python[30835:1868817] +[CATransaction synchronize] called within transaction
Resetting monitor window.
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Start training centroid...
['sleap-train', '/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmp3zbfmn4v/231016_063130_training_job.json', '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp', '--zmq', '--save_viz']
INFO:sleap.nn.training:Versions:
SLEAP: 1.3.3
TensorFlow: 2.9.2
Numpy: 1.22.3
Python: 3.9.15
OS: macOS-13.5-arm64-arm-64bit

centorid set-up

INFO:sleap.nn.training:Training labels file: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp
INFO:sleap.nn.training:Training profile: /var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmp3zbfmn4v/231016_063130_training_job.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmp3zbfmn4v/231016_063130_training_job.json",
    "labels_path": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "base_checkpoint": null,
    "tensorboard": false,
    "save_viz": true,
    "zmq": true,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": null,
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": null,
            "validation_inds": null,
            "test_inds": null,
            "search_path_hints": [],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 0.5,
            "pad_to_stride": null,
            "resize_and_pad_to_target": true,
            "target_height": null,
            "target_width": null
        },
        "instance_cropping": {
            "center_on_part": "thorax",
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 2,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": {
                "anchor_part": "thorax",
                "sigma": 2.5,
                "output_stride": 2,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "centered_instance": null,
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        },
        "base_checkpoint": null
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -180.0,
            "rotation_max_angle": 180.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": true,
            "flip_horizontal": false
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": null,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": null,
        "min_val_batches_per_epoch": 10,
        "epochs": 200,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 20
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "231016_063130.centroid.n=149",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.3.3",
    "filename": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmp3zbfmn4v/231016_063130_training_job.json"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 134 / Validation = 15.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
Metal device set to: Apple M2 Pro

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

2023-10-16 06:31:34.872132: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-10-16 06:31:34.872288: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2023-10-16 06:31:35.156811: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
INFO:sleap.nn.training:Loaded test example. [0.820s]
INFO:sleap.nn.training:  Input shape: (512, 512, 3)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=16, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=3, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 16
INFO:sleap.nn.training:  Parameters: 1,953,393
INFO:sleap.nn.training:  Heads: 
INFO:sleap.nn.training:    [0] = CentroidConfmapsHead(anchor_part='thorax', sigma=2.5, output_stride=2, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs: 
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 256, 256, 1), dtype=tf.float32, name=None), name='CentroidConfmapsHead/BiasAdd:0', description="created by layer 'CentroidConfmapsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 134
INFO:sleap.nn.training:Validation set: n = 15
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=20)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
INFO:sleap.nn.training:  ZMQ controller subcribed to: tcp://127.0.0.1:9000
INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set
INFO:sleap.nn.training:  ZMQ progress reporter publish on: tcp://127.0.0.1:9001
INFO:sleap.nn.training:Created run path: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149
INFO:sleap.nn.training:Setting up visualization...
2023-10-16 06:31:37.027020: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-10-16 06:31:37.064815: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-10-16 06:31:37.084587: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -34 } dim { size: -35 } dim { size: -36 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -37 } dim { size: -38 } dim { size: 1 } } }
2023-10-16 06:31:37.683964: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-10-16 06:31:37.721064: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-10-16 06:31:37.739780: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -34 } dim { size: -35 } dim { size: -36 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -37 } dim { size: -38 } dim { size: 1 } } }
INFO:sleap.nn.training:Finished trainer set up. [2.9s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [4.2s]
INFO:sleap.nn.training:Starting training loop...

centroid epochs

Epoch 1/200
2023-10-16 06:31:42.576707: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-10-16 06:32:18.788127: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
200/200 - 39s - loss: 1.8696e-04 - val_loss: 7.9245e-05 - lr: 1.0000e-04 - 39s/epoch - 194ms/step
Epoch 2/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 5.5605e-05 - val_loss: 4.3189e-05 - lr: 1.0000e-04 - 36s/epoch - 182ms/step
Epoch 3/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 3.8371e-05 - val_loss: 3.1048e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step
Epoch 4/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 3.3563e-05 - val_loss: 2.5386e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step
Epoch 5/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 3.2097e-05 - val_loss: 2.8539e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step
Epoch 6/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.9414e-05 - val_loss: 2.6226e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step
Epoch 7/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.8567e-05 - val_loss: 2.5540e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step
Epoch 8/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.7141e-05 - val_loss: 2.2127e-05 - lr: 1.0000e-04 - 36s/epoch - 180ms/step
Epoch 9/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.6605e-05 - val_loss: 2.4407e-05 - lr: 1.0000e-04 - 36s/epoch - 180ms/step
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Epoch 10/200
200/200 - 36s - loss: 2.5723e-05 - val_loss: 2.1715e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step
Epoch 11/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.4948e-05 - val_loss: 2.0652e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step
Epoch 12/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.3232e-05 - val_loss: 1.8315e-05 - lr: 1.0000e-04 - 36s/epoch - 182ms/step
Epoch 13/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.4580e-05 - val_loss: 2.1906e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step
Epoch 14/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.3585e-05 - val_loss: 2.5078e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Epoch 15/200
200/200 - 36s - loss: 2.3469e-05 - val_loss: 2.0460e-05 - lr: 1.0000e-04 - 36s/epoch - 180ms/step
Epoch 16/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.3363e-05 - val_loss: 1.9228e-05 - lr: 1.0000e-04 - 36s/epoch - 182ms/step
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Epoch 17/200

Epoch 17: ReduceLROnPlateau reducing learning rate to 4.999999873689376e-05.
200/200 - 36s - loss: 2.2882e-05 - val_loss: 1.8767e-05 - lr: 1.0000e-04 - 36s/epoch - 182ms/step
Epoch 18/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.0425e-05 - val_loss: 1.7738e-05 - lr: 5.0000e-05 - 36s/epoch - 182ms/step
Epoch 19/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 1.9551e-05 - val_loss: 1.8845e-05 - lr: 5.0000e-05 - 36s/epoch - 180ms/step
Epoch 20/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 2.0076e-05 - val_loss: 1.7287e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step
Epoch 21/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 1.9783e-05 - val_loss: 1.7333e-05 - lr: 5.0000e-05 - 36s/epoch - 180ms/step
Epoch 22/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 1.9134e-05 - val_loss: 1.5902e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step
Epoch 23/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 1.9437e-05 - val_loss: 1.9749e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step
Epoch 24/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 1.8544e-05 - val_loss: 1.6604e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step
Epoch 25/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 1.9098e-05 - val_loss: 1.7043e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step
Epoch 26/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 1.8231e-05 - val_loss: 1.6614e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step
Epoch 27/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png

Epoch 27: ReduceLROnPlateau reducing learning rate to 2.499999936844688e-05.
200/200 - 36s - loss: 1.8520e-05 - val_loss: 1.5647e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step
Epoch 28/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 1.7531e-05 - val_loss: 1.7952e-05 - lr: 2.5000e-05 - 36s/epoch - 182ms/step
Epoch 29/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
200/200 - 36s - loss: 1.7436e-05 - val_loss: 1.5768e-05 - lr: 2.5000e-05 - 36s/epoch - 182ms/step
Epoch 30/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png

centered-instance set-up

           "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": true,
            "flip_horizontal": false
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": null,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": null,
        "min_val_batches_per_epoch": 10,
        "epochs": 200,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 10
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "231016_070843.centered_instance.n=149",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.3.3",
    "filename": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpuot85hlo/231016_070843_training_job.json"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 134 / Validation = 15.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
Metal device set to: Apple M2 Pro

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

2023-10-16 07:08:48.695045: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-10-16 07:08:48.695183: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2023-10-16 07:08:48.976602: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2023-10-16 07:08:49.894760: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
INFO:sleap.nn.training:Loaded test example. [1.276s]
INFO:sleap.nn.training:  Input shape: (144, 144, 3)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=24, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=2, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 16
INFO:sleap.nn.training:  Parameters: 4,311,877
INFO:sleap.nn.training:  Heads: 
INFO:sleap.nn.training:    [0] = CenteredInstanceConfmapsHead(part_names=['head', 'thorax', 'abdomen', 'wingL', 'wingR', 'forelegL4', 'forelegR4', 'midlegL4', 'midlegR4', 'hindlegL4', 'hindlegR4', 'eyeL', 'eyeR'], anchor_part='thorax', sigma=2.5, output_stride=4, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs: 
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 36, 36, 13), dtype=tf.float32, name=None), name='CenteredInstanceConfmapsHead/BiasAdd:0', description="created by layer 'CenteredInstanceConfmapsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 134
INFO:sleap.nn.training:Validation set: n = 15
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
INFO:sleap.nn.training:  ZMQ controller subcribed to: tcp://127.0.0.1:9000
INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set
INFO:sleap.nn.training:  ZMQ progress reporter publish on: tcp://127.0.0.1:9001
INFO:sleap.nn.training:Created run path: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149
INFO:sleap.nn.training:Setting up visualization...
2023-10-16 07:08:50.792925: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
2023-10-16 07:08:51.415832: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
INFO:sleap.nn.training:Finished trainer set up. [2.8s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
2023-10-16 07:08:54.762347: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
2023-10-16 07:08:56.073253: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
INFO:sleap.nn.training:Finished creating training datasets. [4.8s]
INFO:sleap.nn.training:Starting training loop...

centered-instance epochs

2023-10-16 07:08:56.464164: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
Epoch 1/200
2023-10-16 07:08:57.322132: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-10-16 07:09:09.114265: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
2023-10-16 07:09:09.463435: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-10-16 07:09:10.410374: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-10-16 07:09:10.420501: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 13 } dim { size: 36 } dim { size: 36 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -5 } dim { size: -6 } dim { size: 1 } } }
200/200 - 14s - loss: 0.0109 - head: 0.0070 - thorax: 0.0062 - abdomen: 0.0112 - wingL: 0.0125 - wingR: 0.0127 - forelegL4: 0.0112 - forelegR4: 0.0109 - midlegL4: 0.0131 - midlegR4: 0.0132 - hindlegL4: 0.0131 - hindlegR4: 0.0130 - eyeL: 0.0091 - eyeR: 0.0090 - val_loss: 0.0090 - val_head: 0.0033 - val_thorax: 0.0036 - val_abdomen: 0.0076 - val_wingL: 0.0086 - val_wingR: 0.0093 - val_forelegL4: 0.0101 - val_forelegR4: 0.0106 - val_midlegL4: 0.0122 - val_midlegR4: 0.0125 - val_hindlegL4: 0.0128 - val_hindlegR4: 0.0118 - val_eyeL: 0.0070 - val_eyeR: 0.0074 - lr: 1.0000e-04 - 14s/epoch - 72ms/step
Epoch 2/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:09:22.245987: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0081 - head: 0.0020 - thorax: 0.0028 - abdomen: 0.0061 - wingL: 0.0069 - wingR: 0.0078 - forelegL4: 0.0097 - forelegR4: 0.0093 - midlegL4: 0.0119 - midlegR4: 0.0122 - hindlegL4: 0.0121 - hindlegR4: 0.0117 - eyeL: 0.0068 - eyeR: 0.0066 - val_loss: 0.0075 - val_head: 0.0013 - val_thorax: 0.0026 - val_abdomen: 0.0042 - val_wingL: 0.0056 - val_wingR: 0.0059 - val_forelegL4: 0.0086 - val_forelegR4: 0.0100 - val_midlegL4: 0.0117 - val_midlegR4: 0.0123 - val_hindlegL4: 0.0114 - val_hindlegR4: 0.0114 - val_eyeL: 0.0062 - val_eyeR: 0.0062 - lr: 1.0000e-04 - 12s/epoch - 60ms/step
Epoch 3/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:09:34.223959: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0073 - head: 0.0011 - thorax: 0.0019 - abdomen: 0.0039 - wingL: 0.0048 - wingR: 0.0061 - forelegL4: 0.0094 - forelegR4: 0.0090 - midlegL4: 0.0116 - midlegR4: 0.0117 - hindlegL4: 0.0113 - hindlegR4: 0.0114 - eyeL: 0.0063 - eyeR: 0.0061 - val_loss: 0.0073 - val_head: 8.8589e-04 - val_thorax: 0.0018 - val_abdomen: 0.0035 - val_wingL: 0.0046 - val_wingR: 0.0049 - val_forelegL4: 0.0097 - val_forelegR4: 0.0097 - val_midlegL4: 0.0117 - val_midlegR4: 0.0121 - val_hindlegL4: 0.0118 - val_hindlegR4: 0.0115 - val_eyeL: 0.0060 - val_eyeR: 0.0066 - lr: 1.0000e-04 - 12s/epoch - 60ms/step
Epoch 4/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:09:46.088737: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0069 - head: 7.7567e-04 - thorax: 0.0014 - abdomen: 0.0031 - wingL: 0.0042 - wingR: 0.0053 - forelegL4: 0.0093 - forelegR4: 0.0088 - midlegL4: 0.0113 - midlegR4: 0.0114 - hindlegL4: 0.0111 - hindlegR4: 0.0110 - eyeL: 0.0061 - eyeR: 0.0060 - val_loss: 0.0071 - val_head: 5.7390e-04 - val_thorax: 0.0015 - val_abdomen: 0.0035 - val_wingL: 0.0041 - val_wingR: 0.0043 - val_forelegL4: 0.0094 - val_forelegR4: 0.0099 - val_midlegL4: 0.0114 - val_midlegR4: 0.0120 - val_hindlegL4: 0.0116 - val_hindlegR4: 0.0110 - val_eyeL: 0.0061 - val_eyeR: 0.0064 - lr: 1.0000e-04 - 12s/epoch - 59ms/step
Epoch 5/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:09:58.092878: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0067 - head: 7.3147e-04 - thorax: 0.0011 - abdomen: 0.0027 - wingL: 0.0038 - wingR: 0.0050 - forelegL4: 0.0091 - forelegR4: 0.0086 - midlegL4: 0.0109 - midlegR4: 0.0109 - hindlegL4: 0.0110 - hindlegR4: 0.0107 - eyeL: 0.0060 - eyeR: 0.0059 - val_loss: 0.0070 - val_head: 7.0433e-04 - val_thorax: 0.0011 - val_abdomen: 0.0031 - val_wingL: 0.0046 - val_wingR: 0.0048 - val_forelegL4: 0.0090 - val_forelegR4: 0.0099 - val_midlegL4: 0.0113 - val_midlegR4: 0.0120 - val_hindlegL4: 0.0118 - val_hindlegR4: 0.0111 - val_eyeL: 0.0060 - val_eyeR: 0.0063 - lr: 1.0000e-04 - 12s/epoch - 59ms/step
Epoch 6/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:10:09.993887: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0064 - head: 6.7371e-04 - thorax: 9.1523e-04 - abdomen: 0.0024 - wingL: 0.0036 - wingR: 0.0048 - forelegL4: 0.0089 - forelegR4: 0.0084 - midlegL4: 0.0106 - midlegR4: 0.0105 - hindlegL4: 0.0107 - hindlegR4: 0.0104 - eyeL: 0.0060 - eyeR: 0.0058 - val_loss: 0.0066 - val_head: 6.7408e-04 - val_thorax: 0.0012 - val_abdomen: 0.0027 - val_wingL: 0.0041 - val_wingR: 0.0042 - val_forelegL4: 0.0090 - val_forelegR4: 0.0093 - val_midlegL4: 0.0102 - val_midlegR4: 0.0115 - val_hindlegL4: 0.0108 - val_hindlegR4: 0.0108 - val_eyeL: 0.0057 - val_eyeR: 0.0062 - lr: 1.0000e-04 - 12s/epoch - 60ms/step
Epoch 7/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:10:22.083929: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0063 - head: 5.7252e-04 - thorax: 8.0404e-04 - abdomen: 0.0024 - wingL: 0.0035 - wingR: 0.0046 - forelegL4: 0.0087 - forelegR4: 0.0083 - midlegL4: 0.0103 - midlegR4: 0.0101 - hindlegL4: 0.0106 - hindlegR4: 0.0100 - eyeL: 0.0059 - eyeR: 0.0058 - val_loss: 0.0066 - val_head: 7.2597e-04 - val_thorax: 0.0011 - val_abdomen: 0.0031 - val_wingL: 0.0047 - val_wingR: 0.0047 - val_forelegL4: 0.0091 - val_forelegR4: 0.0086 - val_midlegL4: 0.0098 - val_midlegR4: 0.0112 - val_hindlegL4: 0.0107 - val_hindlegR4: 0.0102 - val_eyeL: 0.0057 - val_eyeR: 0.0059 - lr: 1.0000e-04 - 12s/epoch - 60ms/step
Epoch 8/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:10:34.155638: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0061 - head: 5.7429e-04 - thorax: 7.5300e-04 - abdomen: 0.0022 - wingL: 0.0032 - wingR: 0.0046 - forelegL4: 0.0084 - forelegR4: 0.0081 - midlegL4: 0.0099 - midlegR4: 0.0098 - hindlegL4: 0.0104 - hindlegR4: 0.0096 - eyeL: 0.0058 - eyeR: 0.0058 - val_loss: 0.0063 - val_head: 5.3473e-04 - val_thorax: 7.9216e-04 - val_abdomen: 0.0021 - val_wingL: 0.0038 - val_wingR: 0.0040 - val_forelegL4: 0.0089 - val_forelegR4: 0.0083 - val_midlegL4: 0.0103 - val_midlegR4: 0.0104 - val_hindlegL4: 0.0109 - val_hindlegR4: 0.0102 - val_eyeL: 0.0057 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 60ms/step
Epoch 9/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:10:46.228547: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0060 - head: 5.5613e-04 - thorax: 7.6087e-04 - abdomen: 0.0020 - wingL: 0.0033 - wingR: 0.0045 - forelegL4: 0.0082 - forelegR4: 0.0079 - midlegL4: 0.0096 - midlegR4: 0.0096 - hindlegL4: 0.0102 - hindlegR4: 0.0095 - eyeL: 0.0058 - eyeR: 0.0058 - val_loss: 0.0061 - val_head: 3.8938e-04 - val_thorax: 7.1305e-04 - val_abdomen: 0.0016 - val_wingL: 0.0046 - val_wingR: 0.0040 - val_forelegL4: 0.0081 - val_forelegR4: 0.0085 - val_midlegL4: 0.0095 - val_midlegR4: 0.0103 - val_hindlegL4: 0.0103 - val_hindlegR4: 0.0100 - val_eyeL: 0.0057 - val_eyeR: 0.0060 - lr: 1.0000e-04 - 12s/epoch - 60ms/step
Epoch 10/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:10:58.254701: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0058 - head: 4.8496e-04 - thorax: 6.5602e-04 - abdomen: 0.0018 - wingL: 0.0031 - wingR: 0.0043 - forelegL4: 0.0080 - forelegR4: 0.0078 - midlegL4: 0.0095 - midlegR4: 0.0094 - hindlegL4: 0.0100 - hindlegR4: 0.0091 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0061 - val_head: 4.1555e-04 - val_thorax: 6.8679e-04 - val_abdomen: 0.0019 - val_wingL: 0.0046 - val_wingR: 0.0040 - val_forelegL4: 0.0081 - val_forelegR4: 0.0084 - val_midlegL4: 0.0091 - val_midlegR4: 0.0107 - val_hindlegL4: 0.0099 - val_hindlegR4: 0.0099 - val_eyeL: 0.0053 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 60ms/step
Epoch 11/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:11:10.393284: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0057 - head: 4.7110e-04 - thorax: 6.2252e-04 - abdomen: 0.0017 - wingL: 0.0031 - wingR: 0.0044 - forelegL4: 0.0079 - forelegR4: 0.0076 - midlegL4: 0.0093 - midlegR4: 0.0093 - hindlegL4: 0.0098 - hindlegR4: 0.0089 - eyeL: 0.0058 - eyeR: 0.0057 - val_loss: 0.0061 - val_head: 4.9512e-04 - val_thorax: 9.8132e-04 - val_abdomen: 0.0019 - val_wingL: 0.0041 - val_wingR: 0.0035 - val_forelegL4: 0.0082 - val_forelegR4: 0.0083 - val_midlegL4: 0.0098 - val_midlegR4: 0.0099 - val_hindlegL4: 0.0103 - val_hindlegR4: 0.0101 - val_eyeL: 0.0060 - val_eyeR: 0.0063 - lr: 1.0000e-04 - 12s/epoch - 60ms/step
Epoch 12/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:11:22.584388: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0056 - head: 4.3117e-04 - thorax: 5.9805e-04 - abdomen: 0.0016 - wingL: 0.0029 - wingR: 0.0043 - forelegL4: 0.0077 - forelegR4: 0.0075 - midlegL4: 0.0092 - midlegR4: 0.0091 - hindlegL4: 0.0095 - hindlegR4: 0.0087 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0061 - val_head: 4.1128e-04 - val_thorax: 6.8850e-04 - val_abdomen: 0.0019 - val_wingL: 0.0042 - val_wingR: 0.0045 - val_forelegL4: 0.0081 - val_forelegR4: 0.0082 - val_midlegL4: 0.0096 - val_midlegR4: 0.0101 - val_hindlegL4: 0.0101 - val_hindlegR4: 0.0096 - val_eyeL: 0.0056 - val_eyeR: 0.0063 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 13/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:11:34.902982: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0056 - head: 4.3588e-04 - thorax: 5.8300e-04 - abdomen: 0.0015 - wingL: 0.0029 - wingR: 0.0042 - forelegL4: 0.0077 - forelegR4: 0.0075 - midlegL4: 0.0090 - midlegR4: 0.0091 - hindlegL4: 0.0094 - hindlegR4: 0.0085 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0059 - val_head: 4.6890e-04 - val_thorax: 6.5690e-04 - val_abdomen: 0.0015 - val_wingL: 0.0040 - val_wingR: 0.0042 - val_forelegL4: 0.0080 - val_forelegR4: 0.0080 - val_midlegL4: 0.0089 - val_midlegR4: 0.0100 - val_hindlegL4: 0.0104 - val_hindlegR4: 0.0095 - val_eyeL: 0.0056 - val_eyeR: 0.0060 - lr: 1.0000e-04 - 12s/epoch - 62ms/step
Epoch 14/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:11:47.214435: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0055 - head: 4.3798e-04 - thorax: 5.6456e-04 - abdomen: 0.0014 - wingL: 0.0028 - wingR: 0.0041 - forelegL4: 0.0076 - forelegR4: 0.0074 - midlegL4: 0.0089 - midlegR4: 0.0090 - hindlegL4: 0.0093 - hindlegR4: 0.0084 - eyeL: 0.0058 - eyeR: 0.0057 - val_loss: 0.0059 - val_head: 4.2385e-04 - val_thorax: 5.2439e-04 - val_abdomen: 0.0014 - val_wingL: 0.0033 - val_wingR: 0.0037 - val_forelegL4: 0.0087 - val_forelegR4: 0.0078 - val_midlegL4: 0.0095 - val_midlegR4: 0.0099 - val_hindlegL4: 0.0104 - val_hindlegR4: 0.0095 - val_eyeL: 0.0059 - val_eyeR: 0.0059 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 15/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:11:59.501349: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0054 - head: 4.2132e-04 - thorax: 5.4735e-04 - abdomen: 0.0015 - wingL: 0.0028 - wingR: 0.0041 - forelegL4: 0.0075 - forelegR4: 0.0073 - midlegL4: 0.0088 - midlegR4: 0.0089 - hindlegL4: 0.0091 - hindlegR4: 0.0083 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0059 - val_head: 4.4959e-04 - val_thorax: 8.1589e-04 - val_abdomen: 0.0019 - val_wingL: 0.0039 - val_wingR: 0.0038 - val_forelegL4: 0.0084 - val_forelegR4: 0.0076 - val_midlegL4: 0.0092 - val_midlegR4: 0.0092 - val_hindlegL4: 0.0100 - val_hindlegR4: 0.0092 - val_eyeL: 0.0056 - val_eyeR: 0.0060 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 16/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:12:11.712830: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0054 - head: 4.2216e-04 - thorax: 5.4495e-04 - abdomen: 0.0014 - wingL: 0.0028 - wingR: 0.0040 - forelegL4: 0.0075 - forelegR4: 0.0072 - midlegL4: 0.0088 - midlegR4: 0.0087 - hindlegL4: 0.0088 - hindlegR4: 0.0081 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0060 - val_head: 4.4052e-04 - val_thorax: 6.4149e-04 - val_abdomen: 0.0017 - val_wingL: 0.0049 - val_wingR: 0.0037 - val_forelegL4: 0.0079 - val_forelegR4: 0.0083 - val_midlegL4: 0.0096 - val_midlegR4: 0.0100 - val_hindlegL4: 0.0097 - val_hindlegR4: 0.0089 - val_eyeL: 0.0057 - val_eyeR: 0.0062 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 17/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:12:23.892308: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0053 - head: 3.9981e-04 - thorax: 5.1178e-04 - abdomen: 0.0013 - wingL: 0.0026 - wingR: 0.0038 - forelegL4: 0.0074 - forelegR4: 0.0072 - midlegL4: 0.0086 - midlegR4: 0.0087 - hindlegL4: 0.0087 - hindlegR4: 0.0080 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0057 - val_head: 4.6381e-04 - val_thorax: 5.2619e-04 - val_abdomen: 0.0014 - val_wingL: 0.0043 - val_wingR: 0.0037 - val_forelegL4: 0.0073 - val_forelegR4: 0.0080 - val_midlegL4: 0.0087 - val_midlegR4: 0.0097 - val_hindlegL4: 0.0093 - val_hindlegR4: 0.0094 - val_eyeL: 0.0058 - val_eyeR: 0.0058 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 18/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:12:36.195365: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0052 - head: 4.1838e-04 - thorax: 5.1378e-04 - abdomen: 0.0014 - wingL: 0.0026 - wingR: 0.0037 - forelegL4: 0.0073 - forelegR4: 0.0072 - midlegL4: 0.0086 - midlegR4: 0.0086 - hindlegL4: 0.0086 - hindlegR4: 0.0078 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0058 - val_head: 4.9995e-04 - val_thorax: 6.1695e-04 - val_abdomen: 0.0015 - val_wingL: 0.0039 - val_wingR: 0.0037 - val_forelegL4: 0.0078 - val_forelegR4: 0.0084 - val_midlegL4: 0.0081 - val_midlegR4: 0.0104 - val_hindlegL4: 0.0094 - val_hindlegR4: 0.0088 - val_eyeL: 0.0056 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 19/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:12:48.376745: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0051 - head: 4.2751e-04 - thorax: 5.1252e-04 - abdomen: 0.0013 - wingL: 0.0024 - wingR: 0.0035 - forelegL4: 0.0073 - forelegR4: 0.0071 - midlegL4: 0.0085 - midlegR4: 0.0085 - hindlegL4: 0.0083 - hindlegR4: 0.0075 - eyeL: 0.0057 - eyeR: 0.0056 - val_loss: 0.0058 - val_head: 5.3449e-04 - val_thorax: 4.2500e-04 - val_abdomen: 0.0015 - val_wingL: 0.0039 - val_wingR: 0.0036 - val_forelegL4: 0.0075 - val_forelegR4: 0.0080 - val_midlegL4: 0.0091 - val_midlegR4: 0.0100 - val_hindlegL4: 0.0090 - val_hindlegR4: 0.0099 - val_eyeL: 0.0057 - val_eyeR: 0.0062 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 20/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:13:00.626163: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0051 - head: 4.1717e-04 - thorax: 4.6850e-04 - abdomen: 0.0013 - wingL: 0.0025 - wingR: 0.0034 - forelegL4: 0.0072 - forelegR4: 0.0071 - midlegL4: 0.0083 - midlegR4: 0.0085 - hindlegL4: 0.0081 - hindlegR4: 0.0076 - eyeL: 0.0056 - eyeR: 0.0057 - val_loss: 0.0055 - val_head: 6.7695e-04 - val_thorax: 5.0953e-04 - val_abdomen: 0.0014 - val_wingL: 0.0038 - val_wingR: 0.0034 - val_forelegL4: 0.0077 - val_forelegR4: 0.0078 - val_midlegL4: 0.0081 - val_midlegR4: 0.0093 - val_hindlegL4: 0.0091 - val_hindlegR4: 0.0081 - val_eyeL: 0.0056 - val_eyeR: 0.0059 - lr: 1.0000e-04 - 12s/epoch - 62ms/step
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
Epoch 21/200
2023-10-16 07:13:12.832365: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0050 - head: 4.2067e-04 - thorax: 4.5395e-04 - abdomen: 0.0012 - wingL: 0.0023 - wingR: 0.0034 - forelegL4: 0.0072 - forelegR4: 0.0071 - midlegL4: 0.0081 - midlegR4: 0.0084 - hindlegL4: 0.0079 - hindlegR4: 0.0073 - eyeL: 0.0056 - eyeR: 0.0056 - val_loss: 0.0056 - val_head: 5.4406e-04 - val_thorax: 6.7325e-04 - val_abdomen: 0.0018 - val_wingL: 0.0036 - val_wingR: 0.0031 - val_forelegL4: 0.0078 - val_forelegR4: 0.0075 - val_midlegL4: 0.0090 - val_midlegR4: 0.0091 - val_hindlegL4: 0.0090 - val_hindlegR4: 0.0089 - val_eyeL: 0.0056 - val_eyeR: 0.0058 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 22/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:13:24.987091: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0050 - head: 4.2020e-04 - thorax: 4.7335e-04 - abdomen: 0.0013 - wingL: 0.0022 - wingR: 0.0032 - forelegL4: 0.0072 - forelegR4: 0.0070 - midlegL4: 0.0082 - midlegR4: 0.0083 - hindlegL4: 0.0077 - hindlegR4: 0.0072 - eyeL: 0.0057 - eyeR: 0.0056 - val_loss: 0.0058 - val_head: 5.7998e-04 - val_thorax: 4.9280e-04 - val_abdomen: 0.0016 - val_wingL: 0.0036 - val_wingR: 0.0030 - val_forelegL4: 0.0080 - val_forelegR4: 0.0082 - val_midlegL4: 0.0085 - val_midlegR4: 0.0097 - val_hindlegL4: 0.0100 - val_hindlegR4: 0.0097 - val_eyeL: 0.0058 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
Epoch 23/200
2023-10-16 07:13:37.162642: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0049 - head: 4.1866e-04 - thorax: 4.2666e-04 - abdomen: 0.0013 - wingL: 0.0022 - wingR: 0.0032 - forelegL4: 0.0071 - forelegR4: 0.0069 - midlegL4: 0.0080 - midlegR4: 0.0081 - hindlegL4: 0.0077 - hindlegR4: 0.0072 - eyeL: 0.0055 - eyeR: 0.0056 - val_loss: 0.0056 - val_head: 4.9101e-04 - val_thorax: 4.3201e-04 - val_abdomen: 0.0014 - val_wingL: 0.0037 - val_wingR: 0.0032 - val_forelegL4: 0.0077 - val_forelegR4: 0.0074 - val_midlegL4: 0.0090 - val_midlegR4: 0.0090 - val_hindlegL4: 0.0090 - val_hindlegR4: 0.0088 - val_eyeL: 0.0060 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 24/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:13:49.387795: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0048 - head: 4.1083e-04 - thorax: 4.0826e-04 - abdomen: 0.0012 - wingL: 0.0019 - wingR: 0.0030 - forelegL4: 0.0071 - forelegR4: 0.0070 - midlegL4: 0.0080 - midlegR4: 0.0080 - hindlegL4: 0.0073 - hindlegR4: 0.0069 - eyeL: 0.0056 - eyeR: 0.0056 - val_loss: 0.0053 - val_head: 5.3566e-04 - val_thorax: 4.5736e-04 - val_abdomen: 0.0013 - val_wingL: 0.0035 - val_wingR: 0.0033 - val_forelegL4: 0.0073 - val_forelegR4: 0.0079 - val_midlegL4: 0.0083 - val_midlegR4: 0.0087 - val_hindlegL4: 0.0086 - val_hindlegR4: 0.0081 - val_eyeL: 0.0053 - val_eyeR: 0.0060 - lr: 1.0000e-04 - 12s/epoch - 62ms/step
Epoch 25/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:14:01.756489: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0047 - head: 4.3522e-04 - thorax: 4.3224e-04 - abdomen: 0.0012 - wingL: 0.0021 - wingR: 0.0030 - forelegL4: 0.0070 - forelegR4: 0.0069 - midlegL4: 0.0077 - midlegR4: 0.0078 - hindlegL4: 0.0072 - hindlegR4: 0.0068 - eyeL: 0.0055 - eyeR: 0.0055 - val_loss: 0.0052 - val_head: 5.3260e-04 - val_thorax: 4.2599e-04 - val_abdomen: 0.0011 - val_wingL: 0.0033 - val_wingR: 0.0030 - val_forelegL4: 0.0078 - val_forelegR4: 0.0075 - val_midlegL4: 0.0088 - val_midlegR4: 0.0091 - val_hindlegL4: 0.0073 - val_hindlegR4: 0.0075 - val_eyeL: 0.0055 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 62ms/step
Epoch 26/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:14:14.094093: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0046 - head: 4.5979e-04 - thorax: 4.6782e-04 - abdomen: 0.0012 - wingL: 0.0019 - wingR: 0.0028 - forelegL4: 0.0070 - forelegR4: 0.0067 - midlegL4: 0.0074 - midlegR4: 0.0077 - hindlegL4: 0.0069 - hindlegR4: 0.0064 - eyeL: 0.0054 - eyeR: 0.0055 - val_loss: 0.0053 - val_head: 6.2012e-04 - val_thorax: 5.0883e-04 - val_abdomen: 0.0013 - val_wingL: 0.0038 - val_wingR: 0.0033 - val_forelegL4: 0.0074 - val_forelegR4: 0.0073 - val_midlegL4: 0.0088 - val_midlegR4: 0.0094 - val_hindlegL4: 0.0081 - val_hindlegR4: 0.0073 - val_eyeL: 0.0057 - val_eyeR: 0.0057 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 27/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:14:26.352288: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0046 - head: 4.1399e-04 - thorax: 4.1495e-04 - abdomen: 0.0012 - wingL: 0.0020 - wingR: 0.0028 - forelegL4: 0.0069 - forelegR4: 0.0067 - midlegL4: 0.0074 - midlegR4: 0.0075 - hindlegL4: 0.0068 - hindlegR4: 0.0064 - eyeL: 0.0053 - eyeR: 0.0054 - val_loss: 0.0056 - val_head: 4.5388e-04 - val_thorax: 4.2171e-04 - val_abdomen: 0.0015 - val_wingL: 0.0032 - val_wingR: 0.0035 - val_forelegL4: 0.0079 - val_forelegR4: 0.0072 - val_midlegL4: 0.0095 - val_midlegR4: 0.0100 - val_hindlegL4: 0.0094 - val_hindlegR4: 0.0082 - val_eyeL: 0.0058 - val_eyeR: 0.0062 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 28/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:14:38.578576: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0046 - head: 4.3010e-04 - thorax: 4.7511e-04 - abdomen: 0.0012 - wingL: 0.0019 - wingR: 0.0027 - forelegL4: 0.0069 - forelegR4: 0.0068 - midlegL4: 0.0074 - midlegR4: 0.0074 - hindlegL4: 0.0069 - hindlegR4: 0.0065 - eyeL: 0.0053 - eyeR: 0.0054 - val_loss: 0.0051 - val_head: 5.2094e-04 - val_thorax: 3.8217e-04 - val_abdomen: 8.2952e-04 - val_wingL: 0.0036 - val_wingR: 0.0027 - val_forelegL4: 0.0078 - val_forelegR4: 0.0071 - val_midlegL4: 0.0081 - val_midlegR4: 0.0080 - val_hindlegL4: 0.0084 - val_hindlegR4: 0.0083 - val_eyeL: 0.0053 - val_eyeR: 0.0056 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 29/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:14:50.993716: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0044 - head: 4.1449e-04 - thorax: 4.1248e-04 - abdomen: 0.0011 - wingL: 0.0018 - wingR: 0.0025 - forelegL4: 0.0069 - forelegR4: 0.0067 - midlegL4: 0.0069 - midlegR4: 0.0070 - hindlegL4: 0.0065 - hindlegR4: 0.0061 - eyeL: 0.0052 - eyeR: 0.0052 - val_loss: 0.0053 - val_head: 5.5684e-04 - val_thorax: 4.1731e-04 - val_abdomen: 0.0016 - val_wingL: 0.0038 - val_wingR: 0.0029 - val_forelegL4: 0.0068 - val_forelegR4: 0.0078 - val_midlegL4: 0.0085 - val_midlegR4: 0.0088 - val_hindlegL4: 0.0084 - val_hindlegR4: 0.0087 - val_eyeL: 0.0052 - val_eyeR: 0.0055 - lr: 1.0000e-04 - 12s/epoch - 62ms/step
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
Epoch 30/200
2023-10-16 07:15:03.164209: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0043 - head: 3.9878e-04 - thorax: 3.9097e-04 - abdomen: 0.0011 - wingL: 0.0019 - wingR: 0.0025 - forelegL4: 0.0068 - forelegR4: 0.0066 - midlegL4: 0.0067 - midlegR4: 0.0068 - hindlegL4: 0.0063 - hindlegR4: 0.0060 - eyeL: 0.0051 - eyeR: 0.0051 - val_loss: 0.0053 - val_head: 4.8584e-04 - val_thorax: 4.9301e-04 - val_abdomen: 0.0013 - val_wingL: 0.0040 - val_wingR: 0.0026 - val_forelegL4: 0.0072 - val_forelegR4: 0.0081 - val_midlegL4: 0.0090 - val_midlegR4: 0.0090 - val_hindlegL4: 0.0078 - val_hindlegR4: 0.0072 - val_eyeL: 0.0054 - val_eyeR: 0.0060 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 31/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:15:15.299034: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0043 - head: 4.0585e-04 - thorax: 3.9195e-04 - abdomen: 0.0011 - wingL: 0.0018 - wingR: 0.0026 - forelegL4: 0.0066 - forelegR4: 0.0065 - midlegL4: 0.0067 - midlegR4: 0.0068 - hindlegL4: 0.0063 - hindlegR4: 0.0061 - eyeL: 0.0050 - eyeR: 0.0050 - val_loss: 0.0051 - val_head: 4.4100e-04 - val_thorax: 4.1315e-04 - val_abdomen: 0.0011 - val_wingL: 0.0034 - val_wingR: 0.0031 - val_forelegL4: 0.0079 - val_forelegR4: 0.0075 - val_midlegL4: 0.0077 - val_midlegR4: 0.0084 - val_hindlegL4: 0.0071 - val_hindlegR4: 0.0070 - val_eyeL: 0.0058 - val_eyeR: 0.0058 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 32/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:15:27.416545: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0042 - head: 4.0799e-04 - thorax: 3.5961e-04 - abdomen: 0.0011 - wingL: 0.0017 - wingR: 0.0024 - forelegL4: 0.0067 - forelegR4: 0.0065 - midlegL4: 0.0065 - midlegR4: 0.0067 - hindlegL4: 0.0062 - hindlegR4: 0.0060 - eyeL: 0.0050 - eyeR: 0.0050 - val_loss: 0.0052 - val_head: 3.9909e-04 - val_thorax: 6.4130e-04 - val_abdomen: 0.0014 - val_wingL: 0.0039 - val_wingR: 0.0023 - val_forelegL4: 0.0075 - val_forelegR4: 0.0076 - val_midlegL4: 0.0093 - val_midlegR4: 0.0085 - val_hindlegL4: 0.0079 - val_hindlegR4: 0.0082 - val_eyeL: 0.0051 - val_eyeR: 0.0050 - lr: 1.0000e-04 - 12s/epoch - 60ms/step
Epoch 33/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:15:39.671708: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0040 - head: 3.8553e-04 - thorax: 3.5829e-04 - abdomen: 0.0010 - wingL: 0.0017 - wingR: 0.0023 - forelegL4: 0.0065 - forelegR4: 0.0064 - midlegL4: 0.0062 - midlegR4: 0.0063 - hindlegL4: 0.0058 - hindlegR4: 0.0056 - eyeL: 0.0047 - eyeR: 0.0048 - val_loss: 0.0051 - val_head: 4.5146e-04 - val_thorax: 4.6433e-04 - val_abdomen: 0.0013 - val_wingL: 0.0032 - val_wingR: 0.0027 - val_forelegL4: 0.0077 - val_forelegR4: 0.0075 - val_midlegL4: 0.0078 - val_midlegR4: 0.0081 - val_hindlegL4: 0.0079 - val_hindlegR4: 0.0070 - val_eyeL: 0.0058 - val_eyeR: 0.0057 - lr: 1.0000e-04 - 12s/epoch - 62ms/step
Epoch 34/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:15:51.898574: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0040 - head: 3.9407e-04 - thorax: 3.7663e-04 - abdomen: 0.0010 - wingL: 0.0016 - wingR: 0.0022 - forelegL4: 0.0064 - forelegR4: 0.0063 - midlegL4: 0.0061 - midlegR4: 0.0062 - hindlegL4: 0.0059 - hindlegR4: 0.0057 - eyeL: 0.0046 - eyeR: 0.0046 - val_loss: 0.0053 - val_head: 4.3299e-04 - val_thorax: 4.6334e-04 - val_abdomen: 0.0011 - val_wingL: 0.0031 - val_wingR: 0.0026 - val_forelegL4: 0.0064 - val_forelegR4: 0.0075 - val_midlegL4: 0.0095 - val_midlegR4: 0.0089 - val_hindlegL4: 0.0092 - val_hindlegR4: 0.0081 - val_eyeL: 0.0057 - val_eyeR: 0.0056 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 35/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:16:04.141239: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0038 - head: 3.8693e-04 - thorax: 3.5724e-04 - abdomen: 0.0010 - wingL: 0.0016 - wingR: 0.0022 - forelegL4: 0.0063 - forelegR4: 0.0061 - midlegL4: 0.0059 - midlegR4: 0.0059 - hindlegL4: 0.0056 - hindlegR4: 0.0054 - eyeL: 0.0043 - eyeR: 0.0044 - val_loss: 0.0053 - val_head: 4.1844e-04 - val_thorax: 3.9218e-04 - val_abdomen: 0.0010 - val_wingL: 0.0029 - val_wingR: 0.0027 - val_forelegL4: 0.0078 - val_forelegR4: 0.0069 - val_midlegL4: 0.0086 - val_midlegR4: 0.0088 - val_hindlegL4: 0.0079 - val_hindlegR4: 0.0071 - val_eyeL: 0.0067 - val_eyeR: 0.0070 - lr: 1.0000e-04 - 12s/epoch - 61ms/step
Epoch 36/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
2023-10-16 07:16:16.566185: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } }
200/200 - 12s - loss: 0.0037 - head: 3.9279e-04 - thorax: 3.7463e-04 - abdomen: 0.0010 - wingL: 0.0015 - wingR: 0.0020 - forelegL4: 0.0061 - forelegR4: 0.0060 - midlegL4: 0.0055 - midlegR4: 0.0057 - hindlegL4: 0.0055 - hindlegR4: 0.0054 - eyeL: 0.0044 - eyeR: 0.0044 - val_loss: 0.0048 - val_head: 3.7802e-04 - val_thorax: 3.4178e-04 - val_abdomen: 0.0010 - val_wingL: 0.0037 - val_wingR: 0.0025 - val_forelegL4: 0.0068 - val_forelegR4: 0.0080 - val_midlegL4: 0.0080 - val_midlegR4: 0.0078 - val_hindlegL4: 0.0065 - val_hindlegR4: 0.0068 - val_eyeL: 0.0052 - val_eyeR: 0.0058 - lr: 1.0000e-04 - 12s/epoch - 62ms/step
Epoch 37/200
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png
</details>

</details>


## How to reproduce

<!-- Use this section to describe the steps that a user would take to experience this bug. eg:-->
1. Open an .slp in GUI
2. Generate suggestions
3. Save
4. Export package with suggestions
5. File > Open the .pkg.slp in GUI
6. Run Training
7. See error