Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New label placed outside the frame #1551

Open
1 of 4 tasks
roaldarbol opened this issue Oct 14, 2023 · 5 comments
Open
1 of 4 tasks

New label placed outside the frame #1551

roaldarbol opened this issue Oct 14, 2023 · 5 comments
Labels
bug Something isn't working fixed in future release Fix or feature is merged into develop and will be available in future release.

Comments

@roaldarbol
Copy link

roaldarbol commented Oct 14, 2023 •
edited

Bug description

I am labelling videos of different frame sizes in sleap v1.3.3, and when I move a large frame to a small frame and add a new instance, the labels are placed based (it seems to me) based on the previous positions in pixels. However, because the new frame is much small, some labels are moved outside the new frame and are impossible to re-position (there's the Alt/Option trick from #896 (comment), to move the entire instance as a workaround ).

Expected behaviour

I would expect the new labels to be placed in relative position. So if the previous frame was 2000pixels wide and a point was at 1000, then in a frame 500px wide, it should be placed at 250. Similarly the other way around.

Actual behaviour

In the above example it seems to be placed at px 1000 still in the new frame, which is then outside the frame.

Your personal set up

  • OS:
    macOS 13.6
  • Version(s):
Environment packages 
(sleap) ➜  ~ conda list                                               
# packages in environment at /Users/roaldarbol/mambaforge/envs/sleap:
#
# Name                    Version                   Build  Channel
abseil-cpp                20211102.0           hddbf539_3    conda-forge
absl-py                   2.0.0              pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.6            py39ha09f3b3_1    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
aom                       3.5.0                hf0c8a7f_0    conda-forge
astunparse                1.6.3              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.3              pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
blinker                   1.6.3              pyhd8ed1ab_0    conda-forge
blosc                     1.21.5               heccf04b_0    conda-forge
brotli                    1.0.9                hb7f2c08_9    conda-forge
brotli-bin                1.0.9                hb7f2c08_9    conda-forge
brotli-python             1.0.9            py39h7a8716b_9    conda-forge
brunsli                   0.1                  h046ec9c_0    conda-forge
bzip2                     1.0.8                h0d85af4_4    conda-forge
c-ares                    1.20.1               h10d778d_0    conda-forge
c-blosc2                  2.10.5               h354e526_0    conda-forge
ca-certificates           2023.7.22            h8857fd0_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.3.1              pyhd8ed1ab_0    conda-forge
cairo                     1.16.0            h904041c_1014    conda-forge
cattrs                    1.1.1              pyhd8ed1ab_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.16.0           py39h18ef598_0    conda-forge
cfitsio                   4.2.0                hd56cc12_0    conda-forge
charls                    2.3.4                he49afe7_0    conda-forge
charset-normalizer        3.3.0              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
contourpy                 1.1.1            py39h8ee36c8_1    conda-forge
cryptography              39.0.0           py39h7eb6a14_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
dav1d                     1.2.1                h0dc2134_0    conda-forge
efficientnet              1.0.0                    pypi_0    pypi
expat                     2.5.0                hf0c8a7f_1    conda-forge
ffmpeg                    4.4.2           gpl_h8b4fe81_112    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h5bb23bf_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.43.1           py39ha09f3b3_0    conda-forge
freetype                  2.12.1               h60636b9_2    conda-forge
frozenlist                1.4.0            py39hdc70f33_1    conda-forge
gast                      0.4.0              pyh9f0ad1d_0    conda-forge
geos                      3.12.0               he965462_0    conda-forge
gettext                   0.21.1               h8a4c099_0    conda-forge
giflib                    5.2.1                hb7f2c08_3    conda-forge
glib                      2.78.0               h7d26f99_0    conda-forge
glib-tools                2.78.0               h7d26f99_0    conda-forge
gmp                       6.2.1                h2e338ed_0    conda-forge
gnutls                    3.7.8                h207c4f0_0    conda-forge
google-auth               2.23.3             pyhca7485f_0    conda-forge
google-auth-oauthlib      0.4.6              pyhd8ed1ab_0    conda-forge
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
graphite2                 1.3.13            h2e338ed_1001    conda-forge
grpc-cpp                  1.46.4               hea9d06c_3    conda-forge
grpcio                    1.46.4           py39h389b050_3    conda-forge
gst-plugins-base          1.22.6               hb5d3a86_2    conda-forge
gstreamer                 1.22.6               h840fbdc_2    conda-forge
h5py                      3.8.0           nompi_py39h7d40783_100    conda-forge
harfbuzz                  5.3.0                h08f8713_0    conda-forge
hdf5                      1.12.2          nompi_h70e9adf_101    conda-forge
icu                       70.1                 h96cf925_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
image-classifiers         1.0.0                    pypi_0    pypi
imagecodecs               2022.9.26        py39h8036967_4    conda-forge
imageio                   2.31.5             pyh8c1a49c_0    conda-forge
imgaug                    0.4.0              pyhd8ed1ab_1    conda-forge
imgstore                  0.2.9                    pypi_0    pypi
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib-resources       6.1.0              pyhd8ed1ab_0    conda-forge
importlib_resources       6.1.0              pyhd8ed1ab_0    conda-forge
jasper                    2.0.33               h7c6fec8_1    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   hb7f2c08_3    conda-forge
jsmin                     3.0.1              pyhd8ed1ab_0    conda-forge
jsonpickle                1.2                        py_0    conda-forge
jsonschema                4.19.0                   pypi_0    pypi
jsonschema-specifications 2023.7.1                 pypi_0    pypi
jxrlib                    1.1                  h35c211d_2    conda-forge
keras                     2.9.0              pyhd8ed1ab_0    conda-forge
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2              pyhd8ed1ab_0    conda-forge
kiwisolver                1.4.5            py39h8ee36c8_1    conda-forge
krb5                      1.20.1               h0165f36_0    conda-forge
lame                      3.100             hb7f2c08_1003    conda-forge
lazy_loader               0.3                pyhd8ed1ab_0    conda-forge
lcms2                     2.14                 h90f4b2a_0    conda-forge
lerc                      4.0.0                hb486fe8_0    conda-forge
libabseil                 20211102.0      cxx17_h844d122_3    conda-forge
libaec                    1.1.2                he965462_1    conda-forge
libavif                   0.11.1               h22361c6_2    conda-forge
libblas                   3.9.0           18_osx64_openblas    conda-forge
libbrotlicommon           1.0.9                hb7f2c08_9    conda-forge
libbrotlidec              1.0.9                hb7f2c08_9    conda-forge
libbrotlienc              1.0.9                hb7f2c08_9    conda-forge
libcblas                  3.9.0           18_osx64_openblas    conda-forge
libclang                  13.0.1          root_62804_h2961583_3    conda-forge
libclang13                16.0.1          default_h395b6a0_0    conda-forge
libcurl                   7.87.0               haf73cf8_0    conda-forge
libcxx                    16.0.6               hd57cbcb_0    conda-forge
libdeflate                1.14                 hb7f2c08_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 haf1e3a3_1    conda-forge
libexpat                  2.5.0                hf0c8a7f_1    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgfortran               5.0.0           13_2_0_h97931a8_1    conda-forge
libgfortran5              13.2.0               h2873a65_1    conda-forge
libglib                   2.78.0               hc62aa5d_0    conda-forge
libiconv                  1.17                 hac89ed1_0    conda-forge
libidn2                   2.3.4                hb7f2c08_0    conda-forge
liblapack                 3.9.0           18_osx64_openblas    conda-forge
liblapacke                3.9.0           18_osx64_openblas    conda-forge
libllvm13                 13.0.1               h64f94b2_2    conda-forge
libllvm16                 16.0.1               h7001e86_0    conda-forge
libnghttp2                1.51.0               h0dd9d14_0    conda-forge
libogg                    1.3.4                h35c211d_1    conda-forge
libopenblas               0.3.24          openmp_h48a4ad5_0    conda-forge
libopencv                 4.6.0            py39h93c2317_3    conda-forge
libopus                   1.3.1                hc929b4f_1    conda-forge
libpng                    1.6.39               ha978bb4_0    conda-forge
libpq                     15.1                 hdbdeef1_3    conda-forge
libprotobuf               3.20.3               hbc0c0cd_0    conda-forge
libsodium                 1.0.18               hbcb3906_1    conda-forge
libsqlite                 3.43.2               h92b6c6a_0    conda-forge
libssh2                   1.10.0               h7535e13_3    conda-forge
libtasn1                  4.19.0               hb7f2c08_0    conda-forge
libtiff                   4.4.0                h6268bbc_5    conda-forge
libunistring              0.9.10               h0d85af4_0    conda-forge
libvorbis                 1.3.7                h046ec9c_0    conda-forge
libvpx                    1.11.0               he49afe7_3    conda-forge
libwebp-base              1.3.2                h0dc2134_0    conda-forge
libxcb                    1.13              h0d85af4_1004    conda-forge
libxml2                   2.10.3               h201ad9d_4    conda-forge
libxslt                   1.1.37               h5d22bc9_0    conda-forge
libzlib                   1.2.13               h8a1eda9_5    conda-forge
libzopfli                 1.0.3                h046ec9c_0    conda-forge
llvm-openmp               17.0.2               hff08bdf_0    conda-forge
lz4-c                     1.9.4                hf0c8a7f_0    conda-forge
markdown                  3.5                pyhd8ed1ab_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3            py39hdc70f33_1    conda-forge
matplotlib-base           3.8.0            py39h7070ae8_2    conda-forge
mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
multidict                 6.0.4            py39ha30fb19_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.32               h7ebae80_0    conda-forge
mysql-libs                8.0.32               hc37e033_0    conda-forge
ncurses                   6.4                  hf0c8a7f_0    conda-forge
ndx-pose                  0.1.1                    pypi_0    pypi
nettle                    3.8.1                h96f3785_1    conda-forge
networkx                  3.1                pyhd8ed1ab_0    conda-forge
nixio                     1.5.3                    pypi_0    pypi
nspr                      4.35                 hea0b92c_0    conda-forge
nss                       3.94                 hd6ac835_0    conda-forge
numpy                     1.22.4           py39h677350a_0    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
opencv                    4.6.0            py39h6e9494a_3    conda-forge
openh264                  2.3.1                hf0c8a7f_2    conda-forge
openjpeg                  2.5.0                h5d0d7b0_1    conda-forge
openssl                   1.1.1w               h8a1eda9_0    conda-forge
opt_einsum                3.3.0              pyhc1e730c_2    conda-forge
p11-kit                   0.24.1               h65f8906_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    2.1.1            py39hcc9a0c8_1    conda-forge
patsy                     0.5.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                h1c4e4bc_0    conda-forge
pillow                    9.2.0            py39h35d4919_3    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pixman                    0.42.2               he965462_0    conda-forge
protobuf                  3.20.3           py39h7a8716b_1    conda-forge
psutil                    5.9.5            py39hdc70f33_1    conda-forge
pthread-stubs             0.4               hc929b4f_1001    conda-forge
py-opencv                 4.6.0            py39h71a6800_3    conda-forge
pyasn1                    0.5.0              pyhd8ed1ab_0    conda-forge
pyasn1-modules            0.3.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
pyjwt                     2.8.0              pyhd8ed1ab_0    conda-forge
pykalman                  0.9.5                      py_1    conda-forge
pynwb                     2.5.0                    pypi_0    pypi
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyside2                   5.15.8           py39heedc0ee_2    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.9.15          h531fd05_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-flatbuffers        1.12               pyhd8ed1ab_1    conda-forge
python-rapidjson          1.12             py39h110ca85_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.9                      4_cp39    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pywavelets                1.4.1            py39hd01001f_1    conda-forge
pyyaml                    6.0.1            py39hdc70f33_1    conda-forge
pyzmq                     25.1.1           py39hdac9eea_1    conda-forge
qimage2ndarray            1.10.0                   pypi_0    pypi
qt-main                   5.15.8               h1d3b3f8_6    conda-forge
qtpy                      2.4.0              pyhd8ed1ab_0    conda-forge
re2                       2022.06.01           hb486fe8_1    conda-forge
readline                  8.2                  h9e318b2_1    conda-forge
referencing               0.30.2                   pypi_0    pypi
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
rich                      13.6.0             pyhd8ed1ab_0    conda-forge
rpds-py                   0.10.3                   pypi_0    pypi
rsa                       4.9                pyhd8ed1ab_0    conda-forge
ruamel-yaml               0.17.32                  pypi_0    pypi
ruamel-yaml-clib          0.2.7                    pypi_0    pypi
scikit-image              0.22.0           py39h5d65943_2    conda-forge
scikit-learn              1.0              py39hd2caeff_1    conda-forge
scikit-video              1.1.11             pyh24bf2e0_0    conda-forge
scipy                     1.9.0            py39h29d19b3_0    conda-forge
seaborn                   0.13.0               hd8ed1ab_0    conda-forge
seaborn-base              0.13.0             pyhd8ed1ab_0    conda-forge
segmentation-models       1.0.1                    pypi_0    pypi
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shapely                   2.0.2            py39h808d968_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleap                     1.3.3                    pypi_0    pypi
snappy                    1.1.10               h225ccf5_0    conda-forge
sqlite                    3.43.2               h7461747_0    conda-forge
statsmodels               0.14.0           py39h5b4affa_2    conda-forge
svt-av1                   1.4.1                hf0c8a7f_0    conda-forge
tensorboard               2.9.0              pyhd8ed1ab_0    conda-forge
tensorboard-data-server   0.6.1            py39h7eb6a14_4    conda-forge
tensorboard-plugin-wit    1.8.1              pyhd8ed1ab_0    conda-forge
tensorflow                2.9.1           cpu_py39hb0a6171_0    conda-forge
tensorflow-base           2.9.1           cpu_py39hdd40284_0    conda-forge
tensorflow-estimator      2.9.1           cpu_py39hd441b4c_0    conda-forge
tensorflow-hub            0.12.0             pyhca92ed8_0    conda-forge
termcolor                 2.3.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
tifffile                  2022.10.10         pyhd8ed1ab_0    conda-forge
tk                        8.6.13               hef22860_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
tzlocal                   5.0.1                    pypi_0    pypi
unicodedata2              15.1.0           py39hdc70f33_0    conda-forge
urllib3                   2.0.6              pyhd8ed1ab_0    conda-forge
werkzeug                  3.0.0              pyhd8ed1ab_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
wrapt                     1.15.0           py39hdc70f33_1    conda-forge
x264                      1!164.3095           h775f41a_2    conda-forge
x265                      3.5                  hbb4e6a2_3    conda-forge
xorg-libxau               1.0.11               h0dc2134_0    conda-forge
xorg-libxdmcp             1.1.3                h35c211d_0    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
yaml                      0.2.5                h0d85af4_2    conda-forge
yarl                      1.9.2            py39ha09f3b3_1    conda-forge
zeromq                    4.3.5                h93d8f39_0    conda-forge
zfp                       1.0.0                hf3d7188_4    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h8a1eda9_5    conda-forge
zlib-ng                   2.0.7                hb7f2c08_0    conda-forge
zstd                      1.5.5                h829000d_0    conda-forge

Screenshots

Screenshot 2023-10-14 at 18 13 19

How to reproduce

  1. Load two videos of different frame sizes.
  2. Label a frame from the biggest one first
  3. Add instance in a frame from the smaller video
@roaldarbol roaldarbol added the bug Something isn't working label Oct 14, 2023
@roaldarbol
Copy link
Author

I guess the root issue is that it's expecting frames to be similar. I'm also getting some weird rotations on new instances that are supposed to be the same as the previous frame:
Screenshot 2023-10-14 at 19 30 13

@roomrys
Copy link
Collaborator

roomrys commented Oct 16, 2023

Hi @roaldarbol,

I have an inkling that this shifting of predictions likely has to do with the resize_and_pad_to_target and related parameters in the training_config.json. This parameter should be set to true, but what I believe might be happening is that the target_height and target_width are set to the incorrect dimensions. I see you have variable sized videos... so, what I would try is:

  1. Find the training_config.json that you have been selecting for use in the GUI
  2. Find the resize_and_pad_to_target, target_height, and target_width parameters
  3. Ensure that target_height and target_width are both set to "null" (no quotes)
  4. Save the training_config.json
  5. Repeat 1-4 for each training_config.json to be used (2 for the top-down)
  1. (Re)Open the Training Widget GUI
  2. Select the edited training_config.json
  3. Optionally, if on 1.3.3+ "Copy to Clipboard" and paste the config info somewhere to verify your changes were applied
  4. Repeat 6-8 for each training_config.json to be used
  1. Rerun training

Let me know if you need more guidance/clarification.

Thanks,
Liezl

@roaldarbol
Copy link
Author

Hi @roomrys! Thanks for the reply!
I am not doing predictions just yet, this is just labelling frames and using Add instance (or Cmd+I). So I wouldn't assume that should be affected by training_config.json - in fact I don't even have such a file yet as I haven't trained.

@roomrys
Copy link
Collaborator

roomrys commented Oct 18, 2023 •
edited

Ah yes, thanks for clarifying! I was definitely adding in my own assumptions (and overlooked the number of labeled frames in the project). This is a case that we haven't covered! While we have a bit of code to ensure that non-visible points are initialized within the frame, SLEAP just copies the visible points from the copy_instance

sleap/sleap/gui/commands.py

Lines 2951 to 2957 in 1e0627a

# just copy x, y, and visible
# we don't want to copy a PredictedPoint or score attribute
new_instance[node] = Point(
x=copy_instance[node].x,
y=copy_instance[node].y,
visible=copy_instance[node].visible,
complete=mark_complete,

We would need to add a check here that ensures the point is within the current frame... maybe shifting all points over to fit in the current frame.

Until this is fixed and released, using the force-directed method to add an instance should add the points at the location you are right-clicking.

@roaldarbol
Copy link
Author

roaldarbol commented Oct 18, 2023 •
edited

Yeah, that sounds just right - I think you've nailed the root of the issue, and your proposed fix sounds exactly like what I'd expect! I think you could just include a scaling factor like new_x = old_x * (new_width/old_width) and the same for y coordinates and height. And I think the check could just be new_width == old_width (and again similar for height).

@roomrys roomrys changed the title Labels placed outside the frame New label placed outside the frame Oct 19, 2023
@roomrys roomrys mentioned this issue Oct 25, 2023
Merged
11 tasks
@roomrys roomrys mentioned this issue Aug 16, 2023
Open
4 tasks
@roomrys roomrys added the fixed in future release Fix or feature is merged into develop and will be available in future release. label Jan 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working fixed in future release Fix or feature is merged into develop and will be available in future release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@roaldarbol @roomrys