Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test spatialelement table join #208

Merged
merged 35 commits into from
Mar 19, 2024
Merged

Test spatialelement table join #208

merged 35 commits into from
Mar 19, 2024

Conversation

LucaMarconato
Copy link
Member

@LucaMarconato LucaMarconato commented Mar 13, 2024

Closes #181

Adds a test for checking that the plot is correct even when some rows of the table are not annotating instances of the spatial element or when the spatial element contains instances not in the table.

Also, checks that this works both for the case when the instance_key column has dtype int and when it's a str.

Eventually, the instance_key column will only have dtype int, as discussed here: scverse/spatialdata#158, but for the moment it would be easier if both were supported, as this PR will check.

).sum()
)
assert widget.viewer_model.viewer.layers[0].metadata.get("region_key") is not None


def test_sdatawidget_points(caplog, make_napari_viewer: Any):
sdata.points["many_points"] = PointsModel.parse(
@pytest.mark.xfail
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove this and adjust the asserts in the test.

@LucaMarconato
Copy link
Member Author

LucaMarconato commented Mar 18, 2024

Some additional comments, I will put them here while I test.

  • single_molecule from the merfish dataset can't be visualized.
    The error is
--> 431 subsample_points = points.iloc[subsample] if subsample else points
        points =                    x            y     cell_type
0        1278.683956  6020.642260  outside_VISp
1        1326.970330  6023.884788  outside_VISp
2        1292.026844  6059.337093  outside_VISp
3        1300.886241  6097.786264  outside_VISp
4        1232.410068  6102.884182  outside_VISp
...              ...          ...           ...
3714637  3161.427603  5192.594981  outside_VISp
3714638  3099.698528  5221.596008  outside_VISp
3714639  3084.582240  5297.234605  outside_VISp
3714640  3054.192051  5342.142346   VISp_II/III
3714641  3058.963217  5348.150185   VISp_II/III

[3714642 rows x 3 columns]
        subsample = <class 'numpy.ndarray'> (100000,) int64
    432 if subsample:
    433     adata = _left_join_spatialelement_table(
    434         {"points": {original_name: subsample_points}}, table_name, match_rows="left"
    435     )

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

@LucaMarconato
Copy link
Member Author

LucaMarconato commented Mar 18, 2024

  • Unfortunately also genes from the mibitof dataset can't be visualized (I checked them because they contain labels/table mismatches).
File ~/embl/projects/basel/napari-spatialdata/src/napari_spatialdata/_widgets.py:217, in AListWidget._(self=<napari_spatialdata._widgets.AListWidget object>, vec=cell_id
2      -0.042887
4      -0.004494
5     ...-0.039209
Name: vec, Length: 1023, dtype: float32, **kwargs={'key': 'CD31'})
    215 vec = pd.Series(vec, name="vec", index=instance_key_col)
    216 layer_meta = self.model.layer.metadata if self.model.layer is not None else None
--> 217 element_indices = pd.Series(layer_meta["indices"], name="element_indices")
        layer_meta = {'sdata': SpatialData object with:
├── Images
│     ├── 'point8_image': SpatialImage[cyx] (3, 1024, 1024)
│     ├── 'point16_image': SpatialImage[cyx] (3, 1024, 1024)
│     └── 'point23_image': SpatialImage[cyx] (3, 1024, 1024)
├── Labels
│     ├── 'point8_labels': SpatialImage[yx] (1024, 1024)
│     ├── 'point16_labels': SpatialImage[yx] (1024, 1024)
│     └── 'point23_labels': SpatialImage[yx] (1024, 1024)
└── Tables
      └── 'table': AnnData (3309, 36)
with coordinate systems:
▸ 'point8', with elements:
        point8_image (Images), point8_labels (Labels)
▸ 'point16', with elements:
        point16_image (Images), point16_labels (Labels)
▸ 'point23', with elements:
        point23_image (Images), point23_labels (Labels), 'adata': AnnData object with n_obs × n_vars = 1023 × 36
    obs: 'row_num', 'point', 'cell_id', 'X1', 'center_rowcoord', 'center_colcoord', 'cell_size', 'category', 'donor', 'Cluster', 'batch', 'library_id'
    uns: 'spatialdata_attrs'
    obsm: 'X_scanorama', 'X_umap', 'spatial', 'region_key': 'library_id', 'instance_key': 'cell_id', 'table_names': ['table'], 'name': 'point16_labels', '_active_in_cs': {'point16'}, '_current_cs': 'point16'}
        pd = <module 'pandas' from '/Users/macbook/miniconda3/envs/ome/lib/python3.10/site-packages/pandas/__init__.py'>
    218 diff_element_table = set(instance_key_col).symmetric_difference(element_indices)
    219 merge_vec = pd.merge(element_indices, vec, left_on="element_indices", right_index=True, how="left")[
    220     "vec"
    221 ].fillna(0, axis=0)
    ```

@LucaMarconato
Copy link
Member Author

LucaMarconato commented Mar 18, 2024

I could reproduce the bug appearing with the mibitof dataset in test_partial_table_matching_with_arbitrary_ordering() (it appears both for labels and multiscale_labels`. I could not reproduce the points bug with the test (because no subsampling is happening).

I verified the robustness against shuffling and it works. I will now check the get_extent() code for the automatic determination of the default points size.

@melonora melonora marked this pull request as ready for review March 19, 2024 14:29
@melonora melonora merged commit 8f9ac6a into multi_table Mar 19, 2024
1 of 2 checks passed
melonora added a commit that referenced this pull request Mar 19, 2024
* Multi table (#198)

* update napari dependency

* change deprecated is_categorical_dtype

* Iinitial table widget

* add table_names to model

* More self explainable function name

* allow changing tables

* fix some tests

* Fixed tests

* Multi table (#200)

* update napari dependency

* change deprecated is_categorical_dtype

* Iinitial table widget

* add table_names to model

* More self explainable function name

* allow changing tables

* fix some tests

* Fixed tests

* remove legacy squidpy

* remove legacy squidpy

* remove coordinates

* fix tests

* silence warnings

* fix last tests

* fix mypy

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Multi table (#202)

* update napari dependency

* change deprecated is_categorical_dtype

* Iinitial table widget

* add table_names to model

* More self explainable function name

* allow changing tables

* fix some tests

* Fixed tests

* remove legacy squidpy

* remove legacy squidpy

* remove coordinates

* fix tests

* silence warnings

* fix last tests

* fix mypy

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add table for adatascatter

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix adding layer without table

* fix removing layer

* fix wrong point size when affine specified (#193)

* add color by in adata view (#204)

* Issue #180 (#205)

* Multi table (#196)

* update napari dependency

* change deprecated is_categorical_dtype

* Iinitial table widget

* add table_names to model

* More self explainable function name

* allow changing tables

* fix some tests

* Fixed tests

* Revert "Multi table (#196)" (#197)

This reverts commit aa56cca.

* Avoid exception when registering shortcut (#201)

* Fix undefined attribute _labels_key

Accessing the property `model.labels_key` gave an `AttributeError("'ImageModel' object has no attribute '_labels_key'")`

* Avoid exception when key binding collides with existing

If the shortcut is already registered, Napari does not warn but raise an error. To avoid crashing Napari, we must provide the override argument (no matter whether preserving existing or overriding with the new command).

* add color by in adata view

* remove _scale from model

* remove points coordinates

* remove point_diameter

* remove spot diameter

---------

Co-authored-by: aeisenbarth <54448967+aeisenbarth@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>

* Issue #192 (#207)

* Multi table (#196)

* update napari dependency

* change deprecated is_categorical_dtype

* Iinitial table widget

* add table_names to model

* More self explainable function name

* allow changing tables

* fix some tests

* Fixed tests

* Revert "Multi table (#196)" (#197)

This reverts commit aa56cca.

* Avoid exception when registering shortcut (#201)

* Fix undefined attribute _labels_key

Accessing the property `model.labels_key` gave an `AttributeError("'ImageModel' object has no attribute '_labels_key'")`

* Avoid exception when key binding collides with existing

If the shortcut is already registered, Napari does not warn but raise an error. To avoid crashing Napari, we must provide the override argument (no matter whether preserving existing or overriding with the new command).

* add color by in adata view

* remove _scale from model

* remove points coordinates

* remove point_diameter

* remove spot diameter

* fix #192

---------

Co-authored-by: aeisenbarth <54448967+aeisenbarth@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>

* Test spatialelement table join (#208)

* fixed all the non-cytassist images

* refactor get_init_table_list

* fix permutation

* use join

* use join for view

* change hardcoded sample threshold

* bunch of updates

* adjust test

* change into left join

* change into left join

* random hex colors

* get actual index

* change indices and adata

* match rows left

* fix joins

* rename to vars

* fix color by

* fix radii

* pass cs

* fix _calc_default_radii()

* fix tests

* improve speed test

* removed comments

* added code to show circles as ellipses

* correct point layer size after scaling

* fix labels color by

* fix mibitof example

* last bug fixes

* initial step points columns

* add dask import back

* fix points columns

---------

Co-authored-by: wmv_hpomen <w-mv@hotmail.com>
Co-authored-by: Wouter-Michiel Vierdag <michiel.vierdag@embl.de>

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: LucaMarconato <2664412+LucaMarconato@users.noreply.github.com>
Co-authored-by: aeisenbarth <54448967+aeisenbarth@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>
@LucaMarconato LucaMarconato deleted the test_for_join branch March 23, 2024 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants