Failing metric fails to cancel

[jimmymathews]

In some cases a pending metric that raises a ValueError fails to end the job and wrap up.

This causes infinitely pending metric calculation.

ondemand    | 03-18 23:26:49 [  DEBUG  ] ondemand.providers.squidpy_provider:172: Computed feature value of 14: lesion 0_2, 1.0
ondemand    | /usr/local/lib/python3.11/site-packages/spatialprofilingtoolbox/workflow/common/squidpy.py:158: UserWarning: All phenotypes provided had identical values. Only one cluster could be made.
ondemand    |   warn('All phenotypes provided had identical values. Only one cluster could be made.')
ondemand    | 03-18 23:26:51 [  DEBUG  ] ondemand.providers.squidpy_provider:172: Computed feature value of 14: lesion 0_3, 1.0
ondemand    | Exception in thread Thread-11 (have_feature_computed):
ondemand    | Traceback (most recent call last):
ondemand    |   File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
ondemand    |     self.run()
ondemand    |   File "/usr/local/lib/python3.11/threading.py", line 982, in run
ondemand    |     self._target(*self._args, **self._kwargs)
ondemand    |   File "/usr/local/lib/python3.11/site-packages/spatialprofilingtoolbox/ondemand/providers/squidpy_provider.py", line 163, in have_feature_computed
ondemand    |     value = compute_squidpy_metric_for_one_sample(
ondemand    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ondemand    |   File "/usr/local/lib/python3.11/site-packages/spatialprofilingtoolbox/workflow/common/squidpy.py", line 69, in compute_squidpy_metric_for_one_sample
ondemand    |     return _summarize_ripley(_ripley(adata))
ondemand    |                              ^^^^^^^^^^^^^^
ondemand    |   File "/usr/local/lib/python3.11/site-packages/spatialprofilingtoolbox/workflow/common/squidpy.py", line 186, in _ripley
ondemand    |     result = ripley(adata, 'cluster', copy=True)
ondemand    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ondemand    |   File "/usr/local/lib/python3.11/site-packages/squidpy/gr/_ripley.py", line 143, in ripley
ondemand    |     distances, _ = tree_c.kneighbors(random, n_neighbors=n_neigh)
ondemand    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ondemand    |   File "/usr/local/lib/python3.11/site-packages/sklearn/neighbors/_base.py", line 810, in kneighbors
ondemand    |     raise ValueError(
ondemand    | ValueError: Expected n_neighbors <= n_samples,  but n_samples = 1, n_neighbors = 2
apiserver   | 03-18 23:26:52 [  DEBUG  ] ondemand.service_client:204: Host and port (with service "squidpy"): ('fast-counts-server', 8016)
ondemand    | 03-18 23:26:52 [  INFO   ] ondemand.request_handler: Request: b'ripley\x1dMelanoma intralesional IL2\x1dSOX10\x1dCD20\x1eCD3\x1eCD4\x1eCD8\x1eFOXP3'
ondemand    | 03-18 23:26:52 [  DEBUG  ] ondemand.request_handler:228: ['Melanoma intralesional IL2', 'SOX10', 'CD20\x1eCD3\x1eCD4\x1eCD8\x1eFOXP3']
ondemand    | 03-18 23:26:52 [  DEBUG  ] ondemand.providers.pending_provider:29: Requesting computation.
ondemand    | 03-18 23:26:52 [  DEBUG  ] ondemand.providers.pending_provider:99: Number of values possible to be computed: 7
ondemand    | 03-18 23:26:52 [  DEBUG  ] ondemand.providers.pending_provider:128: Actual number computed: 3
ondemand    | 03-18 23:26:52 [  DEBUG  ] ondemand.providers.pending_provider:42: Already pending.

[jimmymathews]

Related, there are times when a lightly-caught exception in the ondemand service is mishandled by the API service:

 Host and port (with service "squidpy"): ('fast-counts-server', 8016)
ondemand    | 03-18 23:33:34 [  INFO   ] ondemand.request_handler: Request: b'co-occurrence\x1dMelanoma intralesional IL2\x1dCD3\x1eCD4\x1dCD20\x1eCD56\x1eCD8\x1eFOXP3\x1eSOX10\x1d100.0'
ondemand    | 03-18 23:33:34 [  DEBUG  ] ondemand.request_handler:228: ['Melanoma intralesional IL2', 'CD3\x1eCD4', 'CD20\x1eCD56\x1eCD8\x1eFOXP3\x1eSOX10', '100.0']
ondemand    | 03-18 23:33:34 [  INFO   ] ondemand.request_handler: Query to on-demand computation service could not be handled.
apiserver   | INFO:     172.18.0.1:38940 - "GET /request-spatial-metrics-computation-custom-phenotype/?negative_marker=CD20&negative_marker=CD56&negative_marker=CD8&negative_marker=FOXP3&negative_marker=SOX10&positive_marker=CD3&positive_marker=CD4&study=Melanoma+intralesional+IL2&feature_class=co-occurrence&radius=100 HTTP/1.1" 500 Internal Server Error
apiserver   | ERROR:    Exception in ASGI application
apiserver   | Traceback (most recent call last):
apiserver   |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 108, in __call__
apiserver   |     response = await self.dispatch_func(request, call_next)
apiserver   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
...
apiserver   |     |   File "/usr/local/lib/python3.11/site-packages/spatialprofilingtoolbox/apiserver/app/main.py", line 286, in get_squidpy_metrics
apiserver   |     |     metrics = requester.get_squidpy_metrics(
apiserver   |     |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
apiserver   |     |   File "/usr/local/lib/python3.11/site-packages/spatialprofilingtoolbox/ondemand/service_client.py", line 169, in get_squidpy_metrics
apiserver   |     |     values=response['metrics'],
apiserver   |     |            ~~~~~~~~^^^^^^^^^^^
apiserver   |     | KeyError: 'metrics'
apiserver   |     +------------------------------------