Session is closed error shuts down watcher service

[swelborn]

Not sure where this is coming from, but posting so that the logs don't get rotated out. Come back to investigate later.

2024-01-19 15:37:18,601 - watch - INFO - 99.97829861111111
2024-01-19 15:37:19,572 - watch - INFO - 99.98857821637426
2024-01-19 15:37:19,581 - watch - INFO - 99.9802022417154
2024-01-19 15:37:19,589 - watch - INFO - 99.98705531189084
2024-01-19 15:37:19,597 - watch - INFO - 99.98857821637426
2024-01-19 15:37:19,604 - watch - INFO - 99.9802022417154
2024-01-19 15:37:19,610 - watch - INFO - 99.98705531189084
2024-01-19 15:37:19,618 - watch - INFO - 99.97829861111111
2024-01-19 15:37:20,588 - watch - INFO - 99.98857821637426
2024-01-19 15:37:20,607 - watch - ERROR - Exception in monitoring loop.
Traceback (most recent call last):
  File "/home/kitware/distiller/cli/watch/distiller/watch.py", line 88, in monitor
    await handler.on_event(event)
  File "/home/kitware/distiller/cli/watch/distiller/modes/scan_4d_files.py", line 238, in on_event
    await self._send_disk_usage(str(path))
  File "/home/kitware/distiller/cli/watch/distiller/modes/scan_4d_files.py", line 126, in _send_disk_usage
    microscope = await get_microscope_by_id(self.session, self.microscope_id)
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/tenacity/_asyncio.py", line 86, in async_wrapped
    return await fn(*args, **kwargs)
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/tenacity/_asyncio.py", line 48, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/tenacity/__init__.py", line 349, in iter
    return fut.result()
  File "/home/kitware/.pyenv/versions/3.9.7/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/home/kitware/.pyenv/versions/3.9.7/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/tenacity/_asyncio.py", line 51, in __call__
    result = await fn(*args, **kwargs)
  File "/home/kitware/distiller/cli/watch/distiller/utils.py", line 77, in get_microscope_by_id
    r.raise_for_status()
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1000, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 503, message='Service Temporarily Unavailable', url=URL('https://distiller.lbl.gov/api/v1/microscopes/1')
2024-01-19 15:37:21,613 - watch - ERROR - Exception posting file event.
Traceback (most recent call last):
  File "/home/kitware/distiller/cli/watch/distiller/modes/scan_4d_files.py", line 228, in _log_exception
    task.result()
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/tenacity/_asyncio.py", line 86, in async_wrapped
    return await fn(*args, **kwargs)
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/tenacity/_asyncio.py", line 48, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/tenacity/__init__.py", line 349, in iter
    return fut.result()
  File "/home/kitware/.pyenv/versions/3.9.7/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/home/kitware/.pyenv/versions/3.9.7/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/tenacity/_asyncio.py", line 51, in __call__
    result = await fn(*args, **kwargs)
  File "/home/kitware/distiller/cli/watch/distiller/modes/scan_4d_files.py", line 79, in post_file_event
    async with session.post(
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/aiohttp/client.py", line 1117, in __aenter__
    self._resp = await self._coro
  File "/home/kitware/.pyenv/versions/3.9.7/envs/distiller/lib/python3.9/site-packages/aiohttp/client.py", line 381, in _request
    raise RuntimeError("Session is closed")
RuntimeError: Session is closed