Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump _restclient's aiohttp and aiohttp_client_cache version deps #237

Merged
merged 2 commits into from
Sep 19, 2023

Conversation

aaraney
Copy link
Member

@aaraney aaraney commented Sep 19, 2023

bump aiohttp and aiohttp_client_cache version deps

See #103 for background as to why aiohttp was previously pinned. In short, the issues that led to pinning aiohttp have be resolved upstream.

aiohttp_client_cache >= 0.9.0 needed b.c. requests-cache/aiohttp-client-cache#173

Changes

  • _restclient depends on aiohttp~=3.8 and aiohttp_client_cache>=0.9.0

Checklist

  • PR has an informative and human-readable title
  • PR is well outlined and documented. See #12 for an example
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (see CONTRIBUTING.md)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output) using numpy docstring formatting
  • Placeholder code is flagged / future todos are captured in comments
  • Reviewers requested with the Reviewers tool ➡️

See NOAA-OWP#103 for background as to why aiohttp was previously pinned. In
short, the issues that led to pinning aiohttp have be resolved upstream.

aiohttp_client_cache >= 0.9.0 needed b.c.
requests-cache/aiohttp-client-cache#173
@jarq6c jarq6c merged commit 1c1db5a into NOAA-OWP:main Sep 19, 2023
3 checks passed
@jarq6c
Copy link
Collaborator

jarq6c commented Sep 19, 2023

Simple enough!

@aaraney
Copy link
Member Author

aaraney commented Sep 19, 2023

I will open up another PR that bumps the version of this when i'm done testing things out. Im running into an issue with the nwis_client cli.

@jarq6c
Copy link
Collaborator

jarq6c commented Sep 19, 2023

Any idea why this PR would cause the deploy-gh workflow to fail?

@aaraney
Copy link
Member Author

aaraney commented Sep 20, 2023

I think it is because the source didnt change. Still kind of odd that it failed.

@aaraney
Copy link
Member Author

aaraney commented Sep 25, 2023

Having spend a day or so looking at this last week, I don't know that it is going to be as straight forward to fix as I was hoping. Here is a brief-ish explanation.

hydrotools._restclient uses and expands upon aiohttp_cache_client to provide a request client that handles request caching and retry / backoff capabilities. aiohttp_cache_client supports several cache storage backends, sqlite being one of them. sqlite is the only backend that _restclient supports at the moment. aiohttp_cache_client uses aiosqlite as a backend driver to support its sqlite backend capabilities. aiosqlite uses threading to provide a async sqlite client. To accomplish this, aiosqlite runs its own event loop (see aiosqlite.Connection.run()) in a separate thread that effectively acts like a scheduler.

Inside aiosqlite's loop, a tuple of a callable and an unfulfilled asyncio.Future are pulled from a work queue. The callable is called and the result of the callable is scheduled to be set as the result of the unfulfilled asyncio.Future on the main event loop. This includes closing the aiosqlite.Connection instance.

This means that the runtime of the main event loop must be strictly longer than the aiosqlite.Connection or else you cannot close the aiosqlite.Connection instance because the created future will never have a result. Likewise, this means that if a aiosqlite.Connection is created, but not explicitly or implicitly (async context manager) closed, the aiosqlite.Connection event loop will run forever and the program will not exit. Consequently, this means that any code that manages a aiosqlite.Connection instance is forced to require users of that code to explicitly or implicitly close the, possibly encapsulated, aiosqlite.Connection. Likely meaning that either a close() method is added to the public api of the object or async context manager is added and either is required to correctly use the object or the code will hang without exiting. A full description of the issue along with code examples can be found here.

These changes would require changes to the semantics of the _restclient.RestClient. Namely, addition of a close() or context manager implementation and a requirement that downstream code use _restclient.RestClient in a particular way that has not been required in the past.

@jarq6c
Copy link
Collaborator

jarq6c commented Sep 25, 2023

Thanks for the deep-dive @aaraney ! Should we link this to a new issue on the hydrotools repo?

@aaraney
Copy link
Member Author

aaraney commented Sep 25, 2023

@jarq6c, for sure! Yeah, I will open an issue to track this.

@aaraney
Copy link
Member Author

aaraney commented Sep 25, 2023

I opened #238 to track this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants