Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading is not allowed after reader was completed #2692

Open
nathanmascitelli opened this issue Apr 3, 2024 · 2 comments
Open

Reading is not allowed after reader was completed #2692

nathanmascitelli opened this issue Apr 3, 2024 · 2 comments

Comments

@nathanmascitelli
Copy link

nathanmascitelli commented Apr 3, 2024

Hi,

We are getting the following exception intermittently in production:

at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_NoReadingAllowed()
at System.IO.Pipelines.Pipe.AdvanceReader(SequencePosition& consumed, SequencePosition& examined)
at System.IO.Pipelines.Pipe.DefaultPipeReader.AdvanceTo(SequencePosition consumed, SequencePosition examined)
at StackExchange.Redis.PhysicalConnection.ReadFromPipe() in /_/src/StackExchange.Redis/PhysicalConnection.cs:line 1850

When this happens its alway after we have called IDatabase.StreamAddAsync.

Looking at PhysicalConnection I'm wondering if there is a race condition between ReadFromPipe and something calling Dispose?
image

Does that make sense? I'm pretty sure this class is used by multiple threads and so I think the above is posslbe. Looking at the exception handler it seems like we don't even really want to report this exception?
image

So to sum up:

  • Is this error happening due to the described race condition?
  • Do we even want to report this exception up to the caller?

We have seen this happen on multiple OS, dotnet, and package versions:

  • Ubuntu 20.04 and 22.04
  • Dotnet 7.0.17 and 8.0.3
  • StackExchange.Redis 2.7.20 and 2.7.33

Let me know if you need more info.

Thanks!

@NickCraver
Copy link
Collaborator

It sounds like the connection is being used post-disposal here (at which point we shouldn't be sending any more commands) - is that accurate?

@nathanmascitelli
Copy link
Author

@NickCraver so the application has one instance of IConnectionMultiplexer which is used by multiple threads. At some point one of the threads will dispose the connection (either the app is shutting down or some configruation has changed) yes. When that happens any other threads thats currently trying to use the IConnectionMultiplexer gets this error, always when trying to write a value to Redis.

In the screenshot above it looks like we actualy handle this OK when reading but if isReading is false then we will always report an exception. Maybe we just need to change that part?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants