Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log non-serialisable exception before it is discarded #26323

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tommyk-gears
Copy link
Contributor

If serialisation of an exception fails, then produce a log message including the exception, before it is discarded. This will help developers find the source of the non-serialisable exception.

Fixes #26322

Breaking changes (list specific methods/types/messages): No breaking changes

Checklist:

  • Labels (Team:, Type:, Source:, Module:) and Milestone set
  • Add Add to Release Notes label if changes should be mentioned in release notes or Not Release Notes content if changes are not relevant for release notes
  • Request reviewers if possible
  • New public APIs have @Nonnull/@Nullable annotations
  • New public APIs have @since tags in Javadoc
  • Send backports/forwardports if fix needs to be applied to past/future releases

@hz-devops-test hz-devops-test added the Source: Community PR or issue was opened by a community user label May 3, 2024
@devOpsHazelcast
Copy link
Collaborator

Can one of the admins verify this patch?

2 similar comments
@devOpsHazelcast
Copy link
Collaborator

Can one of the admins verify this patch?

@devOpsHazelcast
Copy link
Collaborator

Can one of the admins verify this patch?

@devOpsHazelcast
Copy link
Collaborator

Internal PR hazelcast/hazelcast-mono#1780
Internal message only. Nothing to see here, move along

Comment on lines +136 to +137
if (rootObject instanceof Throwable throwable) {
LOGGER.warning("Failed to serialize '" + clazz + "'. It will be logged here and "
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
if (rootObject instanceof Throwable throwable) {
LOGGER.warning("Failed to serialize '" + clazz + "'. It will be logged here and "
if (rootObject instanceof Throwable throwable) {
LOGGER.warning("Failed to serialize throwable '" + clazz + "'. It will be logged here and "

Copy link
Member

@srknzl srknzl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this change makes sense, i am curious to hear others if any other solution is possible to make an exception easy to track that's failed to be serialized

@TomaszGaweda
Copy link
Contributor

I think there is no other way; we can't add them to supressed exceptions, because those will be also serialized (and we already know, that we can't serialize it). Logging makes most sense.

@srknzl
Copy link
Member

srknzl commented May 15, 2024

run-lab-run

@hz-devops-test
Copy link

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file
--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.4.1:enforce (enforce-tools) on project hazelcast-root: 
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.4.1:enforce (enforce-tools) on project hazelcast-root: 
--------------------------
[ERROR] Rule 0: org.apache.maven.enforcer.rules.version.RequireJavaVersion failed with message:
--------------------------
[ERROR] Detected JDK version 11.0.18 (JAVA_HOME=/usr/share/jdks/oracle-11) is not in the allowed range [17,).
--------------------------
[ERROR] -> [Help 1]
--------------------------
[ERROR] 
--------------------------
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
--------------------------
[ERROR] 
--------------------------
[ERROR] For more information about the errors and possible solutions, please read the following articles:
--------------------------
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
--------------------------
[ERROR] 
--------------------------
[ERROR] After correcting the problems, you can resume the build with the command
--------------------------
[ERROR]   mvn  -rf :hazelcast-root
--------------------------

@JackPGreen
Copy link
Contributor

If the real issue is that the cause of the original exception is lost, perhaps it'd be better to add the exceptions' message to the new HazelcastSerializationException?

Currently, something else may catch the HazelcastSerializationException because it's actually expected (obvious case is a test, but there are others that may apply) but with this there'd be extra logging in place that can't be easily disabled.

@srknzl
Copy link
Member

srknzl commented May 20, 2024

Currently, something else may catch the HazelcastSerializationException

Can you give an example? The new log will only run when there is a Throwable that could not be serialized.

Also I think only message of the exception is not enough, because the author wants to track the exception. So I think stacktrace is important for that.

but with this there'd be extra logging in place that can't be easily disabled.

As much as I don't like a lot of new configuration, we can put this behind a property that would be disabled by default.

@JackPGreen
Copy link
Contributor

Currently, something else may catch the HazelcastSerializationException

Can you give an example? The new log will only run when there is a Throwable that could not be serialized.

Two (obscure) cases come to mind:

  • You have a unit test that tests serializing the unserializable. The output of the test will be full of [ERROR]s that are actually expected for that test
  • You are testing an unknown datastream against possible serialization formats, you try a couple, expecting all-but-one to fail

Also I think only message of the exception is not enough, because the author wants to track the exception. So I think stacktrace is important for that.

I'm sure there's a textual representation we could agree on that would have that information - effectively here were saying - if you can't serialize the exception over-the-wire, serialize to a file instead (in the form of a log message).

@tommyk-gears
Copy link
Contributor Author

but with this there'd be extra logging in place that can't be easily disabled.

As much as I don't like a lot of new configuration, we can put this behind a property that would be disabled by default.

What if we log this to a dedicated logger (e.g. com.hazelcast.internal.serialization.impl.SerializationUtil.NonSerializableThrowable)? Then the logging of this specific message can be easily disabled (it kind of already can, since this is the only message logged on the com.hazelcast.internal.serialization.impl.SerializationUtil logger, but that may change in the future).

@srknzl
Copy link
Member

srknzl commented May 21, 2024

Two (obscure) cases come to mind:

You have a unit test that tests serializing the unserializable. The output of the test will be full of [ERROR]s that are actually > expected for that test
You are testing an unknown datastream against possible serialization formats, you try a couple, expecting all-but-one to fail

Note that these will log only if the object is a Throwable, which is not very likely imo.

What if we log this to a dedicated logger (e.g. com.hazelcast.internal.serialization.impl.SerializationUtil.NonSerializableThrowable)? Then the logging of this specific message can be easily disabled (it kind of already can, since this is the only message logged on the com.hazelcast.internal.serialization.impl.SerializationUtil logger, but that may change in the future).

Can be possible but I think Jack does not like the default behaviour of logging those throwables that can't be serialized. Adding dedicated logger will still log them by default.

Copy link
Contributor

@ihsandemir ihsandemir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: The problem is about at the client side not seeing exception details, right? We actually put the stack trace information into the error codec, and it was deserialized and set at the client exception factory while exception is created. Was this not enough?

If you also want to communicate the details of the object that can not be serialized and if it is of type Throwable, the we can set this object as the cause for the constructed HazelcastSerializationException and if there is any other exception we throw during the serialization. At the end, user will see top level HazelcastSerializationException`, a cause and the cause of the cause which is the throwable object. Then, you do not need to do logging, does this make sense?

@srknzl
Copy link
Member

srknzl commented May 23, 2024

Question: The problem is about at the client side not seeing exception details, right? We actually put the stack trace information into the error codec, and it was deserialized and set at the client exception factory while exception is created. Was this not enough?

nope not enough. We don't put any other info than the class name about the object to be serialized's (if it's Throwable) in to HazelcastSerializationException's message.

If you also want to communicate the details of the object that can not be serialized and if it is of type Throwable, the we can set this object as the cause for the constructed HazelcastSerializationException and if there is any other exception we throw during the serialization. At the end, user will see top level HazelcastSerializationException`, a cause and the cause of the cause which is the throwable object. Then, you do not need to do logging, does this make sense?

this seems ok to me, however what if it already has a cause? Also it's not the cause of serialization exception actually, but the object tried to be serialized. We need to make this clear

@ihsandemir
Copy link
Contributor

Question: The problem is about at the client side not seeing exception details, right? We actually put the stack trace information into the error codec, and it was deserialized and set at the client exception factory while exception is created. Was this not enough?

nope not enough. We don't put any other info than the class name about the object to be serialized's (if it's Throwable) in to HazelcastSerializationException's message.

If you also want to communicate the details of the object that can not be serialized and if it is of type Throwable, the we can set this object as the cause for the constructed HazelcastSerializationException and if there is any other exception we throw during the serialization. At the end, user will see top level HazelcastSerializationException`, a cause and the cause of the cause which is the throwable object. Then, you do not need to do logging, does this make sense?

this seems ok to me, however what if it already has a cause? Also it's not the cause of serialization exception actually, but the object tried to be serialized. We need to make this clear

The exception is created during serialization by us, hence we control the cause part of the exception and we can check the places we throw during serialization and arrange the cause properly to include this object.

Copy link
Member

@srknzl srknzl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Staling my review until we understand the use case. We believe Throwables should never be returned from our APIs as responses.

So @tommyk-gears, can you please share your executor code reproducer that tries to serialize a Throwable and fails to do so? We think you don't need this change in this PR, but can simply do one of the following:

  1. If there is an error, just throw it in the executor service's task, instead of returning it as a response. It'll skip serialization and be returned to the user via ErrorCodecs
  2. If you really want to log it in the member side, you can add try/catch in the executable task in ExecutorService and log it in the catch() block.

@ihsandemir
Copy link
Contributor

ihsandemir commented May 24, 2024

Discussed with @srknzl and we concluded that why we need to try serialize a throwable in the first place? Normally, in a callable, you either return a real non-throwable object or you throw an exception. Hence, why is the throwable used as the return value of a callable?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Log exceptions that cannot be serialized
7 participants