-
Notifications
You must be signed in to change notification settings - Fork 8.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-19167 Bug Fix: Change of Codec configuration does not work #6807
Conversation
HDFS-17510 Change of Codec configuration does not work
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks @skyskyhu for your report.
💔 -1 overall
This message was automatically generated. |
@ZanderXu I do not have permission to merge my commit. Could you help merge the commit? Thanks! |
@steveloughran, @jojochuang, @Hexiaoqiao, Can you help review and merge the commit when you have free time~ Thanks a lot. |
@ZanderXu should have the right permissions |
@ZanderXu Could you help merge the commit when you have time? Thanks a lot |
Merged. Thanks @skyskyhu for your contribution. |
Thanks to @ZanderXu for your kind help! |
HADOOP-19167 Change of Codec configuration does not work
Description of PR
In one of my projects, I need to dynamically adjust compression level for different files.
However, I found that in most cases the new compression level does not take effect as expected, the old compression level continues to be used.
Here is the relevant code snippet:
Take my unit test as another example:
In the current code, the compression level of comp2 is 2, rather than the intended level of 3.
The reason is SequenceFile.Writer.init() method will call CodecPool.getCompressor(codec) to get a compressor, eventually CodecPool.getCompressor(codec, null) will be called.
If the compressor is a reused instance, the conf is not applied because it is passed as null:
Please also refer to my unit test to reproduce the bug.
To address this bug, I modified the code to ensure that the configuration is read back from the codec when a compressor is reused.
How was this patch tested?
unit test