Embulk execution sometimes fails and is not reproducible

[Tobitawara]

Environment

• OS version: CentOS 7
• Java version: 1.8.0_212
• Embulk version: 0.9.17
• My Embulk configuration (YAML):

in:
  type: mysql
  host: <<host>>
  port: <<port>>
  user: <<user>>
  password: <<pass>>
  database: <<db>>
  table: <<table>>
  incremental: true
  incremental_columns: [id]
  options: {useUnicode: true, characterEncoding: utf8}

out:
  type: mysql
  host: <<host>>
  port: <<port>>
  user: <<user>>
  password: <<pass>>
  database: <<db>>
  table: <<table>>
  mode: insert
  options: {useUnicode: true, characterEncoding: utf8}

• Plugin versions:
  • embulk-input-mysql (0.10.0)
  • embulk-output-mysql (0.8.5)
  • liquid (4.0.0)

Bug Report

I regularly run the following command.
$ embulk <<conf file>> -r <<resume file>> -c <<diff file>>

Most of them succeed, but sometimes (about 1/100) it fails with the following error.
When I rerun immediately after the error it succeeds and is not reproducible.
What's happening?

973-2020-10-14 10:05:06.191 +0900: Embulk v0.9.17
974-2020-10-14 10:05:06.752 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
975-2020-10-14 10:05:09.041 +0900 [INFO] (main): Gem's home and path are set by default: "/home/tblapp/.embulk/lib/gems"
976:LoadError: load error: liquid/interrupts -- java.lang.NullPointerException: Inflater has been closed
977-  require at org/jruby/RubyKernel.java:955
978-  require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55
979-   <main> at uri:classloader:/gems/liquid-4.0.0/lib/liquid.rb:56
980-  require at org/jruby/RubyKernel.java:955
981-   (root) at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
982-   <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:133
983:org.jruby.embed.EvalFailedException: (LoadError) load error: liquid/interrupts -- java.lang.NullPointerException: Inflater has been closed
984-	at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:131)
985-	at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1307)
986-	at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1300)
987-	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
988-	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
989-	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
990-	at java.lang.reflect.Method.invoke(Method.java:498)
991-	at org.embulk.jruby.ScriptingContainerDelegateImpl.runScriptlet(ScriptingContainerDelegateImpl.java:802)
992-	at org.embulk.EmbulkRunner.runLiquid(EmbulkRunner.java:385)
993-	at org.embulk.EmbulkRunner.readConfig(EmbulkRunner.java:353)
994-	at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:259)
995-	at org.embulk.EmbulkRunner.run(EmbulkRunner.java:156)
996-	at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:433)
997-	at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
998-	at org.embulk.cli.Main.main(Main.java:64)
999:Caused by: org.jruby.exceptions.RaiseException: (LoadError) load error: liquid/interrupts -- java.lang.NullPointerException: Inflater has been closed
1000-	at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
1001-	at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55)
1002-	at RUBY.<main>(uri:classloader:/gems/liquid-4.0.0/lib/liquid.rb:56)
1003-	at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
1004-	at RUBY.(root)(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1)
1005-	at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:133)
1006-
1007:Error: (LoadError) load error: liquid/interrupts -- java.lang.NullPointerException: Inflater has been closed

[dmikurube]

I'm afraid of this: https://bugs.openjdk.java.net/browse/JDK-8246714 which has impacted JRuby by itself. JRuby needs to be updated to workaround this JDK-8246714 (if it's the cause of your issue), but we've postponed upgrading JRuby for a long time because updating JRuby has caused other compatibility problems so far.

Since v0.11 (on the middle way of development versions v0.10.*), JRuby will be optional. Users can (have to) specify their own version of JRuby package out of Embulk like:

embulk run -X jruby=file:///home/user/jruby-complete-9.X.Y.Z.jar example.yml

It means that choosing JRuby version is on user's responsibility, but using a later version of JRuby may solve this problem. We are just now working on it (#1336).

[Tobitawara]

Thank you for your reply.
I will try the jruby upgrade.

[dmikurube]

It will happen in the (near) future. It has not happened yet, and the fixed version 9.1.15.0 of JRuby is embedded in Embulk now -- so you cannot upgrade JRuby by yourself yet. Sorry about that, but we're working so that it happens.