Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hangup in Java11HttpTransport #3540

Open
basilevs opened this issue Feb 29, 2024 · 2 comments
Open

Hangup in Java11HttpTransport #3540

basilevs opened this issue Feb 29, 2024 · 2 comments

Comments

@basilevs
Copy link
Contributor

basilevs commented Feb 29, 2024 •
edited

When running the build with --threads 2C, I've got an indefinite hangup on a wake up from a sleep state.

Environment:
Tycho 4.0.4
Mac Book Pro 2021

% mvn --version
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /opt/homebrew/Cellar/maven/3.9.6/libexec
Java version: 21.0.1, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk/21.0.1/libexec/openjdk.jdk/Contents/Home
Default locale: en_GE, platform encoding: UTF-8
OS name: "mac os x", version: "12.7.1", arch: "aarch64", family: "mac"

Filtered jstack:

"mvn-builder-com.fnfr.svt.help" #73 [48899] prio=5 os_prio=31 cpu=735.78ms elapsed=8257.91s tid=0x0000000145330800 nid=48899 waiting on condition  [0x000000029c79c000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21.0.1/Native Method)
	- parking to wait for  <0x00000007fb372ae8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.1/LockSupport.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.1/AbstractQueuedSynchronizer.java:519)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.1/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.1/ForkJoinPool.java:3725)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.1/AbstractQueuedSynchronizer.java:1707)
	at java.util.concurrent.ArrayBlockingQueue.take(java.base@21.0.1/ArrayBlockingQueue.java:420)
	at jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.current(java.net.http@21.0.1/ResponseSubscribers.java:458)
	at jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(java.net.http@21.0.1/ResponseSubscribers.java:508)
	at java.io.InputStream.transferTo(java.base@21.0.1/InputStream.java:796)
	at org.eclipse.tycho.p2maven.transport.SharedHttpCacheStorage$CacheLine.fetchFile(SharedHttpCacheStorage.java:286)
	- locked <0x00000007fb376b78> (a org.eclipse.tycho.p2maven.transport.SharedHttpCacheStorage$CacheLine)
	at org.eclipse.tycho.p2maven.transport.SharedHttpCacheStorage$2.getCacheFile(SharedHttpCacheStorage.java:133)
	at org.eclipse.tycho.p2maven.transport.HttpTransportProtocolHandler.getFile(HttpTransportProtocolHandler.java:50)
	at org.eclipse.tycho.p2maven.transport.TychoRepositoryTransport.stream(TychoRepositoryTransport.java:144)
	at org.eclipse.tycho.p2maven.transport.TychoRepositoryTransport.download(TychoRepositoryTransport.java:105)
	at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.downloadArtifact(SimpleArtifactRepository.java:734)
	at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.downloadArtifact(SimpleArtifactRepository.java:656)
	at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.getArtifact(SimpleArtifactRepository.java:793)
	at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider.getArtifactFromOneMirror(RepositoryArtifactProvider.java:243)
	at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider.getArtifactFromAnyMirror(RepositoryArtifactProvider.java:223)
	at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider$1.perform(RepositoryArtifactProvider.java:198)
	at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.getArtifact(SimpleArtifactRepository.java:775)
	at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.getArtifacts(SimpleArtifactRepository.java:855)
	at org.eclipse.tycho.p2.repository.LazyArtifactRepository.getArtifacts(LazyArtifactRepository.java:102)
	at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider.getArtifactFromAnyChildRepository(RepositoryArtifactProvider.java:210)
	at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider.getArtifactFromAnyFormatAvailableInRepository(RepositoryArtifactProvider.java:177)
	at org.eclipse.tycho.p2.repository.RepositoryArtifactProvider.getArtifactFromAnySource(RepositoryArtifactProvider.java:163)
	at org.eclipse.tycho.p2.repository.CompositeArtifactProviderBaseImpl.getArtifact(CompositeArtifactProviderBaseImpl.java:55)
	at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.downloadCanonicalArtifact(MirroringArtifactProvider.java:339)
	at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.downloadMostSpecificNeededFormatOfArtifact(MirroringArtifactProvider.java:332)
	at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.downloadArtifact(MirroringArtifactProvider.java:317)
	at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.makeOneFormatLocallyAvailable(MirroringArtifactProvider.java:256)
	at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.makeLocallyAvailable(MirroringArtifactProvider.java:206)
	at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.getArtifactFile(MirroringArtifactProvider.java:137)
	at org.eclipse.tycho.p2.repository.CompositeArtifactProvider.getArtifactFile(CompositeArtifactProvider.java:89)
	at org.eclipse.tycho.p2resolver.TargetPlatformBaseImpl.getLocalArtifactFile(TargetPlatformBaseImpl.java:169)
	at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResult.lambda$addArtifact$2(DefaultP2ResolutionResult.java:90)
	at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResult$$Lambda/0x000000b8017b37c0.get(Unknown Source)
	at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResultEntry.getLocation(DefaultP2ResolutionResultEntry.java:73)
	at org.eclipse.tycho.p2resolver.P2DependencyResolver.lambda$newDefaultTargetPlatform$6(P2DependencyResolver.java:383)
	at org.eclipse.tycho.p2resolver.P2DependencyResolver$$Lambda/0x000000b8017b7988.get(Unknown Source)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifactFile$2(ArtifactCollection.java:69)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection$$Lambda/0x000000b8017b7ba8.apply(Unknown Source)
	at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation(DefaultArtifactDescriptor.java:72)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3(ArtifactCollection.java:148)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection$$Lambda/0x000000b8017b8240.apply(Unknown Source)
	at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation(DefaultArtifactDescriptor.java:72)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3(ArtifactCollection.java:148)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection$$Lambda/0x000000b8017b8240.apply(Unknown Source)
	at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation(DefaultArtifactDescriptor.java:72)
	at org.eclipse.tycho.core.maven.MavenDependencyInjector.collectExternalDependencies(MavenDependencyInjector.java:216)
	at org.eclipse.tycho.core.maven.MavenDependencyInjector.addDependency(MavenDependencyInjector.java:180)
	at org.eclipse.tycho.core.maven.MavenDependencyInjector.injectMavenDependencies(MavenDependencyInjector.java:82)
	at org.eclipse.tycho.p2resolver.P2DependencyResolver.injectDependenciesIntoMavenModel(P2DependencyResolver.java:409)
	at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject(DefaultTychoResolver.java:155)
	- locked <0x0000000601c8a658> (a org.eclipse.tycho.core.osgitools.DefaultReactorProject)
	at org.eclipse.tycho.core.maven.TychoProjectExecutionListener.beforeProjectLifecycleExecution(TychoProjectExecutionListener.java:108)
	at org.apache.maven.lifecycle.internal.CompoundProjectExecutionListener.beforeProjectLifecycleExecution(CompoundProjectExecutionListener.java:42)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:103)
	at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:193)
	at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:180)
	at java.util.concurrent.FutureTask.run(java.base@21.0.1/FutureTask.java:317)
	at java.util.concurrent.Executors$RunnableAdapter.call(java.base@21.0.1/Executors.java:572)
	at java.util.concurrent.FutureTask.run(java.base@21.0.1/FutureTask.java:317)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.1/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.1/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith(java.base@21.0.1/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.1/Thread.java:1583)

   Locked ownable synchronizers:
	- <0x0000000608c5e5c0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	- <0x000000060a1bc7a8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"mvn-builder-com.spirent.itest.platform.chromium.ui" #74 [49155] prio=5 os_prio=31 cpu=740.06ms elapsed=8257.91s tid=0x0000000146964200 nid=49155 waiting on condition  [0x000000029c9a9000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21.0.1/Native Method)
	- parking to wait for  <0x0000000608c5e5c0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.1/LockSupport.java:221)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.1/AbstractQueuedSynchronizer.java:754)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.1/AbstractQueuedSynchronizer.java:990)
	at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@21.0.1/ReentrantLock.java:153)
	at java.util.concurrent.locks.ReentrantLock.lock(java.base@21.0.1/ReentrantLock.java:322)
	at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.makeOneFormatLocallyAvailable(MirroringArtifactProvider.java:235)
	at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.makeLocallyAvailable(MirroringArtifactProvider.java:206)
	at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.getArtifactFile(MirroringArtifactProvider.java:137)
	at org.eclipse.tycho.p2.repository.CompositeArtifactProvider.getArtifactFile(CompositeArtifactProvider.java:89)
	at org.eclipse.tycho.p2resolver.TargetPlatformBaseImpl.getLocalArtifactFile(TargetPlatformBaseImpl.java:169)
	at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResult.lambda$addArtifact$2(DefaultP2ResolutionResult.java:90)
	at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResult$$Lambda/0x000000b8017b37c0.get(Unknown Source)
	at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResultEntry.getLocation(DefaultP2ResolutionResultEntry.java:73)
	at org.eclipse.tycho.p2resolver.P2DependencyResolver.lambda$newDefaultTargetPlatform$6(P2DependencyResolver.java:383)
	at org.eclipse.tycho.p2resolver.P2DependencyResolver$$Lambda/0x000000b8017b7988.get(Unknown Source)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifactFile$2(ArtifactCollection.java:69)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection$$Lambda/0x000000b8017b7ba8.apply(Unknown Source)
	at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation(DefaultArtifactDescriptor.java:72)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3(ArtifactCollection.java:148)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection$$Lambda/0x000000b8017b8240.apply(Unknown Source)
	at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation(DefaultArtifactDescriptor.java:72)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3(ArtifactCollection.java:148)
	at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection$$Lambda/0x000000b8017b8240.apply(Unknown Source)
	at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation(DefaultArtifactDescriptor.java:72)
	at org.eclipse.tycho.core.maven.MavenDependencyInjector.collectExternalDependencies(MavenDependencyInjector.java:216)
	at org.eclipse.tycho.core.maven.MavenDependencyInjector.addDependency(MavenDependencyInjector.java:180)
	at org.eclipse.tycho.core.maven.MavenDependencyInjector.injectMavenDependencies(MavenDependencyInjector.java:82)
	at org.eclipse.tycho.p2resolver.P2DependencyResolver.injectDependenciesIntoMavenModel(P2DependencyResolver.java:409)
	at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject(DefaultTychoResolver.java:155)
	- locked <0x0000000601c833a8> (a org.eclipse.tycho.core.osgitools.DefaultReactorProject)
	at org.eclipse.tycho.core.maven.TychoProjectExecutionListener.beforeProjectLifecycleExecution(TychoProjectExecutionListener.java:108)
	at org.apache.maven.lifecycle.internal.CompoundProjectExecutionListener.beforeProjectLifecycleExecution(CompoundProjectExecutionListener.java:42)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:103)
	at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:193)
	at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:180)
	at java.util.concurrent.FutureTask.run(java.base@21.0.1/FutureTask.java:317)
	at java.util.concurrent.Executors$RunnableAdapter.call(java.base@21.0.1/Executors.java:572)
	at java.util.concurrent.FutureTask.run(java.base@21.0.1/FutureTask.java:317)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.1/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.1/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith(java.base@21.0.1/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.1/Thread.java:1583)

   Locked ownable synchronizers:
	- <0x000000060a1075c0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Full jstack

I've captured the heapdump and:

  • the blocked ArrayBlockingQueue is referenced from Java11HttpTransportFactory.
  • the response body is not closed
  • response is referenced from jdk.internal.net.http.HttpClientImpl$SelectorManager#2 [GC root - Java frame, thread object] : HttpClient-1-SelectorManager

Overall this looks like a bug in JDK, but I can't say I understand Java11HttpTransport enough to eliminate a bug there. The symptoms match a failure to dispose a resource after a fatal error.

I see an unrelated(?) exception in logs:

java.util.zip.ZipException: zip file is empty
	at java.base/java.util.zip.ZipFile$Source.zerror(ZipFile.java:1781)
	at java.base/java.util.zip.ZipFile$Source.findEND(ZipFile.java:1562)
	at java.base/java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1657)
	at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1495)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1458)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:724)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:251)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:180)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:194)
	at org.eclipse.jdt.internal.compiler.batch.ClasspathJar.initialize(ClasspathJar.java:204)
	at org.eclipse.jdt.internal.compiler.batch.ClasspathMultiReleaseJar.initialize(ClasspathMultiReleaseJar.java:38)
	at org.eclipse.jdt.internal.compiler.batch.FileSystem.<init>(FileSystem.java:235)
	at org.eclipse.jdt.internal.compiler.batch.Main.getLibraryAccess(Main.java:3501)
	at org.eclipse.jdt.internal.compiler.batch.Main.performCompilation(Main.java:4747)
	at org.eclipse.jdt.internal.compiler.batch.Main.compile(Main.java:1802)
	at org.eclipse.tycho.compiler.jdt.JDTCompiler.compileInProcess(JDTCompiler.java:419)
	at org.eclipse.tycho.compiler.jdt.JDTCompiler.performCompile(JDTCompiler.java:124)
	at copied.org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:414)
	at org.eclipse.tycho.compiler.AbstractOsgiCompilerMojo.doCompile(AbstractOsgiCompilerMojo.java:404)
	at org.eclipse.tycho.compiler.AbstractOsgiCompilerMojo.execute(AbstractOsgiCompilerMojo.java:385)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174)
	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75)
	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162)
	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105)
	at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:193)
	at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:180)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
@laeubi
Copy link
Member

laeubi commented Mar 1, 2024

Tycho 4.0.4

Have you tried Tycho 4.0.5 or Tycho 4.0.6?

@basilevs
Copy link
Contributor Author

basilevs commented Mar 1, 2024

Tycho 4.0.4

Have you tried Tycho 4.0.5 or Tycho 4.0.6?

No.

basilevs referenced this issue Mar 1, 2024
@laeubi
Improve GOAWAY handling
7a5858b 
Currently GOAWAY is detected/handled only in the connection phase but
sometimes can happen in the transfer phase as well.

This now wraps the whole request inside a block that is retried if a
GOAWAY is detected.

(cherry picked from commit deb31c7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@basilevs @laeubi