Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't find bundle for base name sun.text.resources.cldr.FormatData, locale #380

Open
darabos opened this issue Apr 7, 2023 · 4 comments
Open

Can't find bundle for base name sun.text.resources.cldr.FormatData, locale #380

darabos opened this issue Apr 7, 2023 · 4 comments
Assignees
@darabos

Comments

@darabos
Copy link
Contributor

darabos commented Apr 7, 2023

We sometimes hit this issue after a LynxKite startup. Restarting LynxKite fixes it. So something seems to go wrong during the startup that puts the system in a broken state. The stack trace is:

io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug!
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at com.lynxanalytics.biggraph.graph_api.proto.SphynxGrpc$SphynxBlockingStub.hasInSphynxMemory(SphynxGrpc.java:690)
at com.lynxanalytics.biggraph.graph_api.SphynxClient.hasInSphynxMemory(SphynxClient.scala:128)
at com.lynxanalytics.biggraph.graph_api.SphynxMemory.has(SphynxDomain.scala:37)
at com.lynxanalytics.biggraph.graph_api.DataManager.$anonfun$whoHas$1(DataManager.scala:110)
at com.lynxanalytics.biggraph.graph_api.DataManager.$anonfun$whoHas$1$adapted(DataManager.scala:110)
at scala.collection.IndexedSeqOptimized.$anonfun$find$1(IndexedSeqOptimized.scala:53)
at scala.collection.IndexedSeqOptimized.$anonfun$find$1$adapted(IndexedSeqOptimized.scala:53)
at scala.collection.IndexedSeqOptimized.segmentLength(IndexedSeqOptimized.scala:198)
at scala.collection.IndexedSeqOptimized.segmentLength$(IndexedSeqOptimized.scala:195)
at scala.collection.mutable.WrappedArray.segmentLength(WrappedArray.scala:38)
at scala.collection.GenSeqLike.prefixLength(GenSeqLike.scala:98)
at scala.collection.GenSeqLike.prefixLength$(GenSeqLike.scala:98)
at scala.collection.AbstractSeq.prefixLength(Seq.scala:45)
at scala.collection.IndexedSeqOptimized.find(IndexedSeqOptimized.scala:53)
at scala.collection.IndexedSeqOptimized.find$(IndexedSeqOptimized.scala:52)
at scala.collection.mutable.WrappedArray.find(WrappedArray.scala:38)
at com.lynxanalytics.biggraph.graph_api.DataManager.whoHas(DataManager.scala:110)
at com.lynxanalytics.biggraph.graph_api.DataManager.getFuture(DataManager.scala:126)
at com.lynxanalytics.biggraph.graph_api.DataManager.get(DataManager.scala:136)
at com.lynxanalytics.biggraph.graph_api.Scripting$EasyScalarContainer.value(Scripting.scala:68)
at com.lynxanalytics.biggraph.controllers.KiteMonitorThread.kiteCoreWorks(SparkClusterController.scala:248)
at com.lynxanalytics.biggraph.controllers.KiteMonitorThread.$anonfun$run$4(SparkClusterController.scala:305)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.util.MissingResourceException: Can't find bundle for base name sun.text.resources.cldr.FormatData, locale
at java.base/sun.util.resources.Bundles.throwMissingResourceException(Bundles.java:290)
at java.base/sun.util.resources.Bundles.loadBundleOf(Bundles.java:147)
at java.base/sun.util.resources.Bundles.of(Bundles.java:106)
at java.base/sun.util.resources.LocaleData$1.run(LocaleData.java:185)
at java.base/sun.util.resources.LocaleData$1.run(LocaleData.java:182)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.util.resources.LocaleData.getBundle(LocaleData.java:182)
at java.base/sun.util.resources.LocaleData.getNumberFormatData(LocaleData.java:178)
at java.base/sun.util.locale.provider.LocaleResources.getDecimalFormatSymbolsData(LocaleResources.java:176)
at java.base/java.text.DecimalFormatSymbols.initialize(DecimalFormatSymbols.java:628)
at java.base/java.text.DecimalFormatSymbols.<init>(DecimalFormatSymbols.java:113)
at java.base/sun.util.locale.provider.DecimalFormatSymbolsProviderImpl.getInstance(DecimalFormatSymbolsProviderImpl.java:85)
at java.base/java.text.DecimalFormatSymbols.getInstance(DecimalFormatSymbols.java:180)
at java.base/java.util.Formatter.getZero(Formatter.java:2437)
at java.base/java.util.Formatter.<init>(Formatter.java:1956)
at java.base/java.util.Formatter.<init>(Formatter.java:2011)
at java.base/java.lang.String.format(String.java:2938)
at lynxkite_shaded.com.google.common.util.concurrent.ThreadFactoryBuilder.format(ThreadFactoryBuilder.java:186)
at lynxkite_shaded.com.google.common.util.concurrent.ThreadFactoryBuilder.access$000(ThreadFactoryBuilder.java:51)
at lynxkite_shaded.com.google.common.util.concurrent.ThreadFactoryBuilder$1.newThread(ThreadFactoryBuilder.java:169)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:912)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354)
at io.grpc.internal.ManagedChannelImpl$ExecutorHolder.execute(ManagedChannelImpl.java:2240)
at io.grpc.internal.DnsNameResolver.resolve(DnsNameResolver.java:389)
at io.grpc.internal.DnsNameResolver.refresh(DnsNameResolver.java:212)
at io.grpc.internal.ManagedChannelImpl.refreshNameResolution(ManagedChannelImpl.java:492)
at io.grpc.internal.ManagedChannelImpl.refreshAndResetNameResolution(ManagedChannelImpl.java:486)
at io.grpc.internal.ManagedChannelImpl.access$5000(ManagedChannelImpl.java:118)
at io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1LoadBalancerRefreshNameResolution.run(ManagedChannelImpl.java:1499)
at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95)
at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:127)
at io.grpc.internal.InternalSubchannel$TransportListener.transportShutdown(InternalSubchannel.java:575)
at io.grpc.netty.shaded.io.grpc.netty.ClientTransportLifecycleManager.notifyGracefulShutdown(ClientTransportLifecycleManager.java:56)
at io.grpc.netty.shaded.io.grpc.netty.ClientTransportLifecycleManager.notifyShutdown(ClientTransportLifecycleManager.java:62)
at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onConnectionError(NettyClientHandler.java:500)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:641)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:380)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
@darabos
Copy link
Contributor Author

darabos commented Apr 11, 2023

A shorter stack trace I got now:

java.util.MissingResourceException: Can't find bundle for base name sun.text.resources.cldr.FormatData, locale en
	at java.base/sun.util.resources.Bundles.throwMissingResourceException(Bundles.java:290)
	at java.base/sun.util.resources.Bundles.loadBundleOf(Bundles.java:147)
	at java.base/sun.util.resources.Bundles.of(Bundles.java:106)
	at java.base/sun.util.resources.LocaleData$1.run(LocaleData.java:185)
	at java.base/sun.util.resources.LocaleData$1.run(LocaleData.java:182)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.util.resources.LocaleData.getBundle(LocaleData.java:182)
	at java.base/sun.util.resources.LocaleData.getDateFormatData(LocaleData.java:146)
	at java.base/java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:745)
	at java.base/java.text.DateFormatSymbols.<init>(DateFormatSymbols.java:151)
	at java.base/sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(DateFormatSymbolsProviderImpl.java:85)
	at java.base/java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:370)
	at java.base/java.text.DateFormatSymbols.getInstanceRef(DateFormatSymbols.java:360)
	at java.base/java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:622)
	at play.core.cookie.encoding.HttpHeaderDateFormat.<init>(HttpHeaderDateFormat.java:53)
	at play.core.cookie.encoding.HttpHeaderDateFormat.<init>(HttpHeaderDateFormat.java:33)
	at play.core.cookie.encoding.HttpHeaderDateFormat$1.initialValue(HttpHeaderDateFormat.java:43)
	at play.core.cookie.encoding.HttpHeaderDateFormat$1.initialValue(HttpHeaderDateFormat.java:40)
	at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
	at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
	at play.core.cookie.encoding.HttpHeaderDateFormat.get(HttpHeaderDateFormat.java:48)
	at play.core.cookie.encoding.ServerCookieEncoder.encode(ServerCookieEncoder.java:87)
	at play.api.mvc.CookieHeaderEncoding.$anonfun$encodeSetCookieHeader$1(Cookie.scala:273)
	at scala.collection.immutable.List.map(List.scala:293)
	at play.api.mvc.CookieHeaderEncoding.encodeSetCookieHeader(Cookie.scala:264)
	at play.api.mvc.CookieHeaderEncoding.encodeSetCookieHeader$(Cookie.scala:262)
	at play.api.mvc.DefaultCookieHeaderEncoding.encodeSetCookieHeader(Cookie.scala:372)
	at play.api.mvc.Result.bakeCookies(Results.scala:367)
	at play.core.server.common.ServerResultUtils.prepareCookies(ServerResultUtils.scala:291)
	at play.core.server.AkkaHttpServer.$anonfun$runAction$5(AkkaHttpServer.scala:444)
	at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

And now I'm not sure it's a startup problem. Restarting still fixes it, but I think I saw it develop without a restart.

https://stackoverflow.com/questions/12986234/cant-find-bundle-for-base-name-bundle-locale-en-us says it's due to a missing Bundle.properties file. I checked and we don't have this file. But the file is missing all the time. Why do we only get this error sometimes?

@darabos
Copy link
Contributor Author

darabos commented Apr 11, 2023

Oh, the error in the Stack Overflow answer is complaining about /Bundle so that's why they talk about Bundle.properties. For us it's not /Bundle but sun.text.resources.cldr.FormatData. For that the only hit on Google is this issue. 😭 We're on our own.

This file is part of the JRE. I can see sun/text/resources/cldr/FormatData_en.class in jvm/lib/modules. It looks like at some point we end up not having the JRE on the classpath? Spark does a lot of tricks with the classpath. Maybe something goes wrong?

@darabos darabos self-assigned this Apr 24, 2023
@darabos
Copy link
Contributor Author

darabos commented Apr 24, 2023

Logs leading up to the exception: https://gist.github.com/darabos/9d0d286da9cdb6d47a04e73236a813a9 It has a Kite core test timed out logs with full thread dump right before the FormatData stack trace! Plus we can't load FormatData because:

Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
	at java.base/java.security.AccessController.checkPermission(AccessController.java:897)
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
	at com.lynxanalytics.biggraph.scala_sandbox.ScalaScriptSecurityManager.checkPermission(ScalaScript.scala:75)

We're denying it! What sort of thread is this? Our security manager is only supposed to affect Derive attribute boxes, not HTTP serving threads!

@darabos darabos mentioned this issue Apr 24, 2023
Merged
@darabos
Copy link
Contributor Author

darabos commented Apr 24, 2023

I think we deny the call when it happens inside restricted user code. I'm not sure why this happens in the first place. For a long time it doesn't happen then it happens. But once it happened, I think Java caches that this bundle does not exist (Bundles.java#221), so other threads will be impacted too and everything breaks.

For some instances the safest option is to disable our SecurityManager. I've sent #399 to make that possible. For other instances we will need to understand this a bit better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@darabos darabos

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@darabos