Never ending loop "Competing reserved resource accessed via runSync"

[hicolour]

My use case is very easy, I'm using off-heap map to store some counters. Read and write is single threaded.

Very frequently my application is not functioning correctly because it cannot acces collection - the only sign that something is not working correctly is following log.

15:14:04.705 WARN  swaydb.IO$Defer - io-compute-10: Competing reserved resource accessed via runSync. Times accessed: 3843425. Reserve: Some(ClosedChannel)

This counter goes to the moon 3843425

Is there any way to set some configuration parameters to avoid such situations - maybe by disabling compaction etc

Because my use case is very simple - the only compeeting process could be some swaydb -internal compaction etc.

---

Bellow you can find the config that I'm currently using.

/*
    Specifies if key-value IDs should be cached. There are over 1300 key-value Ids, if caching is disabled then on disk binary-search is performed to find the IDs.
    https://swaydb.io/configuration/cacheKeyValueIds/?language=scala
   */
  val cacheKeyValueIdsOverride = false

  /*
    This enables caching raw bytes stored in Persistent Segment and disables caching of actual key-values
    https://swaydb.io/configuration/memoryCache/?language=scala
   */
  val memoryCacheOverride = MemoryCache.off

  /*
    Setting mmap to true in LevelZero will write key-values to memory-mapped write-ahead log files.
    If false, java.nio.FileChannel are used.
    https://swaydb.io/configuration/mmap/map/?language=scala
   */
  val mmapDisabled = MMAP.Off(
    ForceSave.Off
  )

  val threadStateCacheOverride = ThreadStateCache.Limit(hashMapMaxSize = 10, maxProbe = 1)
  //ThreadStateCache.off

  /*
    Configures all Persistent Segment file for the Level
    https://swaydb.io/configuration/segmentConfig/?language=scala
   */
  val segmentConfigOverride = DefaultConfigs
    .segmentConfig()
    .copyWithMmap(
      mmapDisabled
    )
    .copyWithCacheSegmentBlocksOnCreate(false)
    .copyWithFileOpenIOStrategy(IOStrategy.AsyncIO(cacheOnAccess = true))
    .copyWithBlockIOStrategy(
      blockIOStrategy = (_) => IOStrategy.AsyncIO(cacheOnAccess = false)
    )

  // Following configuration disable caching on the file read/write layer
  val fileCacheOverride = FileCache.On(
    0,
    ActorConfig.TimeLoop(
      name = s"${this.getClass.getName} - FileCache TimeLoop Actor",
      delay = 1.seconds,
      ec = DefaultExecutionContext.sweeperEC
    )
  )

  /*
    It stores all keys in sorted order.
    https://swaydb.io/configuration/sortedKeyIndex/?language=scala&q=sortedKeyIndex
   */
  val sortedKeyIndexOverride = DefaultConfigs.sortedKeyIndex(false)
  /*
    hashIndexes can double random read performance and reduce IOps which would increases overall DB performance.
    https://swaydb.io/configuration/randomKeyIndex/?language=scala
   */
  val randomSearchIndexOverride = DefaultConfigs.randomSearchIndex(false)
  /*
    It allows for faster read performance over sortedKeyIndex or linear-search for random reads and is essential for fast forward & reverse iterations.
    https://swaydb.io/configuration/binarySearchIndex/?language=scala
   */
  val binarySearchIndexOverride = DefaultConfigs.binarySearchIndex(false)
  /*
    BloomFilter is a small byte array that can be created in each persistent segment and is used to determine if a key exists in the segment without actually searching the Segment.
    https://swaydb.io/configuration/mightContainKeyIndex/?language=scal
   */
  val mightContainIndexOverride = DefaultConfigs.mightContainIndex(false)

  /*
    Configures storage for values within a persistent segment.
    https://swaydb.io/configuration/valuesConfig/?language=scala&q=mightContainIndex
   */
  val valuesConfigOverride = DefaultConfigs.valuesConfig(false)

  def mapCreate(name: String) =
    persistent.Map[String, Int, Nothing, Glass](
      dir = File.newTemporaryDirectory(name).deleteOnExit().path,
      mmapMaps = mmapDisabled,
      cacheKeyValueIds = cacheKeyValueIdsOverride,
      segmentConfig = segmentConfigOverride,
      sortedKeyIndex = sortedKeyIndexOverride,
      randomSearchIndex = randomSearchIndexOverride,
      binarySearchIndex = binarySearchIndexOverride,
      mightContainIndex = mightContainIndexOverride,
      valuesConfig = valuesConfigOverride,
      fileCache = fileCacheOverride,
      memoryCache = memoryCacheOverride,
    )

[simerplaha]

Hey! This is due to the same issue you reported before (#358). In this case, a read (maybe compaction) is accessing an Async configured operation (IOStrategy.AsyncIO) which also is being concurrently accessed by another thread.

I think if you made all your configurations use IOStrategy.SynchronisedIO, this will not occur. In the settings you've shared, change IOStrategy.AsyncIO to IOStrategy.SynchronisedIO.

Hope this resolves the problem.

[simerplaha]

After giving this some more thought, I'm afraid it might not fully resolve the problem you are having.

This will be fully resolved in issue #318.

[hicolour]

@simerplaha yeah it is not resolving the issue

It completly stagnate here

io-compute-2" #11 daemon prio=5 os_prio=0 cpu=1391856.78ms elapsed=6212.23s tid=0x00007f3860dd3800 nid=0x18 runnable  [0x00007f38485a5000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.FileDispatcherImpl.pread0(java.base@11.0.7/Native Method)
	at sun.nio.ch.FileDispatcherImpl.pread(java.base@11.0.7/FileDispatcherImpl.java:54)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(java.base@11.0.7/IOUtil.java:274)
	at sun.nio.ch.IOUtil.read(java.base@11.0.7/IOUtil.java:245)
	at sun.nio.ch.FileChannelImpl.readInternal(java.base@11.0.7/FileChannelImpl.java:811)
	at sun.nio.ch.FileChannelImpl.read(java.base@11.0.7/FileChannelImpl.java:796)
	at swaydb.core.io.file.ChannelFile.read(ChannelFile.scala:94)
	at swaydb.core.io.file.DBFile.read(DBFile.scala:412)
	at swaydb.core.io.reader.FileReader.read(FileReader.scala:73)
	at swaydb.core.segment.format.a.block.reader.BlockReader$.read(BlockReader.scala:86)
	at swaydb.core.segment.format.a.block.reader.BlockReaderBase.read(BlockReaderBase.scala:62)
	at swaydb.core.segment.format.a.block.reader.BlockReaderBase.read$(BlockReaderBase.scala:61)
	at swaydb.core.segment.format.a.block.reader.UnblockedReader.read(UnblockedReader.scala:81)
	at swaydb.core.segment.format.a.block.reader.BlockReaderBase.readRemaining(BlockReaderBase.scala:74)
	at swaydb.core.segment.format.a.block.reader.BlockReaderBase.readRemaining$(BlockReaderBase.scala:73)
	at swaydb.core.segment.format.a.block.reader.UnblockedReader.readRemaining(UnblockedReader.scala:81)
	at swaydb.core.segment.format.a.block.segment.footer.SegmentFooterBlock$.read(SegmentFooterBlock.scala:192)
	at swaydb.core.segment.format.a.block.segment.SegmentBlockCache.$anonfun$footerBlockCache$3(SegmentBlockCache.scala:348)
	at swaydb.core.segment.format.a.block.segment.SegmentBlockCache.$anonfun$footerBlockCache$2(SegmentBlockCache.scala:347)
	at swaydb.core.segment.format.a.block.segment.SegmentBlockCache$$Lambda$3318/0x0000000841347040.apply(Unknown Source)
	at swaydb.data.cache.SynchronisedIO.$anonfun$value$7(Cache.scala:323)
	at swaydb.data.cache.SynchronisedIO$$Lambda$1918/0x0000000840bd8840.apply(Unknown Source)
	at swaydb.data.cache.LazyIO.$anonfun$getOrSet$3(Lazy.scala:165)
	at swaydb.data.cache.LazyIO$$Lambda$1919/0x0000000840bd8c40.apply(Unknown Source)
	at swaydb.data.cache.LazyValue.$anonfun$getOrSet$2(Lazy.scala:96)
	at swaydb.data.cache.LazyValue$$Lambda$533/0x00000008403f7840.apply(Unknown Source)
	at scala.Option.getOrElse(Option.scala:189)
	at swaydb.data.cache.LazyValue.$anonfun$getOrSet$1(Lazy.scala:95)
	- locked <0x00000000ea6e5ce8> (a swaydb.data.cache.LazyValue)
	at swaydb.data.cache.LazyValue$$Lambda$532/0x00000008403f7440.apply(Unknown Source)
	at scala.Option.getOrElse(Option.scala:189)
	at swaydb.data.cache.LazyValue.getOrSet(Lazy.scala:93)
	at swaydb.data.cache.LazyIO.getOrSet(Lazy.scala:165)
	at swaydb.data.cache.SynchronisedIO.value(Cache.scala:323)
	at swaydb.core.segment.format.a.block.segment.SegmentBlockCache.$anonfun$getFooter$1(SegmentBlockCache.scala:439)
	at swaydb.core.segment.format.a.block.segment.SegmentBlockCache$$Lambda$3343/0x000000084135c840.apply(Unknown Source)
	at scala.Option.getOrElse(Option.scala:189)
	at swaydb.data.cache.LazyValue.getOrElse(Lazy.scala:126)
	at swaydb.data.cache.LazyIO.getOrElse(Lazy.scala:175)
	at swaydb.data.cache.SynchronisedIO.getOrElse(Cache.scala:326)
	at swaydb.core.segment.format.a.block.segment.SegmentBlockCache.getFooter(SegmentBlockCache.scala:439)
	at swaydb.core.segment.SegmentRef$.higher(SegmentRef.scala:330)
	at swaydb.core.segment.PersistentSegmentOne.higher(PersistentSegmentOne.scala:362)
	at swaydb.core.segment.PersistentSegmentOne.higher(PersistentSegmentOne.scala:219)
	at swaydb.core.level.Level.higherFromFloorSegment(Level.scala:1324)
	at swaydb.core.level.Level.higherInThisLevel(Level.scala:1344)
	at swaydb.core.level.Level$$anon$1.higher(Level.scala:359)
	at swaydb.core.level.seek.Higher$.apply(Higher.scala:99)
	at swaydb.core.level.Level.higher(Level.scala:1368)
	at swaydb.core.level.Level.swaydb$core$level$Level$$higherInNextLevel(Level.scala:1356)
	at swaydb.core.level.Level$$anon$2.higher(Level.scala:372)
	at swaydb.core.level.seek.Higher$.apply(Higher.scala:172)
	at swaydb.core.level.Level.higher(Level.scala:1368)
	at swaydb.core.level.Level.swaydb$core$level$Level$$higherInNextLevel(Level.scala:1356)
	at swaydb.core.level.Level$$anon$2.higher(Level.scala:372)
	at swaydb.core.level.seek.Higher$.apply(Higher.scala:172)
	at swaydb.core.level.Level.higher(Level.scala:1368)
	at swaydb.core.level.zero.LevelZero.findHigherInNextLevel(LevelZero.scala:792)
	at swaydb.core.level.zero.LevelZero$$anon$4.higher(LevelZero.scala:837)
	at swaydb.core.level.seek.Higher$.apply(Higher.scala:172)
	at swaydb.core.level.seek.Higher$.seek(Higher.scala:69)
	at swaydb.core.level.zero.LevelZero.findHigher(LevelZero.scala:864)
	at swaydb.core.level.zero.LevelZero.higher(LevelZero.scala:881)
	at swaydb.core.Core.$anonfun$after$1(Core.scala:309)
	at swaydb.core.Core.after(Core.scala:176)
	at swaydb.Map.swaydb$Map$$nextTupleOrNone(Map.scala:323)
	at swaydb.Map$$anon$1.nextOrNull(Map.scala:337)
	at swaydb.Map$$anon$1.nextOrNull(Map.scala:326)
	at swaydb.data.stream.SourceFree.nextOrNull(SourceFree.scala:63)
	at swaydb.data.stream.step.Map.nextOrNull(Map.scala:49)
	at swaydb.data.stream.StreamFree.$anonfun$foldLeft$1(StreamFree.scala:226)
	at swaydb.data.stream.StreamFree$$Lambda$3511/0x00000008413d8c40.apply(Unknown Source)
	at swaydb.Bag$$anon$4.safe(Bag.scala:517)
	at swaydb.data.stream.StreamFree.foldLeft(StreamFree.scala:226)
	at swaydb.data.stream.StreamFree.foldLeft$(StreamFree.scala:218)
	at swaydb.data.stream.step.Map.foldLeft(Map.scala:30)
	at swaydb.data.stream.StreamFree.foreach(StreamFree.scala:244)
	at swaydb.data.stream.StreamFree.foreach$(StreamFree.scala:243)
	at swaydb.data.stream.step.Map.foreach(Map.scala:30)
	at swaydb.Stream.foreach(Stream.scala:170)