Negative values for compaction score

[phs-sakshi]

Setting the level0_file_num_compaction_trigger to a negative value, results in a negative compaction score.

As per the documentation on compaction statistics, compaction score can have values between 0 and 1, and any value greater than 1 means that the level needs to be compacted, but it doesn't specify anything about negative values.

Minimum reproducible example is attached -

void Database_Handler(int N) {

          options.create_if_missing = true;
          options.statistics =  rocksdb::CreateDBStatistics();
          rocksdb::Status status = rocksdb::DB::Open(options, db_path, &db);
      
          if (!status.ok()) {
              logger::debug("Failed to open database: " + status.ToString() );
              return;
          }
          rocksdb::Status new_config_status = db->SetOptions({{"level0_file_num_compaction_trigger", "-1"}});  //tune parameters
          if (!new_config_status.ok()) {
              logger::error("Failed to set options: {}", new_config_status.ToString() );
          }
          std::string stats;
          db->GetProperty("rocksdb.stats",&stats);
          logger->info("compaction stats {}",stats.c_str());
}

shows compaction statistics like this -

[info] compaction stats 
** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0     10/0    1.84 MB -10.0      0.1     0.0      0.1       0.1      0.0       0.0   3.1      7.5     11.0      7.49              0.48       605    0.012    485K      0       0.0       0.0
  L6      1/0   24.28 MB   0.0      2.5     0.0      2.5       2.5      0.0       0.0 104.8    256.2    256.1     10.06              6.80       137    0.073     22M      2       0.0       0.0
 Sum     11/0   26.12 MB   0.0      2.6     0.0      2.5       2.6      0.0       0.0  99.1    150.1    151.6     17.54              7.28       742    0.024     23M      2       0.0       0.0
 Int      0/0    0.00 KB   0.0      2.6     0.0      2.5       2.6      0.0       0.0  99.1    150.1    151.6     17.54              7.28       742    0.024     23M      2       0.0       0.0

** Compaction Stats [default] **
Priority    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Low      0/0    0.00 KB   0.0      2.6     0.0      2.5       2.6      0.0       0.0   0.0    240.4    240.4     10.95              7.00       244    0.045     23M      2       0.0       0.0
High      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      4.1      6.59              0.28       498    0.013       0      0       0.0       0.0

Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0

Uptime(secs): 45.2 total, 45.1 interval
Flush(GB): cumulative 0.026, interval 0.026
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 2.60 GB write, 58.89 MB/s write, 2.57 GB read, 58.31 MB/s read, 17.5 seconds
Interval compaction: 2.60 GB write, 58.92 MB/s write, 2.57 GB read, 58.34 MB/s read, 17.5 seconds
Estimated pending compaction bytes: 1932493
Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 2, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 65, l0-file-count-limit-stops: 4, memtable-limit-delays: 0, memtable-limit-stops: 36, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 65, total-stops: 40
interval: 105 total count
Block cache LRUCache@0x6000003880d8#41433 capacity: 32.00 MB seed: 1535199315 usage: 0.08 KB table_size: 1024 occupancy: 1 collections: 1 last_copies: 0 last_secs: 0.000121 secs_since: 45
Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%)

It also allows for level0_file_num_compaction_trigger to be 0 (without any errors or warnings), which might lead to random compaction score values as per the calculation here.

Unless I am using the API wrong, is this the intended behaviour?

[ajkr]

SetOptions() sanitization is a longstanding issue:

rocksdb/include/rocksdb/db.h

Lines 1452 to 1454 in fc40165

	// USABILITY NOTE: SetOptions is intended only for expert users, and does
	// not apply the same sanitization to options as the standard DB::Open code
	// path does. Use with caution.

[phs-sakshi]

okay, but what does it mean to have negative compaction scores? As per the documentation, it can only have positive values.

[ajkr]

Sorry you are right that negative compaction scores can happen for L0 when L0 compaction is disabled (-1). That just means compaction won't happen. You can change them to report zero if you wish - it should have the same behavior.

[phs-sakshi]

okay, so in that case, with negative compaction scores, will compactions in L0 be triggered by the size of L0 instead of the number of files, or will it not be triggered at all?

Also, it allows for level0_file_num_compaction_trigger to be 0 (without any errors or warnings), which might lead to random compaction score values as per the calculation here. How will compactions be triggered in that case?