Question about ttl.

[hei66]

I want to dynamically create columnFamily, when all are created, after the application is restarted, when TtlDB.open, I don't know the history of the created columnFamily ttl

private static TtlDB ttlDB;
static List<ColumnFamilyDescriptor> columnFamilyDescriptors = new ArrayList<>(); //ColumnFamilyDescriptor集合
static List<ColumnFamilyHandle> columnFamilyHandles = new ArrayList<>();
static String dbPath = "D:\\ttlRocksdb";
static DBOptions options = new DBOptions()
        .setCreateIfMissing(true)
        .setCreateMissingColumnFamilies(true);
static {
    try {

        String columnFamilyName = "asdf";
        ColumnFamilyOptions columnFamilyOptions=new ColumnFamilyOptions();
        List<byte[]> cfArr = TtlDB.listColumnFamilies(new Options(), dbPath);
        List<Integer> ttlLists = Lists.newArrayList();
        if (CollUtil.isNotEmpty(cfArr)) {
            for (byte[] cf : cfArr) {
                String x = new String(cf);
                columnFamilyDescriptors.add(new ColumnFamilyDescriptor(cf, columnFamilyOptions));
                //todo setTTL?
            }
        } else {
            columnFamilyDescriptors.add(new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions));
        }
        Boolean flag = true;
        if (CollUtil.isNotEmpty(cfArr)) {
            for (byte[] cf : cfArr) {
                String cfName = new String(cf);
                if (CharSequenceUtil.equals(cfName, columnFamilyName)) {
                    flag = false;
                    break;
                }
            }
        }
        //ColumnFamilyHandle集合
        ttlDB = TtlDB.open(options, dbPath, columnFamilyDescriptors, columnFamilyHandles, ttlLists, false);

        if (flag) {
            log.info("create column family");
            ColumnFamilyHandle columnFamilyHandle = ttlDB.createColumnFamilyWithTtl(new ColumnFamilyDescriptor(columnFamilyName.getBytes(), columnFamilyOptions), 5);
            columnFamilyHandles.add(columnFamilyHandle);

        }
    }catch (Exception e){
        e.printStackTrace();
    }
}

[rhubner]

Hello @hei66,

you are right, there is not a single method to get the TTL of existing column families. Looking to the C++ code I can't even see anything here regardes getting existing TTL values.

It looks like it is not even stored anywhere in the database, because TTL is implemented via TTL compactions filter. See this code. To know what TTL is used, I suggest to store it somewhere else, or use something default.

Radek

[rhubner]

It looks like I was wrong,

it is stored in the Options file. For example :
compaction_filter_factory={id=TtlCompactionFilterFactory;user_filter_factory=nullptr;ttl=5684557;}
The question is, how to get this data before opening the database.

[hei66]

I have another idea, initialize ttl to 0, and then modify ttl after opening db. I haven't found any way to modify ttl.

[rhubner]

I have another idea, initialize ttl to 0, and then modify ttl after opening db. I haven't found any way to modify ttl.

That's possible, but then you risk that some data may be deleted.

[hei66]

I have another idea, initialize ttl to 0, and then modify ttl after opening db. I haven't found any way to modify ttl.

That's possible, but then you risk that some data may be deleted.

1. TTL is set to 0. Shouldn't these data not expire?
2. After opening db, how can I modify ttl?

[rhubner]

TTL is set to 0. Shouldn't these data not expire?

No, as specified here Expired TTL values are deleted in compaction only:(Timestamp+ttl<time_now)

After opening db, how can I modify ttl?

I don't think this is possible for existing Column Families, after opening, but it's possible for new Column Families.

Checking the code, it should be possible to load Options with OptionsUtil.loadLatestOptions where you can get ColumnFamilyDescriptor then ColumnFamilyOptions. Unfortunately ColumnFamilyOptions doesn't return anything for compactionFilterFactory(). I think it's a limitation of the Java API. Checking C++ code, TtlCompactionFilterFactory doesn't expose ttl_ value. So even in C++ we can't read this information. Maybe custom Option file parser will be able to get this information, but there is no guarantee that the format of the compaction_filter_factory field in the options file will not change in future.

@pdillinger @ajkr Do you think we need a new API to keep track of TTL? Maybe the Option class can store and expose this for each column family. As I mentioned, it's already there under the compaction filter, but It's not readable with the current API. Is this information from Options even used when you reopen TTLDb ?