[Feature Request] Programmatically Add new data source to existed database

[JerryChin]

Hey guys, I want to add new data source as application is running, is that anyone interested in implementing this?

[fbacchella]

It's not going to be easy. The one disk format is using one row of all values for each time stamp. So it's not possible to add a column of datasource without rewriting the whole file. I have code that does a check at startup of the application and migrate to a new rrdb file when needed.

[JerryChin]

I've tried a solution offered by popozhu on Stack-overflow, but it seems that rrdtool DOESN'T work with rrd file created by rrd4j, what else can I do?

[JerryChin]

My business scenario is as follows:

I have several nodes which offer many web API services, all requests are tracked per API, later these request count and outcome(success or failed) are sent in batch to a central handler where all rrd updates are processed here via kafka.

All these APIs have a unique path which are used as data source name, it's likely we will add a new API later, and I need a way to migrate it to existed RRD database without shutdown the central handler.

[fbacchella]

one rrd per api ?

[JerryChin]

@fbacchella nope. it's one datasource per api, one rrd per node, like the following:
Files Hierarchy

/data/rrd/
               |node1-ServiceStat.rrd
               |...
               |nodeN-ServiceStat.rrd

every RRD definitions are like:

RrdDef rrdDef = new RrdDef("/data/rrd/node1-ServiceStat.rrd", System.currentTimeMillis() / 1000, 60);
rrdDef.addDatasource("api_path1", DsType.GAUGE, 120, 0, Double.MAX_VALUE);
rrdDef.addDatasource("....", DsType.GAUGE, 120, 0, Double.MAX_VALUE);
rrdDef.addDatasource("api_pathN", DsType.GAUGE, 120, 0, Double.MAX_VALUE);
add one or more archives;

the problem I have is I am not able to add new data source to existed rrd database (as new APIs are added).

nodes are served as load balance.

[JerryChin]

Hey, @fbacchella can you help to solve the issue? I really add new data source to existed RRD db.