Generate chunkIdx at runtime rather than as a file input

[msashokkumar]

Hi,

I have 20 bag files adding up to 1 TB. All my bag files are stored in HDFS already. I have a DC/OS cluster running containers (mesosphere/spark:2.3.0-2.2.1-2-hadoop-2.7). I am trying to run ros_hadoop as a distributed service to extract data from the 20 bag files in an automated way.

Is there a way to generate the chunkIdx during runtime and pass it to newAPIHadoopFile?

If not, can the idx.bin file be stored in HDFS and presented as "hdfs://...idx.bin" to chunkIdx?

Thanks!

[wiegelmann]

Generate the chunkIdx during runtime should be an option. Would need a few days to investigate the matter.

[msashokkumar]

Making the chunkIdx file readable from hadoop is useful as well.

[vini-almeida]

Hi there,
Is there any updates on this recently?
Thank you!