Elasticsearch backup performance improvement

[andreatera]

Currently using backman v1.30.2 backing up data from our elasticsearch service in internal iAPC.

Backman is taking almost 30min to backup 23MB, now is running since almost 5 hours to backup 600MB and is not yet done.

That's not possible, we have elasticsearch with 4-5GB of data still growing, how long will take to backup those? days?

[denysvitali]

I've created searchdump (Swisscom-internal only for now, open source soon) which is basically solving the performance issue of elasticsearch-dump and is implemented in Go. Therefore the integration with backman should be pretty easy. Me and @JamesClonk already chatted quickly about it already :)

[andreatera]

that's very good news!
Any plan when you'll integrate in backman? So we can track it an make a try when is ready?

[denysvitali]

I'll work on some backman features today, but sadly I haven't planned some time to make searchdump open source or integrate it w/ backman yet

[denysvitali]

FYI: searchdump is now public :)

[buffalonan]

Hello @denysvitali, we are very interested in the feature, since we have big ES instances that we cannot backup with Backman since a couple of years. Any news regarding its integration to Backman? Thanks