Appending data to an existing index

[bobajeff]

Is there a way to append toml data to an existing index (*.st file) rather than build a new one? I was looking through the documentation and didn't see it mentioned anywhere.

I'm trying to index a lot of data and I think it might be slow to have to rebuild it again every time I add to it.

[jameslittle230]

Unfortunately, there's no way to do that today. I've heard that feedback from a few other people before but have had to deprioritize it so far.

If I were to implement this, how would you want to append data to an index? Over the command line, with a different configuration file than the original one? With the same configuration file, except only the new stuff would be appended? Without any configuration file at all? (If you have an example of what you'd want to do to append data, please share! I'm curious!)

In lieu of this, I could also work on optimizing the indexing system. Is there any part of your data that's making it go particularly slowly, do you think?

[bobajeff]

I've already decided on another approach. For my use I'd need to account for a rather large database size (and that's not the focus with this project).

For what it's worth I think something like:

stork --add_documents doc.toml --index idx.st

also for removal something like

stork --remove_documents doc.ids --index idx.st

would make sense.