Use SparseEvent for Detection.event

[calum-chamberlain]

Is your feature request related to a problem? Please describe.
Obspy Events are great, but, storing millions of them in memory is expensive. Currently match-filter called from Tribe.detect will generate an Event for each detection. The information in those events is very sparse (an Origin, ResourceID and some Picks). We do not need all the additional slots or functionality.

Describe the solution you'd like
A set of simple SparseEvent, SparseOrigin and SparsePick classes, with the SparseEvent object having a to_obspy method to convert to a full obspy Event. This would allow simple transitions between a more memory efficient implementation and all the extra stuff (including io) that comes with obspy events.

Describe alternatives you've considered
At the moment I set return_events=False for large Tribe.detect runs. This works fine, but it is kind of annoying, and not all that obvious that generating the events is (one of the things) that takes up heaps of memory.

Additional context
I have written some sparse classes for other work that seem to work quite well and could be dropped into EQcorrscan quite easily.

[d-chambers]

I would be interested in seeing how you have implemented the sparse classes, are they somewhere publicly viewable?

[calum-chamberlain]

They are really basic, I don't have any useful functionality in them, just names. Its super naive and just works as a place-holder that maps across to Event relatively nicely for what I'm doing (so it ends up quackinglooking like what I want it to when I want it to, but would break as soon as anything that wasn't supported was requested, kinda like a duck with only a beak).

What I have been using is here, which suits that purpose (which is just storing some parameters I need for a nationwide repeating earthquake search). It would need more work for EQcorrscan, and in particular to cope more cleanly when methods or parameters that aren't there are asked for.

[d-chambers]

Ok, make sense. Sometimes all you need is the quack, until you realize you are really after a (wild) goose 😉.

I have been thinking for some time about implementing a lazy catalog that would be backed by a SQL database (SQLite by default). If something like that had an option to periodically push newly added picks to disk (keeping them out of memory) would it meet this need? It is something I hope to include in ObsPy eventually but it may take awhile to get right. I would be keen to get your feedback early on to make sure it could be useful to EQcorrscan-like use cases.

[calum-chamberlain]

A lazy catalog could be really good, yes. I would be keen to help out on that, but (I must confess), partially because I have been looking for a good reason to learn more about the world of SQL.