Enable Advanced Visibility

[gedw99]

https://github.com/prabhatsharma/zinc

100% golang built on top of the excellent Bluge golang code.

This will work well with SQLite for a single binary solution.

the http api matches Elastic too so easy to plug into . Not everything is supported yet

has a web gui. Not sure if temporalite has one yet .

no cli yet , but relatively easy to do with cobra and making http calls to server . Might already be an issue for it.

it’s fast too .

I am using it and it’s been great .
It’s a new project and still got some rough edges compared to Elastic.

there is also the possibility to integrate via litestream but I doubt you need it. Litestream ( https://github.com/benbjohnson/litestream) gives you a type of changefeed off SQLite as well as real time backup to s3.

if you have any questions feel free to holler !

[gedw99]

I just noticed that the sqlite driver is upstream.
So maybe the elasticlite (zinc) can be done the exact same way - upstream it.

[jlegrone]

Very cool! I was not aware of the zinc project. Upstreaming support in temporalio/temporal would be my preference, as the storage interfaces tend to change a bit and this would make it much easier to stay up to date with the latest Temporal server releases.

We're also looking at litestream, but not sure yet if it needs to be a direct integration (ie. pulling it in as a library vs. running alongside Temporalite). Mentioned this here: https://youtu.be/Hz7ZZzafBoE?t=655 😄

[gedw99]

hey @jlegrone

Very glad you are looking at litestream. it will be interesting if it can be HA'ed too. I spoke to Ben about the idea of doing CDC with litestream also.

Regarding Zinc. There is also a client implementation of elastic here: https://github.com/opensearch-project/opensearch-go The maintainer of Zinc knows about it...

[jlegrone]

I believe there are plans to add full visibility features (eg. filtering by custom attributes) to all sql storage drivers in temporalio/temporal. This should avoid the need for an elastic replacement in Temporalite, since the full visibility API will be supported natively by the SQLite storage driver.

We can keep this issue open to track enabling advanced visibility in Temporalite though.

[Irvenae]

+1 for this

[karelbilek]

Advanced visibility is already done in temporal 1.22 for sqlite...

[karelbilek]

I don't think this repo is maintained.

But if it is, you can merge #234 and that could enable advanced visibility