Question: read side separation

[maxekman]

@Encounter27 wrote:

@maxekman @bweston92 @mcarriere we are evaluating technology stack for one of our projects and I found that event horizon can solve our need of CQRS/ES in Golang 😊.

So I want add feature to add outbox for complete read side separation. Please review my idea and design !

What I understood about the current flow of Eventhorizon:

So, I was thinking about this since last couple of days, to solve the bellow problems:

1. Store-Event and projector should be independent, so write side and read side can be scale separately.
2. Irrespective of read side projectors (There can be multiple read side projectors) the write side should be 100% available.
3. Indeed, we should be able to publish events to some message queue from one of the read side projectors.

So, I came up with the below design:

So, in the above diagram, Two aggregate is there A1, A2. On A1 [A1:E1, A1:E2, A1:E3] events applied and on A2 [A2:E1, A2:E2] events applied.

1. The event store is the same event store present in event horizon.

2. Event outbox will have each event (having the ISO timestamp) occurred in the system. These events are sorted by timestamp.

3. Writing to event store and outbox will be atomic.
4. Time offset collection will hold the timestamp for each projector till the projector processed the events.
   

Now, say we need 3 projectors:

1. Each projector is independent go-routine.
2. Say projector1 routine prepare read side view in //DB1.
   And projector2 routine prepare read side view in //DB2.
   Projector3 routine publish event to some msg queue say Kafka.

Lifecycle/Responsibility of each projection routine:

1. When the application will start all the projection routine will also start and keep running until the pod stops or pod crashes.
2. Now each projector will check the Time offset collection. if the projector entry is not found it will start read from the first event and update the projector entry and with the timestamp till it processed.
3. Also, I added a string field “projectionDone” to each event present in outbox. So, the length of the field will be equal to number of read side projector.
   For total three projector, for each event of outbox the default value of “projectionDone” field will be “FFF”. 0-index for projector1, 1-index char for projector2 and so on.
4. So, a projector when it will read an event it will mark the field with ‘I’ (In process). Then the projector will write the event to read side DB/Kafka and if succeed, it will mark the field to ‘T’ (Processed) and also update the Time offset.
   Example: projector1
   “FFF” -> “IFF” -> “TFF”
   Also this char position will help to synchronize the same projector present in multiple pods (See the diagram).
   Here the above step 2, 3, 4 will be atomic for one projector.

Benefits:

1. If we reset the time in Time Offset collection say to initial state, we will be able to replay all the projections.
2. Now write side and read side can be scale separately.

Concern:

1. Read side projection delay. May be this eventual delay is ok as per CAP theorem.
2. If same projector is running parallelly in multiple pods, there will be some contention, and these doesn’t make any sense as one read side projection should process one event only once.
   For this I am thinking below sol:
   Run the projector completely in separate and single pod. So basically, for write there can be multiple pod and for projection there will be only one pod

Please review and share the thought on this. I am not sure how much I could express in words :) , may be for more clarification we can have a short video call.
If the idea sounds good to you all, I will create a PR.

Waiting for some feedback…

Thanks,
Pradip

[maxekman]

@Encounter27: Moved your post to this issue.

[toesterdahl]

It looks like the outbox is implementing a persistent pub/sub queue. I would prefer a solution that was open to hook in any such queue. Otherwise I agree that writing to the event store is a synchronous operation and publishing the events is asynchronous. Again, asynchronous distribution of the events is 'for free' with a queue. That projectors are independent 'pods' or 'services' that may go up and down independently I consider the principal use-case. Means for initializing or replay projections are essential. In my view the best protocol is the simplest possible and a sortable UUID + version number should be sufficient. The solution should be very flexible to the nature of the projectors implemented. Also things like browser front-ends listening on a web socket to update a user session need to be supported.

[maxekman]

@Encounter27 There is an error in your diagram: the apply event and store aggregate always happens and must succeed before publishing an event.

[Encounter27]

@Encounter27 There is an error in your diagram: the apply event and store aggregate always happens and must succeed before publishing an event.

@maxekman
Updated diagram:

(I pasted here, as I am not able to replace the first diagram in your comment :) )

[Encounter27]

It looks like the outbox is implementing a persistent pub/sub queue. I would prefer a solution that was open to hook in any such queue. Otherwise I agree that writing to the event store is a synchronous operation and publishing the events is asynchronous. Again, asynchronous distribution of the events is 'for free' with a queue. That projectors are independent 'pods' or 'services' that may go up and down independently I consider the principal use-case. Means for initializing or replay projections are essential. In my view the best protocol is the simplest possible and a sortable UUID + version number should be sufficient. The solution should be very flexible to the nature of the projectors implemented. Also things like browser front-ends listening on a web socket to update a user session need to be supported.

@toesterdahl

1. Yes i am also looking for some persistent queue package written in Go with MongoDB. But I don't want to use any external queue, because then the write side will be depend on the availability of the queue.
   For this I am looking around https://github.com/mongodb/amboy and https://blog.serverdensity.com/replacing-rabbitmq-with-mongodb/. I need time to evaluate the write stack or may be you can share if you know some tool for persistent queue with MongoDB.
2. Regarding sortable UUID, yes you right; it will simplify the problem. And for this I am looking around https://github.com/segmentio/ksuid . This looks good to me.
   Note:This id will be use only for ordering the events in outbox, this will not replace the actual event id assigned by EH core.
3. Definitely the projector need to be very flexible and generic, so the user can inject their conversion logic i.e internal event to external event and also can choose only the events needs for the projection.
   For web socket till now I didn't thought of anything :).

[Encounter27]

I have completed some work on this https://github.com/Encounter27/eh-outbox
I will test it for some time, and eventually create a pull request.

[maxekman]

There is now an official outbox implemented, merged in #335.