[Bug] Message type mismatch causes the dispatcher stopped

[phuaxinyuin]

Hi @iancooper,

We are performing some testing on the message header of command request and the dispatcher will stopped when one of the messages hit the message type mismatch error.

Here is the scenario,

(We try to exclude the message type from the header to perform the testing)

We found out that Brighter will set the messageType to "MT_EVENT" when the messageType key/value was not found from the Kafka message

Brighter will throw a ConfigurationException & dispatcher will stopped after this.

Would like to know if there any way can prevent the dispatcher stop processing the next message?

If one of the messages hit the ConfigurationException, the channel will be disposed and other messages cannot be proceed.

[phuaxinyuin]

Hi, here is another issue that we are facing.

We are using transformer to perform schema validation & send the message to DLQ (not sure is it correct if we do in this way)
when we tested with a message with a invalid message id, it wouldn't proceed to transformer and we cannot process the schema validation and DLQ part. But when we test with a empty value, the message able to process with a new guid value.

[iancooper]

@phuaxinyuin Will take a look

[siewching-lim]

Hi @iancooper, may I know if there is any update for this issue?

[iancooper]

Hi @siewching-lim -

I would like to understand the behavior you would expect. Brighter needs to know if something is a Command (one handler) or Event (many handlers). This may be clearer with an internal bus, but we still have to make choice. If the sender does not tell us, we assume an event (you might have many handlers). To fix this you can simply make your handler an event handler as opposed to a command handler.

Our alternative would be to assume it is a command. The problem here is that makes the most restrictive assumption.

[siewching-lim]

Hi @iancooper, thanks for your reply.

I hope that Brighter will not prevent me from passing the messages into message mappers, since we are using Transformer to perform schema validation for both the headers and the body of the messages in that layer. Could you please provide an option for us to skip the message type validation step (ValidateMessageType)?

e.g.

Additionally, I noticed that when I pass in a value that is not a valid GUID as the MessageId, it blocks my message from reaching the message mapper layer. Could you kindly provide an option to automatically generate a new GUID for the MessageId if the value is invalid?

[siewching-lim]

Hi @iancooper, do you have any update on this issue?

[iancooper]

Hi @siewching-lim

So I think what we are asking here is:

• If you go through ValidateMessageType with a message that does not have a Brighter message type, you would like us to default the type and map it to say MT_EVENT and provide a GUID based message id if there was not one already?

Is that the requirement here?

[iancooper]

I am working on a couple of bug fixes for V10, so I might be able to add that

[siewching-lim]

Hi @iancooper, thank you for your reply.
I believe that assigning a default type of MT_EVENT in the ValidateMessageType function may not resolve my issue. This is because Brighter already assists in setting the default message type to MT_EVENT within the KafkaMessageCreator. My objective is to prevent Brighter from raising an exception and halting the processing of subsequent messages, even when there is a mismatch between the message type and the handler type.

Regarding the MessageId, you’ve understood my request correctly: if one is not provided, a GUID-based message id should be generated.

[iancooper]

a mismatch between the message type and the handler type.
I would be loathe to turn off our protection, because it spots configuration issues. So I assume that we default to MT_EVENT, but you treat as a Command? So the problem might be allowing you to configure our default i.e. you can default to a Command instead of an event. I could look at that requirement. In theory this would allow you to say this external event does not come with MT_COMMAND or MT_EVENT but always treat it as MT_*

[siewching-lim]

Hi @iancooper, thanks for your reply again.

Yes, I'm using a command handler within my application. However, I am of the opinion that the ability to configure the default message type may not provide a solution to my current predicament. This is because there exists a possibility that the producer might publish a message with an MT_EVENT type. This could halt the dispatcher due to a mismatch between the message type and the handler type. This is the crux of the issue we are facing.

To paint a clearer picture, let me delve into the specifics of my use case. Our application is designed with schema validation, which is seamlessly integrated with the Brighter message mapper through a transformer.

Under normal circumstances, if a producer inadvertently dispatches a Kafka message with MT_EVENT or a null message type, it would fail the schema validation. Consequently, the message would be redirected to the dead letter queue, bypassing the process in command handler entirely.

However, we’re currently grappling with a persistent issue. Whenever a user inputs a message type that isn’t MT_COMMAND, the application will be stop processing the next message. Ideally, we would like the application to continue processing subsequent messages, despite any preceding errors. Hope you can help us to solve this issue.

[iancooper]

Whenever a user inputs a message type that isn’t MT_COMMAND, the application will be stop processing the next message. Ideally, we would like the application to continue processing subsequent messages, despite any preceding errors.

Thanks that is a clearer requirement. Let me ponder. It sounds as though we may want a flag that lets the user control whether Brighter considers a message type validation error to be fatal or not. We would need to commit the offset (ack) in this case as we would otherwise never clear that message.

[siewching-lim]

Thanks that is a clearer requirement. Let me ponder. It sounds as though we may want a flag that lets the user control whether Brighter considers a message type validation error to be fatal or not. We would need to commit the offset (ack) in this case as we would otherwise never clear that message.

I appreciate your prompt response. Kindly keep me informed with the latest updates.