Does switching docker-compose mode require to clear docker assets? 🤔

[bossenti]

I just faced a surprising behavior of StreamPipes when using docker-compose as deployment mode:

First, I started StreamPipes with docker-compose -f docker-compose.nats.yaml up -d within installer/compose. After some time I shutdown StreamPipes with docker-compose ... down.
Eventually, I started StreamPipes again and switched the broker via using docker-compose -f docker-compose.yaml. All containers started as expected (Kafka was started, but not NATS). But after logging in, I couldn't really use StreamPipes, e.g. for creating adapters, because it still expected NATS as broker.
So it appears that StreamPipes did not recognize using the wrong configuration/volumes.
One workaround would be simply clear the docker assets when switching between the docker-compose options , but I think this is not ideal...

How are your thoughts about this?

[dominikriemer]

Hi @bossenti, yes, setting the messaging layer is currently a setup-time configuration option since the messaging settings for existing pipelines and adapters are not changed when using another broker. This would only account for newly created resources.
Is this something we would like to have? Then we could add this to the configuration view or as a startup config - I see the following options:

• Iterating over all pipelines and adapters after the broker priority was changed and changing all messaging settings in these resources
• We do not store the messaging info for in the pipeline and adapter descriptions (e.g., only the channel/topic) and dynamically assing them from the settings when starting adapters and pipelines.

[bossenti]

I think the behavior observed by me is only relevant for developing or exploration scenarios.
Usually, users do not frequently change their broker settings, I'd assume.

But if so, the current behavior is a bit confusing IMHO.
Is there a reason why the broker priority is stored in every adapter/pipeline?
The second scenario described by sounds like an ideal solution at the first sight.

[tenthe]

I think this is for some legacy reasons. I do not see a reason why we need to store this information in each resource.
I guess it should be possible to manage the messaging configuration by the backend service. Or are there any edge cases that I am currently not thinking of?

[bossenti]

AFAIK we also use this information in the Python library, but that shouldn't stop us from removing it.
We will surely find an alternative 🙂