Add persistence to example dataGrid in hono chart

[BobClaerhout]

It would be nice if persistence could be enabled in the example dataGrid for hono. A restart of the datagrid should not result in loss of all the cached data.

[sophokles73]

The example datagrid is just that: an example instance that has been added to quickly get started. We deliberately chose to keep its configuration as simplistic as possible to prevent people from using it in production.
While this feature request does not seem to add much complexity, it certainly opens the door for additional elaborate config options to find their way into the chart. Personally, I would rather not do this and advise to set up a separate dedicated Infinispan cluster using the Infinispan operator and use a custom config file that fully supports your use case at hand.

@calohmn any thoughts?

[calohmn]

I think in terms of providing a simple example datagrid, we could even make persistence the default here. This would mean the Hono chart user doesn't have to think about restarting the Hono protocol adapters when restarting the datagrid.
We could then even omit the persistence.enabled setting, and only add pvcSize.

In general, having to set up an Infinispan Operator deployment adds a fair bit of complexity to the whole Hono deployment, in my view. As a future improvement, to make things easier, we could also consider using the Infinispan Helm chart instead of our example-data-grid deployment yml files. This would provide the whole set of standard configuration options for the Infinispan deployment.

[BobClaerhout]

What's the reasoning behind preventing this to use in production? Why shouldn't we create something that is usable in production? The drawback of deploying a separate data-grid is that you need to update the configuration whenever hono expects something else. We are no experts in data-grid nor do we want to setup a fully separate data-grid cluster, especially if hono is the only service depending on this.
If it would be possible to setup a data-grid using the helm chart (using simple configuration as defaults), that would even be better. In that case, you can use the packages helm chart to setup a working cluster to start from (including some simple persistence in order to avoid to restart the adapters after restarting datagrid). if you want a more complex setup, you can do so by configuring the data-grid through the packages chart. This allows for the hono community to update the required configuration and still keep the production-ready/custom configuration for your own data-grid instance.

[BobClaerhout]

I played around with the infinispan helm chart as a dependency: https://github.com/BobClaerhout/packages/tree/infinispanHelm
However, I'm bumping into following issue: infinispan/infinispan-helm-charts#60. In the meantime, isn't it an option to enable configuring persistence in the current form. When the helm chart is fixed and usable as a dependency, we can migrate to that.
WDYT?