[Enhancement][opensearch] readinessProbe uses tcp socket instead of health api endpoint & no livenessProbe

[kamil-stc]

Is your feature request related to a problem? Please describe.
Recently we had an issue when running opensearch 2.1 in k8s cluster (1.22.9). When one of the coordinating nodes is being evicted filebeat's start having connection issues. We are currently testing it with changed readinessProbe to http. Due to helm/helm#5184 it is not possible to overwrite values shipped with embedded helm chart.

Testing if port 9200 is open might be not enough to check if the pod is healthy. Also lack of livenessProbe might lead to directing traffic to pods that are not running.

Describe the solution you'd like
Change readinessProbe to checking health check endpoint over http (https://opensearch.org/docs/1.2/security-plugin/access-control/api/#health-check). It might also be useful to have livenessProbe defined with the same endpoint.

Describe alternatives you've considered
None

Additional context
Add any other context or screenshots about the feature request here.

[prudhvigodithi]

Hey @kamil-stc the link you provided is specific to the security plugin, the health check endpoint should be for the cluster alone.
@TheAlgo can you add your thoughts from this PR #172

[TheAlgo]

@kamil-stc There were various cases because of which it was decided to not use an authenticated probe. It will be great if you can read through the conversations in the PR to understand why this decision was taken

#172 (comment)

[TheAlgo]

Closing this issue for now.