Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Kubernetes Parallel pod management policy #11022

Open
ggjtm opened this issue Oct 31, 2023 · 0 comments
Open

Enable Kubernetes Parallel pod management policy #11022

ggjtm opened this issue Oct 31, 2023 · 0 comments

Comments

@ggjtm
Copy link

ggjtm commented Oct 31, 2023

Problem:

Currently, the Kubernetes IP Finder requires configuring a Kubernetes Service name, and the module queries this object in the Kubernetes API to enumerate Ignite cluster node IP addresses. Kubernetes will list only those nodes which have already successfully become ready to receive query traffic. OrderedReady pod management policy in StatefulSet is the default, and works well, but enforces the slowest cluster startup as each node must present itself as ready prior to the next node's pod startup. Parallel pod management policy does not work because all nodes perform and complete the discovery phase before any nodes have presented themselves as ready for the Kubernetes Service.

Consequences:

Large clusters (many nodes) and clusters with many cache groups may take tens of minutes (or more) to start up. Kubernetes hosted Ignite clusters should behave more like non-Kubernetes hosted clusters with Parallel pod management policy, enabling independent nodes to perform most startup work concurrently. This proposal would enable arbitrarily large Kubernetes clusters to start up in similar time to small clusters.

Proposal:

  1. Continue to use nodes' IPs enumerated by the Service ClusterIPs or Endpoints
  2. Inspect the Service spec.Selector
  3. List pods constrained by the Service Selector
  4. Iterate through listed pods, and query each Pod's status to collect PodIPs
  5. Merge collected PodIPs to the list of IPs from the Service

References:

@ggjtm ggjtm changed the title Enable Kubernetes IP Finder to work with Enable Kubernetes Parallel pod management policy Oct 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant