WIP: MGMT-17317: Use IBI to provision SNO enhancement doc #6130
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@adriengentil: This pull request references MGMT-17317 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.16.0" version, but no target version was set. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
added
the
size/L
|
/retitle WIP: MGMT-17317: Use IBI to provision SNO enhancement doc |
openshift-ci
bot
changed the title
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: adriengentil The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/cc @carbonin |
| Currently, installing an SNO cluster using the Assisted-Installer takes up to | ||
| ~40min using the current flow which installs OCP from the ground-up. | ||
|
|
||
| We can reduce the installation time to ~10min by leveraging the Image-Based | ||
| installation method, and as a way to increase the success rate of SNO | ||
| installations as we restore previously working setup. |
There was a problem hiding this comment.
Maybe we should get more exact timings for this.
Does that 40 minutes include all the pre-install validations we're running or is that from the time the user hits the install button? We're going to still be running all those validations during discovery right?
There was a problem hiding this comment.
I agree. I believe @eranco74 already has some numbers in a doc somewhere
There was a problem hiding this comment.
The actual SNO installation takes 30 minutes.
| ### Open Questions | ||
|
|
||
| When selecting Openshift version: | ||
| - Do we plan to provide “curated” seed images? |
There was a problem hiding this comment.
I don't think this is really an option.
Correct me if I'm wrong @javipolo , but the seed config needs to match the hardware of the install target, right?
There was a problem hiding this comment.
yes. this is one of the real challenges. How do we limit this to a super generic installations only.. x86, no special hw, standard disks,... not even sure what
There was a problem hiding this comment.
Not so much. In the vDU case, we needed to match even the number of CPU cores, and I don't remember if the same memory of the systems, because of the performance profile that is applied, in the case of a vanilla OCP, I don't think the hardware needs to match so so much .....
IIRC it needs to have enough memory and CPU and be of the same architecture, but those requirements are already there for normal installations ....
If there are very special needs, then yes, a "vanilla" seed image won't fit
Whatever does not need extra manifests at installation time could be our rule of thumb ...
There was a problem hiding this comment.
After discussion we decided that in the scope of this enhancement, we will let the user to restore a seed they created. I'll add a section for future work with:
- add an option to install and create a seed in Assisted-Installer
- provide vanilla seed images
|
|
||
| When selecting Openshift version: | ||
| - Do we plan to provide “curated” seed images? | ||
| - Should we allow users to install their own seed? |
There was a problem hiding this comment.
Assuming this is the direction we go, we'll need some kind of UX for the user to tell us the image. Then we'll need to ensure we don't actually try to access it from our application as that could be a security risk.
There was a problem hiding this comment.
how useful would it be that u have to generate 1 seed to install 1 cluster? :) I assume it would be usecase only for repeatable users that run heavy automation - we can check in the telemetry maybe and talk to the relevant teams.
|
|
||
| Installation: | ||
| - I don’t know the details, but I foresee some re-work of the around the states | ||
| the installation goes through? |
There was a problem hiding this comment.
Assuming we continue to do all the same validations I'd guess this will mostly be around the install "stages".
We could probably reuse some (writing image to disk), but probably won't need others.
There was a problem hiding this comment.
I'll add the installation steps
| #### Configuration | ||
|
|
||
| After the image seed is restored on the disk, the `assisted-installer` will | ||
| drop the configuration in `/opt/openshift`: |
There was a problem hiding this comment.
We'll also need to consider static network configuration.
I think we should be able to continue to provide nmconnection files (as we are currently in assisted) but in "real" IBI we're using nmstate and running nmstatectl on the host so we'll want to make sure the nmconnection path continues to work.
There was a problem hiding this comment.
Isn't static networking already based on nmstate in assisted (which generate nmconnection files)? or I miss something?
For sure, we need to consider this flow.
There was a problem hiding this comment.
As discussed, nmconnection file should work fine
| - Partition the disk to create a dedicated `/var/lib/containers` | ||
| - Grow `/` partition | ||
| - Mount `/`, `/boot` and `/var/lib/container` | ||
| - Download and restore the seed on the disk using `lca-cli` |
There was a problem hiding this comment.
Where does lca-cli come from? How do we get it into the discovery image?
There was a problem hiding this comment.
The one I used comes from a container image. So I guess we can pull it, and execute it like any other commands?
@javipolo can we always use the last version to restore any existing? Is there some constraints on the lca-cli version based on the OCP version we try to restore?
There was a problem hiding this comment.
lca-cli is version per OCP release, so we should be able to pull it as long as we now the OCP version of the seed we are going to restore.
| A new flag will be added to the cluster object [TBD], along with | ||
| extra-parameters specific to IBI [TBD]. |
There was a problem hiding this comment.
I think it's worth coming up with at least a first pass on the API you'll add in the enhancement.
There was a problem hiding this comment.
I'll add it for sure once get some time, the rough idea I had was something like that in the cluster object:
{
installation_method: "ibi",
ibi_cluster_image: "quay.io/...." # allowed to be set only when installation_method is "ibi"
openshift_version: nil, # because installation_method is set to "ibi"
ocp_release_image: nil # because installation_method is set to "ibi"
}
| - OLM operator manifests | ||
| - Custom manifests | ||
|
|
||
| ### Implementation Details/Notes/Constraints [optional] |
There was a problem hiding this comment.
We should do some work to call out options/features that we currently support, but won't be able to for IBI clusters.
Things like host ignition customization, host install args (maybe these will work since we're still running coreos-installer?), cluster network (cidr), are some that come to mind immediately. Really anything that will be taken directly from the seed, but is already in our API we'll need to make optional and block the user from setting for IBI clusters.
| - generate cryptographic materials in order to provide a kubeconfig file to the | ||
| user |
There was a problem hiding this comment.
@omertuc brought up a point recently about us baking the private keys into the ISO, it might be wise to have these go over an API request from the agent rather than sit in an artifact that the user might misplace.
There was a problem hiding this comment.
Yeah, but let's keep things simple for now and have that in the back of ours minds as a ticket, until we figure out the security side of things and whether this is a threat that's relevant to protect from
| extra-parameters specific to IBI [TBD]. | ||
|
|
||
| `supported-versions` API with be amended to return only the version of OCP | ||
| available for installation with IBI. |
There was a problem hiding this comment.
Under which circumstanaces?
There was a problem hiding this comment.
It is in case we (Red Hat) want to provide generic images ready to use with IBI. But we need to discuss if we to follow that path.
| When the user will hit the installation button, the `assisted-installer` will | ||
| restore the image seed on the disk selected for installation: | ||
| - Install RHCOS using `coreos-installer` | ||
| - Partition the disk to create a dedicated `/var/lib/containers` |
There was a problem hiding this comment.
What's the point of this?
Also could you elaborate on the verb "partition" - what does it mean exactly
There was a problem hiding this comment.
Found a discussion about the first question here
There was a problem hiding this comment.
Related to this topic, I understand we also have a way that works without the partition for /var/lib/containers. In the case we allow users to restore their own snapshots, should we be able to support both partition schemes? @javipolo what do you think?
Also, when /var/lib/containers is required, how can we determine the right size for it?
There was a problem hiding this comment.
As discussed, let's add a label for the OCI container that says which method was done:
- partition
- directory
- none
this way we could support both methods, or having precaching at all
| After the image seed is restored on the disk, the `assisted-installer` will | ||
| drop the configuration in `/opt/openshift`: | ||
| - A | ||
| [manifest.json](https://github.com/openshift-kni/lifecycle-agent/blob/main/docs/post-pivot-configuration.md#user-specifications) |
There was a problem hiding this comment.
Since this is a technical document, let's link to the SeedReconfiguration struct directly, as it's the source of truth and the linked doc may drift out of date
| - SSHkey: will be provided by the user | ||
| - KubeadminPasswordHash: will be generated by Assisted-Installer and | ||
| provided to the user | ||
| - Pull secret |
There was a problem hiding this comment.
Why is this bullet indented separately?
Co-authored-by: Omer Tuchfeld <omertuchfeld@gmail.com>
Co-authored-by: Omer Tuchfeld <omertuchfeld@gmail.com>
|
@adriengentil: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
| - A | ||
| [manifest.json](https://github.com/openshift-kni/lifecycle-agent/blob/main/docs/post-pivot-configuration.md#user-specifications) | ||
| with: | ||
| - Hostname: will be picked from the inventory of host |
There was a problem hiding this comment.
openshift-installer will provide some commands to generate the config artifacts
| ### Test Plan | ||
|
|
||
| A new end-to-end will be created to test the IBI flow. | ||
|
|
|
|
||
| When the user will hit the installation button, the `assisted-installer` will | ||
| restore the image seed on the disk selected for installation: | ||
| - Install RHCOS using `coreos-installer` |
There was a problem hiding this comment.
we should know in advance what version of OCP the user is going to restore
| A new feature support will be created for the IBI installation. | ||
| The feature will be gated by an OCM capability until we make it available to | ||
| all users. | ||
|
|
There was a problem hiding this comment.
Add word about Infra-env and RHCOS version that should be used
No description provided.