Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PVCs] Create public documentation for data backups and restoration #882

Open
DnPlas opened this issue Apr 23, 2024 · 4 comments
Open

[PVCs] Create public documentation for data backups and restoration #882

DnPlas opened this issue Apr 23, 2024 · 4 comments
Labels
enhancement New feature or request

Comments

@DnPlas
Copy link
Contributor

DnPlas commented Apr 23, 2024

Context

We need to create public documentation for users that want to backup their data and restore from it.

What needs to get done

Create a section in this discourse post with information about data backup and restoration for charms that use PVCs (mlmd and minio).

Definition of Done

A section in the discourse post that exposes information about backup and restoration for PVCs used in CKF.

@DnPlas DnPlas added the enhancement New feature or request label Apr 23, 2024
Copy link

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5599.

This message was autogenerated

@kimwnasptd
Copy link
Contributor

In the first iteration of the guide, it instructs users to have a config for rclone that sets the secret-key of MinIO to a value of minio123.

Our Charm though right now sets an empty value for the secret-key, which means that the MinIO Charm will create a random password and not reflect its value in the config option.
https://github.com/canonical/minio-operator/blob/track/ckf-1.8/src/charm.py#L219-L234

This urged me to change the password in MinIO (although we dropped this approach, since it's bad UX to expect users to change the password to make a backup) but then stumbled upon the issue that the KFP Profile Operator will not correctly propagate the minio secret-key to the user namespaces canonical/kfp-operators#449

The above resulted in the following errors in the UI and ml-pipeline-ui-artifact in the user namespace

Failed to get object in bucket: S3Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.

@kimwnasptd
Copy link
Contributor

To rewrite the instructions we have and tell users which MinIO secret-key to use and where to get the current value from. Currently, if the config option is set to "" we don't have a direct way from the Charm to get this. Ideally in the future this should be a dedicated charm action canonical/minio-operator#168

For now we'll use the following command for getting the secret-key value

juju show-unit kfp-ui/0 \
        | yq '.kfp-ui/0.relation-info.[] | select (.endpoint == "object-storage") | .application-data.data' \
        | yq '.secret-key'

Note that the above command is not targeting MinIO, which is the producer, but a consumer. This was because we couldn't find a way to get this outgoing value from MinIO.

@kimwnasptd
Copy link
Contributor

Then, I've created some follow-up issues so that in the future we move this backup/restore logic to the Charms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants