Is there a way to generate yml files that will produce the existing cluster? #24873
Comments
|
kubectl get po,deployment,rc,rs,ds,no,job -o yaml? |
|
Ah yes, of course! This works, but it is not what what I was looking for. It answers my question, but it doesnt give me files that match the ones I used. I learned that the answer to this question is to read the 'last-applied-configuration' annotation that kubectl adds. this will give the files that were used to produce the config. |
|
@dcowden also see kubectl get --export |
|
ah thats even better! thanks! |
|
Combining other answers, this is what I came up with for bash: for n in $(kubectl get -o=name pvc,configmap,serviceaccount,secret,ingress,service,deployment,statefulset,hpa,job,cronjob)
do
mkdir -p $(dirname $n)
kubectl get -o=yaml --export $n > $n.yaml
doneEdit: |
|
k8s 1.8 kubectl get all --export=true -o yaml |
|
For folks coming here from Google, on my test instance, the last comment's Related: #42885 and #42954 (comment) etc. |
|
A variation on top of the solution provided by @alahijani This is to have all yaml files in a single dir for an easy |
|
Another version: Exporting all yaml's from all namespaces. For each namespace a directory is made.
and for importing again: |
|
Another little tweak to exclude service account tokens: |
|
For those who works in windows Powershell, here's an one-liner: |
|
@mrwulf @acondrat I'm using the following versions of kubectl and k8s cluster |
|
@4m3ndy you are right! Thanks! |
|
Hey guys just made this docker image for exporting the required yaml files for each component per namespace. these backups are exported then encrypted with a password and uploaded to S3 Bucket. If any one would like to commit any changes or share any comments, you're more than welcome 👍 |
|
example: generate pv yaml. kubectl get pv -o yaml --export | sed -e '/resourceVersion: "[0-9]\+"/d' -e '/uid: [a-z0-9-]\+/d' -e '/selfLink: [a-z0-9A-Z/]\+/d' -e '/status:/d' -e '/phase:/d' -e '/creationTimestamp:/d' > pvList.yaml |
|
@xiaoping378 This script should generate all pvc for each namespace then export the yaml file for each pv, have a look |
|
Create a folder ${HOME}/clusterstate/, then run: |
FYI, this appears to need a decent amount of RAM for large deployments, my 2GB RAM CLI jumpbox VM can't handle it (probably need 4 or 8 I'd imagine): I reran on my desktop and tracked kubectl process memory usage, it peaked at just around 4GB, so 8GB it is! Apparently that about matches the total output size of the dump, which includes logs and some of our pods (90 of them) are putting out logs of over 100MB in size. This would indicate to me the dump command is storing everything in RAM even as it is writing out to disk, probably could be optimized to clear out RAM as logs are finished writing. |
|
Can anyone tell me the command or script which will take backup of cluster (includes ns, deployment, svc, secrets, pv pvc, cm yaml files only) with all information and restore it in new cluster. |
|
Thanks to Above bash script was tested with:
|
|
@vhosakot Small simplification to your script. You can replace: |
…volume-expansion-test Bug 1810470: UPSTREAM: <drop> Increase timeout in volume expansion test Origin-commit: 74769b89204e470f4c12391134a817d38febdc72
#!/usr/bin/env bash
while read -r namespace
do
echo "scanning namespace '${namespace}'"
mkdir -p "${HOME}/cluster-backup/${namespace}"
while read -r resource
do
echo " scanning resource '${resource}'"
mkdir -p "${HOME}/cluster-backup/${namespace}/${resource}"
while read -r item
do
echo " exporting item '${item}'"
kubectl get "$resource" -n "$namespace" "$item" -o yaml > "${HOME}/cluster-backup/${namespace}/${resource}/$item.yaml"
done < <(kubectl get "$resource" -n "$namespace" 2>&1 | tail -n +2 | awk '{print $1}')
done < <(kubectl api-resources --namespaced=true 2>/dev/null | tail -n +2 | awk '{print $1}')
done < <(kubectl get namespaces | tail -n +2 | awk '{print $1}')i extended the script above a little (and slowed it down). this loads all namespaces, loads all resources in all namespaces and then loads each config as a single file in each resource in each namespace. it is verbose and shows some errors, bute the end result (the dump) should be clean. |
#!/usr/bin/env bash
ROOT=${HOME}/clusterstate
while read -r resource
do
echo " scanning resource '${resource}'"
while read -r namespace item x
do
mkdir -p "${ROOT}/${namespace}/${resource}"
echo " exporting item '${namespace} ${item}'"
kubectl get "$resource" -n "$namespace" "$item" -o yaml > "${ROOT}/${namespace}/${resource}/$item.yaml" &
done < <(kubectl get "$resource" --all-namespaces 2>&1 | tail -n +2)
done < <(kubectl api-resources --namespaced=true 2>/dev/null | tail -n +2 | awk '{print $1}')
waitInspired by @scones but runs a little quicker because of process forking and reduced loop nesting which is useful if you have a lot of custom resource definitions! |
|
Same as @nathan-c #!/usr/bin/env bash
ROOT=${HOME}/clusterstate
while read -r resource
do
echo " scanning resource '${resource}'"
while read -r namespace item x
do
mkdir -p "${ROOT}/${namespace}/${resource}"
echo " exporting item '${namespace} ${item}'"
kubectl get "$resource" -n "$namespace" "$item" -o yaml > "${ROOT}/${namespace}/${resource}/$item.yaml" &
done < <(kubectl get "$resource" --all-namespaces 2>&1 | tail -n +2)
done < <(kubectl api-resources --namespaced=true 2>/dev/null | grep -v "events" | tail -n +2 | awk '{print $1}')
wait |
|
In case someone still searches here for the answer, like me. |
|
The script looks good, but for cloud maintained cluster seems to have slight issue in connecting to api. It sometimes times out and then reconnect. May be need to slow down the connections to export from cluster. May be some tweaks will do. Unable to connect to the server: dial tcp 20.143.169.19:443: i/o timeout |
This is obviously just a connection issue you are having. |
But unfortunately, it does not work for all resources, please check deployments, replica sets and ingress |
|
Use GitOps |
|
@o6uoq investing even more work to solve a small problem does not solve the problem of investing less work. |
|
I found the best sophisticated project here : https://github.com/pieterlange/kube-backup So there are a lot of short commands to perform and reduce the command lines For example if you get an inventory list to permit iterate after (like this command ) Then you can get any yaml from before list with a simple The project that I indicated before has a excellent implementation here more details https://github.com/pieterlange/kube-backup/blob/master/entrypoint.sh So I have pending to check this code to eliminate part of default yaml statuses (I prefer yaml export but the project used json export, so it could be a better solution: |
|
Inspired by @mohamedelhabib , @nathan-c, @scones https://gist.github.com/xykong/6efdb1ed57535d18cb63aa8e20da3f4b Run script like this: ./k8sdump.sh -n jmeter -r deployments -o /data/workspace |
|
@mohamedelhabib @nathan-c @scones Nice, but what about none namespaced resources? Like pv,crd,psp,clusterrolebindings,clusterroles,sc? Its great to have them too. Second problem with repeated resources, which produce duplicate iterate. See output with and without sort | uniq (v1.16.10): |
|
What about the CRDs? |
Unfortunately this does not backup all the resources in the namespace |
and others
Given a kubernetes cluster that is running some number of pods, services, deployments, etc, I would like to generate one or more files ( yml format preferred) that would re-produce the current configuration, when applied to a new cluster.
My use case is a promotion system. I have my 'stack files' as yml files in a git repo, but I need to allow humans to approve changes before they are applied to the cluster.
One way to do this is to use an 'open loop' system. I can use tags or other mechanisms to determine which versions have been applied to the cluster, and then compare the latest version available with the latest deployed version.
The problem with the open-loop system is that it does not consider that changes could have been made outside the files, or that changes applied could have had problems, etc.
If I could extract the 'equivalent' files from a running cluster, I could compare them with the ones that are about to be applied. This is a much stronger, 'closed loop' system-- it is able to correctly understand what will happen when the changes are applied, even if we have lost track of the real target state.
if there were such a thing as kubectl apply -f --dry-run, which lists only the changes that will be made, rather than actually doing the changes, that would work as well. That is already being discussed over at issue #11488
Does anyone have thoughts on this? We are new to kubernetes, but we have created the the functionality I'm describing above for our RedHat/Satellite rpm-based deployments, so i want to re-create it in K8s. Of course, in k8s, we have the complexity that the infrastructure itself can change, not just installed package versions!
The text was updated successfully, but these errors were encountered: