posts/cloud-init-cloud-image/

[giscus[bot]]

posts/cloud-init-cloud-image/

Documentation for Techno Tim YouTube videos for all your copy pasta needs!

https://technotim.live/posts/cloud-init-cloud-image/

[B1GWH1T3]

Thanks Tim for the awesome guides!

Not sure if its just me but I get this error when trying to mount the disk

Successfully imported disk as 'unused0:fastboi:8000/vm-8000-disk-0.raw'
root@REDQUEENHQPVE1:~# qm set 8000 --scsihw virtio-scsi-pci --scsi0 fastboi:vm-8000-disk-0
unable to parse directory volume name 'vm-8000-disk-0'

What I found is for some reason it creates it under the VM ID
root@REDQUEENHQPVE1:~# qm set 8000 --scsihw virtio-scsi-pci --scsi0 fastboi:8000/vm-8000-disk-0.raw
update VM 8000: -scsi0 fastboi:8000/vm-8000-disk-0.raw -scsihw virtio-scsi-pci

Just a note for anyone struggling with that portion

[andriesh]

Awesome tutorial but to make it even better I would suggest to do this:
Before importing the image as disk to the VM you should install the qemu-guest-tools package like this:

Install necessary tools:
apt install libguestfs-tools

Then import install the package:
virt-customize -a focal-server-cloudimg-amd64.img --install qemu-guest-agent

And don't forget to enable the agent in the settings to be able to see the IP of your newly created VM:
qm set 8000 --agent 1,fstrim_cloned_disks=1

[qeba]

maybe to play safe, since each time when running the virt-customize it will give a machine-id, as below;

[   3.9] Setting the machine ID in /etc/machine-id

maybe can added this, after install that qemu agent.

virt-customize -a focal-server-cloudimg-amd64.img --run-command "sudo truncate -s 0 /etc/machine-id /var/lib/dbus/machine-id"

so that machine ID will be clear and it will insert itself when boot up.

[MDHMatt]

just tried it without running the command to remove machine id and no issues with ip address ect

[sscolnick-IT]

Was running into issues with terraform because this tutorial does not install qemu-guest-agent. When attempting to run virt-customize -a focal-server-cloudimg-amd64.img --install qemu-guest-agent on the cloud image it looks like my proxmox server was having issues locating the ubuntu repos?

Any Ideas how to fix this? do I need to add these ubuntu repos to my /etc/apt/sources.list on the proxmox server?

root@proxmox:~# virt-customize -a focal-server-cloudimg-amd64.img --install qemu-guest-agent
[   0.0] Examining the guest ...
[   3.7] Setting a random seed
virt-customize: warning: random seed could not be set for this type of 
guest
[   3.7] Installing packages: qemu-guest-agent
Err:1 http://security.ubuntu.com/ubuntu focal-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:2 http://archive.ubuntu.com/ubuntu focal InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package qemu-guest-agent
Err:1 http://security.ubuntu.com/ubuntu focal-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:2 http://archive.ubuntu.com/ubuntu focal InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package qemu-guest-agent
virt-customize: error: 
        export DEBIAN_FRONTEND=noninteractive
        apt_opts='-q -y -o Dpkg::Options::=--force-confnew'
        apt-get $apt_opts update
        apt-get $apt_opts install 'qemu-guest-agent'
      : command exited with an error

If reporting bugs, run virt-customize with debugging enabled and include 
the complete output:

  virt-customize -v -x [...]

[sscolnick-IT]

Found the fix for my issue, removed my internal DNS from /etc/resolv.conf on my proxmox server and it was able to locate the repo and download qemu-guest-agent!

[Gabgobie]

Take a look at the comment I left a few minutes ago. That is my preferred way of adding packages to my machines and configuring more parameters to make sure everything is as it should be, no tinkering with the original image required.

[MDHMatt]

If the image file creates a too small of a hard drive space add this after downloading your image file (adds 2Gb to the drive)

qemu-img resize jammy-server-cloudimg-amd64.img +2G

[FedorPRO]

Thanks Tim!

I am trying to automate the whole process and write a bash script for this. And I think it's worth making changes, as in the official example (https://pve.proxmox.com/wiki/Cloud-Init_Support#_preparing_cloud_init_templates)

"--scsihw virtio-scsi-pci " should be used in "qm create "

In this case, the correct controller will be used and the disk will connect to it without problems

Changes like this:
qm create 8000 --memory 2048 --core 2 --name ubuntu-cloud --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-pci

qm set 8000 --scsi0 local-lvm:8000/vm-8000-disk-0.raw

[thisisbramiller]

bump!

[thisisbramiller]

Working through this problem now. I noticed there is a difference depending upon which filesystem you are deploying to.
LVM-Thin works just fine and outputs as: vm-9000-disk-0
while NFS gets changed to: EX2Ultra:9000/vm-9000-disk-0.raw

[Gabgobie]

To further enhance this you can add a vendor config. With cloud-init you can attatch four config files, three of which proxmox will manage through the UI. The vendor config is unset but you can add one without compromising the UIs functionality. It also ensures that you still have a clean image that has never been booted which is not the case when you use virt-customize. In the vendor config you can then use the full range of cloud-inits wealth of features.

First you will need to enable a snipplets storage on your PVE host.

Then you can add your vendor config there. I for example called mine the template-ID combined with a description of what it is: 100000-civendor.yaml

To give an example of how this can look, here is mine:
cat /var/lib/vz/snippets/100000-civendor.yaml

#cloud-config

# don't think these two are needed as cloud-init seems to update the machine on first boot anyways but I still left them in there because I didn't take the time to confirm
package_update: true
package_upgrade: true

# installing some exemplary packages

packages:
  - qemu-guest-agent
  - git

# making sure qemu-guest-agent is running
runcmd:
    - systemctl start qemu-guest-agent

timezone: "Continent/City"

ca_certs:
  # If present and set to True, the 'remove_defaults' parameter will either
  # disable all the trusted CA certifications normally shipped with
  # Alpine, Debian or Ubuntu. On RedHat, this action will delete those
  # certificates.
  # This is mainly for very security-sensitive use cases - most users will not
  # need this functionality.
  remove_defaults: false

  # If present, the 'trusted' parameter should contain a certificate (or list
  # of certificates) to add to the system as trusted CA certificates.
  # Pay close attention to the YAML multiline list syntax.  The example shown
  # here is for a list of multiline certificates.

  # adding my own certs because I'm running a local CA and I want my machines to trust my certs by default
  trusted: 
  - |
   -----BEGIN CERTIFICATE-----
   ...
   -----END CERTIFICATE-----
  - |
   -----BEGIN CERTIFICATE-----
   ...
   -----END CERTIFICATE-----

I highly recommend all of you to include this into your project as it will keep you from having to configure your VMs individually after boot without having to compromise their integrity. I would also encourage you to look into theh cloud-init documentation to find more usefull data you can pass to your image.

[Gabgobie]

Happy make at least one person's experience better!

https://pve.proxmox.com/wiki/Cloud-Init_Support

qm cloudinit dump <VMID> user/network/meta

However I would advise you not to mess with them and at most take a look to know what you can ommit from your vendor config. If you replace these with your own config, you will loose functionality in the WebUI. That's why I put everything that is not configurable with the GUI into the vendor config.

I also took this opportunity to update my markdown syntax in the original comment so everything is displayed correctly. Now it can be copied without screwing things up.

Best,
Gab

[userpasta]

Ack, make sense! I think that I can edit cloud-init drive config after cloning the image, is that correct?

[Gabgobie]

In theory yes but that kinda kills the point of using cloudinit.

If you change the config after cloning you would have a machine that is different from the other machines. don't forget to regenerate the image after making changes.

If you want a machine that is different from the others I'd recommend you to only add the things that the machines should have in common to your cloud-config and doing everything else manually. The true power of cloud-init shows when you want to deploy a lot of machines that are supposed to be identical, for example if you want to deploy 8 identical machines for k3s.

If you make a change to the linked vendor config this will of course also be transferred to the template. If you want a machine with a different config you should make another vendor config and link the new config to the already cloned machine. Also for all I know CI is only executed on the first boot so make your changes before booting the clone. Afterwards you will only be able to change the things that were defined in the Web UI using the regenerate image button as the machine will still grab the credentials and network configuration from there. However I am not an expert either and this is purely speculation based on my experiences. For a definitive answer you should consult the CI documentation.

To get back to the inital point: Theoretically you can change aspects about your cloudinit configuration after cloning but that is not the intended purpose, therefore my recommendation would be to create a template for every use case you have.

For example I would create one general template that I use for individual VMs.
In that templates config I would only hand over my local CAs certificate, tell it to install qemu-guest-agent and set my local timezone.

In another template, for example catered towards k3s nodes I would then add other relevant packages on top of the other config.

To achieve this I would copy the general template to a file with, except for vmid which I use as an identifier, the same name in the same folder and then link it to a new template. I also put the process of creating a new template into a simple bash script so I can be sure that they will all be configured propperly.

[userpasta]

Thanks Gab! This is extremely helpful.

If you change the config after cloning you would have a machine that is different from the other machines. don't forget to regenerate the image after making changes.
If you want a machine that is different from the others I'd recommend you to only add the things that the machines should have in common to your cloud-config and doing everything else manually. The true power of cloud-init shows when you want to deploy a lot of machines that are supposed to be identical, for example if you want to deploy 8 identical machines for k3s.

IIRC in the tutorial Tim creates a template after setting the cloud-init in the UI. My understanding is that it should be fine to:

• update cloud-init in the UI for the template and regenerate image (if I want to update the values long term)
• update cloud-init in the UI after full cloning the template and also regenerate the image, before the first boot.

As the UI cloud-init takes the priority and will be executed before vendor linked one. Are these correct assumptions?

For example I would create one general template that I use for individual VMs.
In that templates config I would only hand over my local CAs certificate, tell it to install qemu-guest-agent and set my local timezone.

This is the route I took.

I also put the process of creating a new template into a simple bash script so I can be sure that they will all be configured propperly.

Did the same thing in case I need to recreate the template with different values in the future or for a different distro.

I was also thinking about further automating:

1. Create random MAC address.
2. In my router add that address to DHCP under specific IP address (in my case it's a VMID)
3. Use Ansible's community.general.proxmox to copy my Ubuntu template, if needed update values and boot the VM.
4. Use Ansible to copy update configs like .bashrc, .vimrc, .tmux.conf etc. and install docker.

This way I could create and provision a new VM using Ansible by providing:

• MAC address
• new size of the image
• VMID

I could even go one step further and use Python + OPNsense API to automate (1) and (2).

All this hassle just to occasionally setup a VM 🤣 I spent almost a day learning Ansible to automate (4), but I don't regret it as it's really powerful tool.

[Gabgobie]

Please excuse my late reply. Currently studying for an exam so I try to stay away from distractions.

IIRC in the tutorial Tim creates a template after setting the cloud-init in the UI. My understanding is that it should be fine to:

I think so too but there's not really a difference between changing the settings in the GUI manually and setting them via CLI in your script. To me the script seems like the more comfortable solution.

As the UI cloud-init takes the priority and will be executed before vendor linked one. Are these correct assumptions?

If I understand you correctly that should be the case. As long as it is before the first boot you can change the cloud-init config at any time. After the first boot you will at least be able to change the parts that are available in the GUI. I don't know about the other settings.

I was also thinking about further automating:

1. Create random MAC address.
2. In my router add that address to DHCP under specific IP address (in my case it's a VMID)
3. Use Ansible's community.general.proxmox to copy my Ubuntu template, if needed update values and boot the VM.
4. Use Ansible to copy update configs like .bashrc, .vimrc, .tmux.conf etc. and install docker.

I haven't done anything with Ansible except for using Tim's k3s-ansible playbook. Another thing on my to-do list..
I am pretty sure that your machine will get a random mac adress assigned when you clone it so you won't need to worry about it. It would just need to register with your management system. For that you could use the runcmd: option in your vendor config and then use curl to make an API call but that would of course require you to have some kind of management platform where you can register your new machine that way. In my case, I took the easy route and just manually added the machines to the static leases on my DHCP (yes I have my interfaces set to use DHCP, I only use Static IPs on absolutely critical devices such as the hypervisor, the firewall and my truenas).

There is probably some proxmox command to output the name and the mac adress of your VMs based on VMID. You could use that to get your values after creation from the host. I am sure you can find a million more possible solutions. "Viele Wege führen nach Rom" as we say in Germany.

All in all ansible sounds like a fitting solution to your needs though. I just won't be able to help you with that. ^^

I could even go one step further and use Python + OPNsense API to automate (1) and (2).

Exactly!

Best,
Gab

Edit: Fixed syntax... 🤦‍♂️

[beast3500]

Im actually having an issue with accessing a clone over ssh. I confirmed that Open SH is installed, and I am able to login to other virtual machines not created through Tim's provisioning method.

"The client has disconnected from the server. Reason:
Unable to authenticate using any of the configured authentication methods."