ESXi Cloud Init Bootstrap Ubuntu

How to automatically provision Ubuntu 20.04 in ESXi with help of Cloud Init

Ubuntu Cloud Image VMDK

Ubuntu cloud images are preconfigured distributions to be run as virtual machines and to be bootstraped with cloud init

Before anything else navigate to:

https://cloud-images.ubuntu.com/

And traverse to desider ubuntu version to download its vmdk file, in my case it is here:

https://cloud-images.ubuntu.com/focal/current/

vmdk which is only 500mb of size

After downloading vmdk upload it to esxi server somewhere in storage

ESXi Virtual Machine

The next step is to create virtual machine as usual, execpt that we do not need any disks to be created, so remove them and after creating vm attach existing disk and choose uploaded vmdk file

Notes:

in my case for experiments I am copying to to vm folder, so original vmdk can be reused many times
note that vmdk size is 500mb, but disk will be 10gb, thats because by default cloud init has all required modules enabled so guest operating system will expand itself to take all disk space
disk size can be changed after saving vm and reopening its settings

If you will try to boot VM it will load but you wont be able to login, because there is no users and\or passwords created, we need to configure at least this to move further

ESXi Cloud Init GuestInfo UserData

Now the tricky part, go to virtual machine advanced settings and add two properties:

Key	Value
`guestinfo.userdata.encoding`	`base64`
`guestinfo.userdata`	base64 userdata.yml

Note:

Value is a base64 encoded cloud init file contents
#cloudinit must be very first line for everything to work
cloud init applied once, so if you did something wrong just start from scratch
there is 500kb limit for this fields if your cloud init is bigger there is a way to pass base64 gzip content just google for it

Here is a starting point:

#cloud-config
users:
  - name: mac
    # passwd: $6$rounds=4096$4Jh2rwf9h2jM9TbQ$.CTSPJPIoIOUwKVo4A2Er19Deu945m/oD.JXVEGNH9g/piK.motblke/kpyPQ0npNKF.jZjzi61ZSBPGNbJyK/
    plain_text_passwd: secret
    groups: sudo
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    lock_passwd: false
# guestinfo.userdata.encoding: base64
# guestinfo.userdata: I2Nsb3VkLWNvbmZpZwp1c2VyczoKICAtIG5hbWU6IG1hYwogICAgIyBwYXNzd2Q6ICQ2JHJvdW5kcz00MDk2JDRKaDJyd2Y5aDJqTTlUYlEkLkNUU1BKUElvSU9Vd0tWbzRBMkVyMTlEZXU5NDVtL29ELkpYVkVHTkg5Zy9waUsubW90YmxrZS9rcHlQUTBucE5LRi5qWmp6aTYxWlNCUEdOYkp5Sy8KICAgIHBsYWluX3RleHRfcGFzc3dkOiBzZWNyZXQKICAgIGdyb3Vwczogc3VkbwogICAgc3VkbzogQUxMPShBTEwpIE5PUEFTU1dEOkFMTAogICAgc2hlbGw6IC9iaW4vYmFzaAogICAgbG9ja19wYXNzd2Q6IGZhbHNl

And to grab its base64: cat minimal.yml | base64

Now we should be able to at least login to system

Few useful files:

# general overview
cat /var/run/cloud-init/status.json

# in my case was empty
cat /var/run/cloud-init/result.json

# stdout
cat /var/log/cloud-init-output.log

# debug
cat /var/log/cloud-init.log

Second run of cloud init from withing virtual machine

https://cloudinit.readthedocs.io/en/latest/topics/debugging.html#running-single-cloud-config-modules

While we are inside VM it is a good time to configure cloud init further, there is /etc/cloud/cloud.cfg file, where we can append something and after that run something like:

cloud-init clean --logs

cloud-init single --name cc_ssh --frequency always
cloud-init single --name cc_package_update_upgrade_install
cloud-init single --name cc_ntp
cloud-init single --name cc_timezone

Here you can see that first of all you need to cleanup cloud init state, after that you can rerun concrete module.

This is how we can prepare our cloud init

In my case for demo purposes I ended up with:

#cloud-config

# note that "#cloud-config" must be very first line for everything to work

# guestinfo.userdata: cat cloudinit.yml | base64 | pbcopy
# guestinfo.userdata.encoding: base64

# networking
# this one should be added to:
# guestinfo.metadata:
# guestinfo.metadata.encoding
# ----------
# network:
#   version: 2
#   ethernets:
#     nics:
#       match:
#         name: ens*
#       # DHCP:
#       # ----------
#       # dhcp4: yes
#       # ----------
#       # STATIC:
#       # ----------
#       addresses:
#         - 192.168.106.22/24
#       gateway4: 192.168.106.1
#       nameservers:
#         # search: [lab, home]
#         addresses: [8.8.8.8, 1.1.1.1]

# hostname
preserve_hostname: false
hostname: ub3.marchenko.net.ua
manage_etc_hosts: true

# enable ntp
ntp:
  enabled: true

# timezone
timezone: Europe/Kiev

# disable root login via ssh
disable_root: true

# optional, additional groups
groups:
  # because we gonna install docker
  - docker

users:
  - name: mac
    # allows password authnetication when set to false, true by default
    lock_passwd: False
    # password hash, created by `mkpasswd --method=SHA-512 --rounds=4096`, read the docs before complaining security against plain
    # passwd: $6$rounds=4096$4Jh2rwf9h2jM9TbQ$.CTSPJPIoIOUwKVo4A2Er19Deu945m/oD.JXVEGNH9g/piK.motblke/kpyPQ0npNKF.jZjzi61ZSBPGNbJyK/
    # alternative approach with plain text password
    plain_text_passwd: secret
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyfdNzj4mbl0PiOA7VQxeO8cR/U9u2SdRFZaZlRatdJZ1sYYa/Q0jMDOsr7oklftX49yGpuW1sBtWLevaKtcKz3kHkInFvlebFeFB9uPmMfo8gd+QLuLc8LeOQgn4wlSPEuizDxuMd1Sz1frZ7r7QUH8HdhnCO/FXtIg+c8DHsAVQSxapfalVCv4z4EiC3le8SxlZjxf/KaVmIdYOpbLf0GBP/gP5ObFYxvtTx7Rp1lM1ZSXYVGFBpXC15WYMHKq+LfQ8D0WpwiOIXw0723k+CSUb3tHS5a8Hz0GUtjMOizclIeiv503DqZAA8RxtWLfLUFdCeUgEyvIu430s4xBqSw== rsa-key-20140409
    groups:
      - adm
      - cdrom
      - sudo
      - dip
      - plugdev
      - lxd
      # add user to docker group
      - docker
    sudo:
      # allow sudo
      - ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash

# optional, additional apt sources, docker
apt:
  sources:
    docker.list:
      source: deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

# apt update && apt upgrade, reboot if needed
package_update: true
package_upgrade: true
package_reboot_if_required: true

# install packages
packages:
  - docker-ce
  - docker-ce-cli
  - containerd.io
# guestinfo.userdata.encoding: base64
# guestinfo.userdata: I2Nsb3VkLWNvbmZpZwoKIyBub3RlIHRoYXQgIiNjbG91ZC1jb25maWciIG11c3QgYmUgdmVyeSBmaXJzdCBsaW5lIGZvciBldmVyeXRoaW5nIHRvIHdvcmsKCiMgZ3Vlc3RpbmZvLnVzZXJkYXRhOiBjYXQgY2xvdWRpbml0LnltbCB8IGJhc2U2NCB8IHBiY29weQojIGd1ZXN0aW5mby51c2VyZGF0YS5lbmNvZGluZzogYmFzZTY0CgojIG5ldHdvcmtpbmcKIyB0aGlzIG9uZSBzaG91bGQgYmUgYWRkZWQgdG86CiMgZ3Vlc3RpbmZvLm1ldGFkYXRhOgojIGd1ZXN0aW5mby5tZXRhZGF0YS5lbmNvZGluZwojIC0tLS0tLS0tLS0KIyBuZXR3b3JrOgojICAgdmVyc2lvbjogMgojICAgZXRoZXJuZXRzOgojICAgICBuaWNzOgojICAgICAgIG1hdGNoOgojICAgICAgICAgbmFtZTogZW5zKgojICAgICAgICMgREhDUDoKIyAgICAgICAjIC0tLS0tLS0tLS0KIyAgICAgICAjIGRoY3A0OiB5ZXMKIyAgICAgICAjIC0tLS0tLS0tLS0KIyAgICAgICAjIFNUQVRJQzoKIyAgICAgICAjIC0tLS0tLS0tLS0KIyAgICAgICBhZGRyZXNzZXM6CiMgICAgICAgICAtIDE5Mi4xNjguMTA2LjIyLzI0CiMgICAgICAgZ2F0ZXdheTQ6IDE5Mi4xNjguMTA2LjEKIyAgICAgICBuYW1lc2VydmVyczoKIyAgICAgICAgICMgc2VhcmNoOiBbbGFiLCBob21lXQojICAgICAgICAgYWRkcmVzc2VzOiBbOC44LjguOCwgMS4xLjEuMV0KCiMgaG9zdG5hbWUKcHJlc2VydmVfaG9zdG5hbWU6IGZhbHNlCmhvc3RuYW1lOiB1YjMubWFyY2hlbmtvLm5ldC51YQptYW5hZ2VfZXRjX2hvc3RzOiB0cnVlCgojIGVuYWJsZSBudHAKbnRwOgogIGVuYWJsZWQ6IHRydWUKCiMgdGltZXpvbmUKdGltZXpvbmU6IEV1cm9wZS9LaWV2CgojIGRpc2FibGUgcm9vdCBsb2dpbiB2aWEgc3NoCmRpc2FibGVfcm9vdDogdHJ1ZQoKIyBvcHRpb25hbCwgYWRkaXRpb25hbCBncm91cHMKZ3JvdXBzOgogICMgYmVjYXVzZSB3ZSBnb25uYSBpbnN0YWxsIGRvY2tlcgogIC0gZG9ja2VyCgp1c2VyczoKICAtIG5hbWU6IG1hYwogICAgIyBhbGxvd3MgcGFzc3dvcmQgYXV0aG5ldGljYXRpb24gd2hlbiBzZXQgdG8gZmFsc2UsIHRydWUgYnkgZGVmYXVsdAogICAgbG9ja19wYXNzd2Q6IEZhbHNlCiAgICAjIHBhc3N3b3JkIGhhc2gsIGNyZWF0ZWQgYnkgYG1rcGFzc3dkIC0tbWV0aG9kPVNIQS01MTIgLS1yb3VuZHM9NDA5NmAsIHJlYWQgdGhlIGRvY3MgYmVmb3JlIGNvbXBsYWluaW5nIHNlY3VyaXR5IGFnYWluc3QgcGxhaW4KICAgICMgcGFzc3dkOiAkNiRyb3VuZHM9NDA5NiQ0SmgycndmOWgyak05VGJRJC5DVFNQSlBJb0lPVXdLVm80QTJFcjE5RGV1OTQ1bS9vRC5KWFZFR05IOWcvcGlLLm1vdGJsa2Uva3B5UFEwbnBOS0Yualpqemk2MVpTQlBHTmJKeUsvCiAgICAjIGFsdGVybmF0aXZlIGFwcHJvYWNoIHdpdGggcGxhaW4gdGV4dCBwYXNzd29yZAogICAgcGxhaW5fdGV4dF9wYXNzd2Q6IHNlY3JldAogICAgc3NoX2F1dGhvcml6ZWRfa2V5czoKICAgICAgLSBzc2gtcnNhIEFBQUFCM056YUMxeWMyRUFBQUFCSlFBQUFRRUF5ZmROemo0bWJsMFBpT0E3VlF4ZU84Y1IvVTl1MlNkUkZaYVpsUmF0ZEpaMXNZWWEvUTBqTURPc3I3b2tsZnRYNDl5R3B1VzFzQnRXTGV2YUt0Y0t6M2tIa0luRnZsZWJGZUZCOXVQbU1mbzhnZCtRTHVMYzhMZU9RZ240d2xTUEV1aXpEeHVNZDFTejFmclo3cjdRVUg4SGRobkNPL0ZYdElnK2M4REhzQVZRU3hhcGZhbFZDdjR6NEVpQzNsZThTeGxaanhmL0thVm1JZFlPcGJMZjBHQlAvZ1A1T2JGWXh2dFR4N1JwMWxNMVpTWFlWR0ZCcFhDMTVXWU1IS3ErTGZROEQwV3B3aU9JWHcwNzIzaytDU1ViM3RIUzVhOEh6MEdVdGpNT2l6Y2xJZWl2NTAzRHFaQUE4Unh0V0xmTFVGZENlVWdFeXZJdTQzMHM0eEJxU3c9PSByc2Eta2V5LTIwMTQwNDA5CiAgICBncm91cHM6CiAgICAgIC0gYWRtCiAgICAgIC0gY2Ryb20KICAgICAgLSBzdWRvCiAgICAgIC0gZGlwCiAgICAgIC0gcGx1Z2RldgogICAgICAtIGx4ZAogICAgICAjIGFkZCB1c2VyIHRvIGRvY2tlciBncm91cAogICAgICAtIGRvY2tlcgogICAgc3VkbzoKICAgICAgIyBhbGxvdyBzdWRvCiAgICAgIC0gQUxMPShBTEwpIE5PUEFTU1dEOkFMTAogICAgc2hlbGw6IC9iaW4vYmFzaAoKIyBvcHRpb25hbCwgYWRkaXRpb25hbCBhcHQgc291cmNlcywgZG9ja2VyCmFwdDoKICBzb3VyY2VzOgogICAgZG9ja2VyLmxpc3Q6CiAgICAgIHNvdXJjZTogZGViIFthcmNoPWFtZDY0XSBodHRwczovL2Rvd25sb2FkLmRvY2tlci5jb20vbGludXgvdWJ1bnR1ICRSRUxFQVNFIHN0YWJsZQogICAgICBrZXlpZDogOURDODU4MjI5RkM3REQzODg1NEFFMkQ4OEQ4MTgwM0MwRUJGQ0Q4OAoKIyBhcHQgdXBkYXRlICYmIGFwdCB1cGdyYWRlLCByZWJvb3QgaWYgbmVlZGVkCnBhY2thZ2VfdXBkYXRlOiB0cnVlCnBhY2thZ2VfdXBncmFkZTogdHJ1ZQpwYWNrYWdlX3JlYm9vdF9pZl9yZXF1aXJlZDogdHJ1ZQoKIyBpbnN0YWxsIHBhY2thZ2VzCnBhY2thZ2VzOgogIC0gZG9ja2VyLWNlCiAgLSBkb2NrZXItY2UtY2xpCiAgLSBjb250YWluZXJkLmlvCg==

ESXi Cloud Init Networking

To configure networking we need to do almost same steps but now with metadata, so add to VM following:

Key	Value
`guestinfo.metadata.encoding`	`base64`
`guestinfo.metadata`	base64 metadata.yml

network:
  version: 2
  ethernets:
    nics:
      match:
        name: ens*
      dhcp4: false
      dhcp6: false
      addresses:
        - 192.168.106.22/24
      gateway4: 192.168.106.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1]
# first ens160, second ens192

# guestinfo.metadata: cat metadata.yml | base64 | pbcopy
# guestinfo.metadata.encoding: base64

# guestinfo.metadata.encoding: base64
# guestinfo.metadata: bmV0d29yazoKICB2ZXJzaW9uOiAyCiAgZXRoZXJuZXRzOgogICAgbmljczoKICAgICAgbWF0Y2g6CiAgICAgICAgbmFtZTogZW5zKgogICAgICBkaGNwNDogZmFsc2UKICAgICAgZGhjcDY6IGZhbHNlCiAgICAgIGFkZHJlc3NlczoKICAgICAgICAtIDE5Mi4xNjguMTA2LjIyLzI0CiAgICAgIGdhdGV3YXk0OiAxOTIuMTY4LjEwNi4xCiAgICAgIG5hbWVzZXJ2ZXJzOgogICAgICAgIGFkZHJlc3NlczogWzguOC44LjgsIDEuMS4xLjFdCg==

Notes:

there is no need for any special first line in this file
there is same 500kb limit as for user data
seems like first network card is always ens160 and second ens192

With all this in please we should have bootstraped docker host

More info about available modules can be found here:

https://cloudinit.readthedocs.io/en/latest/topics/modules.html

And here is how to automate esxi vm