ESXi Create VM from Shell

At last have found how VM creation can be created from shell scripts

Starting point - https://github.com/josenk/esxi-vm-create/blob/master/esxi-vm-create

There is an vim-cmd which holds all we need

Note: everything is done from within ESXi server

Here are some basics:

# list virtual machines
vim-cmd vmsvc/getallvms

# get virtual machine by vmid
vim-cmd vmsvc/get.summary 78

# get power status (where 78 is vmid)
vim-cmd vmsvc/power.getstate 78

# power management (hard)
vim-cmd vmsvc/power.on 78
vim-cmd vmsvc/power.off 78
vim-cmd vmsvc/power.reset 78

# power management (soft, requires vm tools to be installed)
vim-cmd vmsvc/power.shutdown 78
vim-cmd vmsvc/power.reboot 78

# delete virtual machine and its files
vim-cmd vmsvc/destroy 78

Also there is a set of commands around snapshots which also might be useful in some cases

ESXi VMX

From ESXi perspective virtual machine is an vmx file with its configuration and vmdx file with disk data

vmx is a ini like configuration file holding key value pairs, where values are always wrapped in double quotes

If you create VM and remove from it all devices it will look like:

.encoding = "UTF-8"
bios.bootRetry.delay = "10"
config.version = "8"
displayName = "test1"
firmware = "efi"
floppy0.present = "FALSE"
guestOS = "other5xlinux-64"
hpet0.present = "TRUE"
memSize = "1024"
nvram = "test1.nvram"
pciBridge0.present = "TRUE"
pciBridge4.functions = "8"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
powerType.powerOff = "default"
powerType.reset = "default"
powerType.suspend = "soft"
RemoteDisplay.maxConnections = "-1"
sched.cpu.affinity = "all"
sched.cpu.latencySensitivity = "normal"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.cpu.units = "mhz"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
svga.autodetect = "TRUE"
svga.present = "TRUE"
tools.syncTime = "FALSE"
tools.upgrade.policy = "manual"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforePowerOff = "TRUE"
toolScripts.beforeSuspend = "TRUE"
uefi.secureBoot.enabled = "TRUE"
uuid.bios = "56 4d 7a c3 90 63 48 ef-69 24 5b 13 98 01 81 96"
uuid.location = "56 4d 7a c3 90 63 48 ef-69 24 5b 13 98 01 81 96"
vc.uuid = "52 0e 56 ab da 71 20 00-99 c0 b5 96 8a 32 4e 7c"
virtualHW.version = "19"
vm.createDate = "1640534518499977"
vmci0.present = "TRUE"

You can create folder in ESXi, put this file there and register VM

I did tried to remove all non required options till I was able to register an vm and it seems that almost everything can be removed, I have stopped after some period where it loose any sence

There are set of generated parameters like uuid.bios, uuid.location, vm.createDate and most of others are just defaults

.encoding = "UTF-8"
config.version = "8"
displayName = "test1"
floppy0.present = "FALSE"
guestOS = "other5xlinux-64"
memSize = "1024"
numvcpus = "2"
virtualHW.version = "19"

Then I did tried to add existing vmdk to see what vmx will looklike

Here is what I got:

.encoding = "UTF-8"
config.version = "8"
displayName = "test1"
guestOS = "ubuntu-64"
memSize = "1024"
virtualHW.version = "19"

tools.upgrade.policy = "manual"
sched.cpu.units = "mhz"
sched.cpu.affinity = "all"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
tools.syncTime = "FALSE"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
ide0:0.fileName = "system.vmdk"
sched.ide0:0.shares = "normal"
sched.ide0:0.throughputCap = "off"
ide0:0.present = "TRUE"
sched.cpu.latencySensitivity = "normal"
tools.guest.desktop.autolock = "FALSE"

So ESXi once again added bunch of things, before powering on, did tried to remove them back and started with:

.encoding = "UTF-8"
config.version = "8"
displayName = "test1"
guestOS = "ubuntu-64"
memSize = "1024"
virtualHW.version = "19"

ide0:0.fileName = "system.vmdk"
ide0:0.present = "TRUE"

And everything worked out!

Having this and cloud init now it is possible to spin up bazillion of virtual machines in a seconds

Here is an example of docker host with two networks:

mkdir /vmfs/volumes/storage/test
vmkfstools --clonevirtualdisk /vmfs/volumes/storage/iso/system.vmdk --diskformat thin /vmfs/volumes/storage/test/test.vmdk
vmkfstools -X 20G /vmfs/volumes/storage/test/test.vmdk

tee /vmfs/volumes/storage/test/test.vmx > /dev/null <<EOT
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "19"
nvram = "test.nvram"
svga.present = "TRUE"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
floppy0.present = "FALSE"
RemoteDisplay.maxConnections = "-1"
numvcpus = "4"
memSize = "8192"
bios.bootRetry.delay = "10"
powerType.powerOff = "default"
powerType.suspend = "soft"
powerType.reset = "default"
tools.upgrade.policy = "manual"
sched.cpu.units = "mhz"
sched.cpu.affinity = "all"
sched.cpu.latencySensitivity = "normal"
scsi0.virtualDev = "lsilogic"
scsi0.present = "TRUE"
sata0.present = "TRUE"
usb.present = "TRUE"
ehci.present = "TRUE"
svga.autodetect = "TRUE"
ethernet0.virtualDev = "vmxnet3"
ethernet0.networkName = "wan"
ethernet0.addressType = "generated"
ethernet0.wakeOnPcktRcv = "FALSE"
ethernet0.uptCompatibility = "TRUE"
ethernet0.present = "TRUE"
ethernet1.virtualDev = "vmxnet3"
ethernet1.networkName = "lan"
ethernet1.addressType = "generated"
ethernet1.wakeOnPcktRcv = "FALSE"
ethernet1.uptCompatibility = "TRUE"
ethernet1.present = "TRUE"
sata0:0.startConnected = "FALSE"
sata0:0.autodetect = "TRUE"
sata0:0.deviceType = "atapi-cdrom"
sata0:0.fileName = "auto detect"
sata0:0.present = "TRUE"
displayName = "test"
guestOS = "ubuntu-64"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
tools.syncTime = "FALSE"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"


sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "test.vmdk"
scsi0:0.present = "TRUE"


guestinfo.metadata.encoding = "base64"
guestinfo.metadata = "$(echo -n '
network:
  version: 2
  ethernets:
    ens160:
      dhcp4: false
      dhcp6: false
      addresses:
        - 178.20.154.77/24
      gateway4: 178.20.154.254
        nameservers:
            addresses: [1.1.1.1, 8.8.8.8]
    ens196:
      dhcp4: false
      dhcp6: false
      addresses:
        - 192.168.106.11/24
' | openssl base64 | awk 'BEGIN{ORS="";} {print}')"
guestinfo.userdata.encoding = "base64"
guestinfo.userdata = "$(echo -n '#cloud-init
# note that "#cloud-config" must be very first line for everything to work

# hostname
preserve_hostname: false
hostname: test.mac-blog.org.ua
manage_etc_hosts: true

# enable ntp
ntp:
  enabled: true

# timezone
timezone: Europe/Kiev

# disable root login via ssh
disable_root: true

# optional, additional groups
groups:
  # because we gonna install docker
  - docker

users:
  - name: mac
    # allows password authnetication when set to false, true by default
    lock_passwd: False
    # password hash, created by `mkpasswd --method=SHA-512 --rounds=4096`, read the docs before complaining security against plain
    # passwd: $6$rounds=4096$4Jh2rwf9h2jM9TbQ$.CTSPJPIoIOUwKVo4A2Er19Deu945m/oD.JXVEGNH9g/piK.motblke/kpyPQ0npNKF.jZjzi61ZSBPGNbJyK/
    # alternative approach with plain text password
    plain_text_passwd: mac
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyfdNzj4mbl0PiOA7VQxeO8cR/U9u2SdRFZaZlRatdJZ1sYYa/Q0jMDOsr7oklftX49yGpuW1sBtWLevaKtcKz3kHkInFvlebFeFB9uPmMfo8gd+QLuLc8LeOQgn4wlSPEuizDxuMd1Sz1frZ7r7QUH8HdhnCO/FXtIg+c8DHsAVQSxapfalVCv4z4EiC3le8SxlZjxf/KaVmIdYOpbLf0GBP/gP5ObFYxvtTx7Rp1lM1ZSXYVGFBpXC15WYMHKq+LfQ8D0WpwiOIXw0723k+CSUb3tHS5a8Hz0GUtjMOizclIeiv503DqZAA8RxtWLfLUFdCeUgEyvIu430s4xBqSw== rsa-key-20140409
    groups:
      - adm
      - cdrom
      - sudo
      - dip
      - plugdev
      - lxd
      # add user to docker group
      - docker
    sudo:
      # allow sudo
      - ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash

# enforce password change
chpasswd:
  list:
    - mac:mac

# optional, additional apt sources, docker
apt:
  sources:
    docker.list:
      source: deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

# apt update && apt upgrade, reboot if needed
package_update: true
package_upgrade: true
package_reboot_if_required: true

# install packages
packages:
  - docker-ce
  - docker-ce-cli
  - containerd.io
' | openssl base64 | awk 'BEGIN{ORS="";} {print}')"
EOT

vim-cmd solo/registervm /vmfs/volumes/storage/test/test.vmx
vim-cmd vmsvc/getallvms
vim-cmd vmsvc/power.on 44
# vim-cmd vmsvc/power.off 44
# vim-cmd vmsvc/destroy 44

Notes:

  • cloud init or how to bootstrap ubuntu vm
  • be very carefull with yaml whitespace hell, got everything working after bazillion of attempts just to realize that I have wrong spaces in network configuration
  • if second interface not needed just remove ethernet1.* from vmx
  • options to change nvram, displayName, scsi0:0.fileName also hostname in guestinfo.userdata
  • do not forget to rename networkName of ethernetX to corresponding ones
  • esxi does not have built int base64 utility so we are using echo -n "hello" | openssl base64 instead, and because there is no tr utility to remove new lines from base64 we use awk 'BEGIN{ORS="";} {print}')