Stjepan Groš

ESX(i) stuff

I'm using VMware from around 2002, or something like that. First, I used Workstation and then, when it was released as a free product, Server. Now I also have access to a production ESX cluster so I'm able to try some features that otherwise, I wouldn't be. :) Furthermore, after VMware released ESXi product as a freeware I'm also using it. Here, I'll write about some experiences I gained with those products. If anyone finds this useful, I'll be glad to hear it...

Note that the ESX server is only one of several competing products on the market today, e.g. Sun's VirtualBox, Hyper-V from Microsoft and Xen. I'll collect information about those and other products and compare them on a separate page.

Some tasks you can done on ESXi make your installation unsupported, and probably VMware not particularly happy either. So, you might think better before doing them. Those tasks are listed on a separate page to distinguish them from "supported" stuff listed here.

Using CLI to manage ESX

I'm addicted to the command line and I equally so don't bare M$ Windows and it's way of doing things (clicky, clicky... :)). So, it's a big problem for me that Virtual Center Server has to be a Windows machine. Furthermore, the problem is that Virtual Center Client is a Windows only application, even though I didn't try to run it under Wine. While I'm eagerly waiting for a Linux versions of those tools and functionalities there are some ways to manage ESX via command line on a Linux/Unix machine. What you have to do is to install RCLI tool from the VMware. Note that there is only download for 32-bit Linux installations and if you have 64-bit OS then you'll have to search a bit on the Google, or take a look on the hacks.

Listing all the registered virtual machines

You'll have to do this from a machine that has RCLI installed, i.e. there is no appropriate tool on the ESXi itself. Maybe this could be a good task, to install RCLI on ESXi? Anyway, to list all the registered virtual machines on a remote ESXi host, run the following command: 

$ vmware-cmd -H 192.168.1.2 -U root -l
Enter password: 

/vmfs/volumes/46573040-e562729a-edae-239487324364/vmachine1/vmachine1.vmx
/vmfs/volumes/46573040-e562729a-ed1e-239487324364/vmachine2/vmachine2.vmx
/vmfs/volumes/46573040-e562729a-edae-239487324364/vmachine3/vmachine3.vmx
/vmfs/volumes/46573040-e562729a-edae-239487324364/vmachine4/vmachine4.vmx
/vmfs/volumes/46573040-e562729a-edae-239487324364/vmachine5/vmachine5.vmx

From the previous output we see that on the target host with the IP address 192.168.1.2 there are 5 registered virtual machines with appropriate names. Each line represents a name of a configuration file that is used as a argument to the vmware-cmd command when issuing commands to a specific virtual machines.

There is also another way to list all virtual machines, albeit little bit less functional, but faster: 

~ # esxcli vms vm list

Finding uptime of virtual machine

To find out for how long a virtual machine has been powered up, you can use getuptime subcommand of the vmware-cmd command like this: 

$ vmware-cmd -H 192.168.1.2 -U root /vmfs/volumes/46573040-e562729a-ed1e-239487324364/vmachine2/vmachine2.vmx getuptime
Enter password: 
getuptime() = 439710$

The last line tells you how many seconds has elapsed since the machine was powered on. The dollar sign is actually a bug in vmware-cmd that doesn't echo new line before exiting.

Creating new VLAN on ESXi

Suppose you have a virtual machine that has to be included into separate, VLAN, while all the other virtual machines either don't use VLANs, or, use them but you need a new one. We'll assume that your network administrator gave you VLAN ID you have to use.

VLANs in ESX are implemented on a virtual switch with port groups. In other words, the process of placing some virtual machine into specific VLAN consists of the following two steps:

  1. Create new port group on a switch
  2. Assign virtual machine to a newly created port group

The first step is acomplished with the command esxcfg-vswitch, i.e. you have to execute the following command on the ESXi server: 

~ # esxcfg-vswitch -p <name of the portgroup> -v <VLAN ID>

You can check that the port group has been created, as well as the existing port groups, by running the following command: 

~ # esxcfg-vswitch -l
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks   
vSwitch0       64          8           64                1500    vmnic0    

  PortGroup Name      VLAN ID  Used Ports  Uplinks   
  DMZ                 12       1           vmnic0    
  VM Network          0        3           vmnic0    
  Management Network  0        1           vmnic0

As you can see, we have three networks, i.e. port groups, on the virtual switch. The DMZ port group is in VLAN 12 which is used to route the network traffic to a specific port of a firewall.

netstat/arp/lsof/etc on ESXi

Each Linux installation has several extremely useful command which help in debugging system and networking problems. ESXi doesn't have those commands, neither the ones installed from Linux would work because of a different architecture (more specifically, differences in /proc system).

Still, not all hope is lost. I found some ways around.

Adding new iSCSI disk

To add iSCSI target (a.k.a. disk) to ESXi using only command line tools use the following steps. Note that all the steps are executed from the remote host unless told otherwise.

  1. If software iSCSI is not already enabled on your ESXi machine, enable it using the following command: 

    $ vicfg-iscsi --server 172.16.5.129 --username root -E -e
Enter password:
Software iSCSI is enabled.
$

  2. To determine HBA name (name of the iSCSI adapter) that is used in subsequent commands issue the following command: 

    $ vicfg-iscsi --server 172.16.5.129 --username root -H -l
Enter password:
vmhba33          iSCSI Software Adapter
$

    In this example the adapter is named vmhba33.

  3. Add target to the ESXi host. Note that this doesn't initiate scanning or connection yet. It only informs ESXi about target that it will use: 

    $ vicfg-iscsi --server 172.16.5.129 --username root -S -a -i 10.0.101.2 --name iqn.2006-01.com.openfiler:tsn.0d56ddacdb1d vmhba33
Enter password:
Adding static discovery target 10.0.101.2:3260, iqn = iqn.2006-01.com.openfiler:tsn.0d56ddacdb1d, ...
$

  4. The target probably requires initiator to authenticate before being allowed to access disk. Authentication options are set using the following command: 

    $ vicfg-iscsi --server 172.16.5.129 --username root -A -c chapPreferred -m CHAP -u iscsi -w iscsi0 vmhba33
Enter password:
$

  5. Now, tell the ESXi to connect to target and scan for the available disks: 

    $ vicfg-rescan --server 172.16.5.129 --username root vmhba33
Enter password:
Scan operation succeeded.
$

  6. You can see if iSCSI disk is visible by issuing the following command: 

    $ vicfg-scsidevs --server 172.16.5.129 --username root -c
Enter password:
Device UID                                             Device Type  Console Device                                                             Size       Plugin   Display Name
mpx.vmhba32:C0:T0:L0                                   cdrom        /vmfs/devices/genscsi/mpx.vmhba32:C0:T0:L0                                 0MB        NMP      Local NECVMWar CD-ROM (mpx.vmhba32:C0:T0:L0)
mpx.vmhba1:C0:T0:L0                                    disk         /vmfs/devices/disks/mpx.vmhba1:C0:T0:L0                                    8192MB     NMP      Local VMware, Disk (mpx.vmhba1:C0:T0:L0)
t10.F405E46494C45400F6079376A737D263A46614D2F4552565   disk         /vmfs/devices/disks/t10.F405E46494C45400F6079376A737D263A46614D2F4552565   95392MB    NMP      OPNFILER iSCSI Disk (t10.F405E46494C45400F6079376A737D263A46614D2F4552565)
$

    The iSCSI disk is named t10.F405E46494C45400F6079376A737D263A46614D2F4552565 and if you connect to ESXi console, you'll find this device in /dev/disks directory.

  7. Before being able to use this disk you have to partition it. This has to be done from the command line using fdisk command. Start fdisk with: 

    $ fdisk /dev/disks/t10.F405E46494C45400F6079376A737D263A46614D2F4552565

    Now proceed as you would on any Linux distribution. Note that the partition type has to be fb otherwise tools will refuse to use it in the subsequent steps. After successful partitioning you'll find new device nodes, something like t10.F405E46494C45400F6079376A737D263A46614D2F4552565:1 in case you made only first partition.

  8. The final step is to make VMFS filesystem and mount new filesystem. This can be done from a remote host or from ESXi itself. This example is from running tool on the ESXi itself: 

    ~ # vmkfstools --createfs vmfs3 --setfsname datastore2 t10.F405E46494C45400F6079376A737D263A46614D2F4552565:1

    9. If you take a look now in the /vmfs/volumes directory you'll find there datastore2 subdirectory with a new disk.

Remove iSCSI target

To remove iSCSI target issue the following command: 

$ vicfg-iscsi --server 172.16.5.129 --username root -S -r -i 10.0.101.2 -n iqn.2006-01.com.openfiler:tsn.0d56ddacdb1d vmhba33Enter password: 
Removing static discovery target 10.0.101.2:3260, iqn = iqn.2006-01.com.openfiler:tsn.0d56ddacdb1d, ...

This will also unmount datastore(s) on the given target.

Machine migration between ESX(i) and ESX(i)

The title of this subsection can be read in four different versions (e.g. ESX -> ESXi, ESXi -> ESXi) but it's all basically the same. At least on the versions I had.

The idea that I was trying to verify is how to more easily implement disaster recovery (DR) site using features of VMware ESX. Basically, what I was suggested is to make a clone of a virtual machine on a source ESX cluster and transfer it to another ESX. Now, it turned out that this idea won't function so easily. First, there is one important requirement:

The server you want to replicate is working in production and you don't want it to be offline. Otherwise, there is one reason less to buy ESX in a first place!

Well, maybe there are those that bought ESX for consolidation purposes only, but the biggest feature of ESX is it's possibility of achieving high availability, IMHO.

Later, I'll mention how simple it is to migrate virtual machine from one ESX to another (even ESXi) if there is no such requirement. But, this requirement means that you have to perform migration on a live virtual machine that creates several problems:

  1. You can try to do simple copy from source to destination over the network, e.g. using rsync, scp or something similar. But, this won't work as all the files belonging to the virtual machine are locked and can not be accessed while the virtual machine is running.
  2. The can use the tool vcbMounter. This tool can access locked files and make a backup of a running machine. In order to do that this tool creates temporary snapshot, performs copying, and removes snapshot. The problem is that the created image can not be just like that imported into ESX but it has to be first converted back to "normal" virtual machine using vcbRestore tool. This is _very_ time and disk demanding! Furthermore, if the target is ESXi there is no vcb* utilities there (unless you explicitly payed for them) and thus vcbMounter/vcbRestore process has to be run on a single cluster.

Ok, this second approach will work. But, you have one additional problem that is not specific to ESX/VMware. Namely, the process of creating snapshot has the same effect as forcefully powering down the machine. When you power down the machine in such way it may end up in an inconsistent state, meaning that data can be lost, and no one wants that. :) But to be honest, there is a mode of snapshoting that also takes an image of the memory of the virtual machine. This way solves the problem, but I don't know weather vcbMounter supports this mode of the operation.

Now, if you don't have requirement to migrate virtual machine while it has to be running, the migration is extremely simple and there are no problems mentioned in the previous paragraph. The process is: power down a virtual machine, copy it to another ESX cluster and that's it. Well almost, in all those cases you have to do one additional adjustment. Namely, configuration file of a virtual machine (the file having extension .vmx) has hardcoded paths you have to change before registering virtual machine. If you do not do that, then you'll be unable to start it!

Performing migration process

To perform virtual machine migration using only command line tools follow the next N steps:

  1. After copying virtual machine to a new ESX you have to check it's configuration file.s. The main configuration file has .vmx extension and there are hard disk configuration files that have extension .vmdk. Hard disk configuration files are referenced in the main configuration file. You have to check those files if they contain hard coded paths from the source ESX and if they do you'll have to change them before registering virtual machine.

    One of the parameters having hardcoded path is sched.swap.derivedName, so adjust it.

    Probably, the virtual machine will have the same IP address as the already running VM so you have to disconnect it's ethernet interface. To do that, in the main configuration file, add the following line (or modify it if it already exist): 

    ethernet0.startConnected = "FALSE"

    If there are more interfaces and you are unsure if something might go wrong during startup it's safest bet to disconnect them all.

  2. Register virtual machine with the ESX. Using RCLI the registration is performed with the following command line: 

    vmware-cmd -H <esxip> -U <username> -s register <configfileandpath> <datacenter> <resourcepool>

    You have to insert your parameters. esxip is the IP addres of ESXi server, or virtual center server (VCS) in case ESX cluster. username is username on the destination server with the appropriate privileges. In case of ESXi this will be user root in the default installation while in case of VCS it will be administrator. configfileandpath is the full path and file name of the main configuration file (the one ending in .vmx). datacenter and resourcepool are parameters of the ESX cluster and in the case of default ESXi installation the datacenter will be ha-dataceter, while resourcepool is not created and you have to create one before creating any virtual machine on the ESXi. The default name is default.

    So the command might look like this: 

    vmware-cmd -H 10.0.0.1 -U root -s register /vmfs/default/48d0a640-d0234234-ed1e-032423423464/vm/vm.vmx ha-datacenter default

    If the registration process was successfull you'll receive the following response: register() =1.

  3. Now, power on the virtual machine: 

    vmware-cmd -H 10.0.0.1 -U root /vmfs/default/48d0a640-d0234234-ed1e-032423423464/vm/vm.vmx start hard

    Now, one interesting absurd. The vmware-cmd will stuck beacuse the virtual machine has UUID generated on another machine. The ESXi will see this and also that this machine is first time powered on. What you have to do is open another terminal, and execute the following command there: 

    vmware-cmd -H 10.0.0.1 -U root /vmfs/default/48d0a640-d0234234-ed1e-032423423464/vm/vm.vmx answer
Enter password: 

Question (id = 0) : msg.uuid.moved:The location of this virtual machine's configuration file has changed since it was last powered on.

If the virtual machine has been copied, you should create a new unique identifier (UUID).  If it has been moved, you should keep its old identifier.

If you are not sure, create a new identifier.

What do you want to do?
	 1) Create
	 2) Keep
	 3) Always Create
	 4) Always Keep
	 0) Cancel

Select choice. Press enter for default <1> : 1

selected 1: Create

    Now, the first vmware-cmd will also exit with the message start() = 1.

The virtual machine will now be powered on, but you can only access it through Virtual Center Client. There is possibility of installing appropriate plugin into firefox.

Running ESXi inside VMware Workstation

It works, but in order for ESXi to boot under Workstation select RedHat Enterprise 4. After finishing configuration you have to manually add the following two lines to the virtual machine configuration file (configuration file has the extension .vmx): 

ethernet0.virtualDev = "e1000"
monitor_control.vt32 = "TRUE"

The first line changes network adapter type to Intel's e1000. This is necessary as ESXi 4 uses MAC address to generate unique ID for a virtual machine and default network adapter type (AMD) is not supported. This causes ESXi to fail in the boot process with a cryptic message Failed to load lvmdriver.

Additionally, if you are using Workstation 6.5 or later, you have to add the following line too: 

monitor.virtual_exec = "hardware"

If you plan to install virtual machines inside this ESXi then you have to prevent them from knowing they are running in a nested environment. This is achieved with the following line in the configuration file: 

monitor_control.restrict_backdoor = "true"

Also, when creating virtual machine use SCSI disk and LSI Logic controller. Otherwise, installation will stuck on not being able to find write device.

Different messages from ESXi

Log messages are placed in /var/log directory. You'll find there the following files:

On ESX, in case you are using that product, the log structure is a bit different:

What follows is a list of error messages I encountered when working with ESXi. Note that the beginning of the line is removed, i.e. there is not ime stamp and IP address of the ESXi that generated the message.

Additional good pages

  1. VMFAQ.com - FAQing about virtualization since 2008
    last checked: 20090726
  2. Welcome to vm-help.com
  3. VMware ESX 2.1/5 Server Administration II
  4. Patching ESXi
  5. What’s the difference between free ESXi and licensed ESXi?
  6. VMware ESXi SSH CLI commands
  7. Xtravirt Knowledge
  8. Contains instructions on how to remaster ESXi ISO image
  9. Re-master ESX Installation Media

ESXi HA

ESXi fail over? Anything open source?
ghettoVCB.sh - Free alternative for backing up VM's for ESX 3.5+ and ESXi
Xtravirt Virtual SAN
So, who wants free VMware High Availability?
VMware High Availability, free, part 2
VMware High Availability, free, part 3

VMFS

VMware VMFS
Open Source VMFS Driver

Backups

Pages related to backup procedures of ESXi.

  1. ghettoVCB.sh - Free alternative for backing up VM's for ESX(i) 3.5 and 4.0+
    last checked: 20090726

Monitoring

Pages related to monitoring ESXi instances.

  1. Monitoring ESXi performance through Nagios
    last checked: 20090726