Hacking ESXi
On this page I'll collect hacks that make ESXi easier to use. Note that by applying the stuff in this page your installation will be unsupported and it might be even non-functional. The information in this file is based on the ESXi 3.5U4 and it can differ significantly from other versions. In any case, be careful when applying tips from here, furthermore I'm not responsible for any damage that might arise from their use!
Console access
The ESXi allows console access even though it's hidden by default. So, just go to the first virtual console (Alt+F1) and type "unsupported" (without quotes). When you do that correctly you'll be presented with a password prompt. After entering password you are in a command line.
Remote access
Also in default configuration the only way to remotely manage ESXi is via VI client. But there is also possibility to access ESXi using SSH client/protocol. To enable SSH access, log into the console and open the file /etc/inetd.conf in text editor (I believe vi is the only choice :D). There find a line starting with #ssh and remove initial #. Write a file and exit the editor.
Now, you can either reboot ESXi or, much quicker, send HUP signal to inetd. To send HUPsignal find it's PID (using ps | grep inetd, PID is a number in a first column). Finally, execute kill -HUP <pid>. That's it.
Install RCLI on 64-bit F10/F11
ESXi can also be managed using RCLI tools. The problem is that there is no 64-bit version of the tools for ESXi prior to version 4.0. But there is workaround for which you'll need RCLI package (presumably you already have it) and 64-bit VIPerl package. First, install VIPerl, and then unpack RCLI package. To complete the installation manually copy the folowing directories:
| Source | Destination |
|---|---|
| external_bin | $DEST/bin |
| lib | $DEST/lib |
| man | $DEST/man |
bin/resxtop won't work as it has hardcoded path to /etc/vmware-rcli where it specifically searches for gtk compatibility library provided with VMware. You can try to hand edit it.
Note that you have to have installed 32-bit runtime.
Alternatively, and probably the better way, is to install newer vSphere version of Perl API that installs all the utilities also (e.g. vmware-cmd). So far it seems to support older versions also.
After installing vSphere version of Perl API (but also in the older versions) you can you get the following error when trying to run some command from the package:
/usr/bin/perl: symbol lookup error: /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi/auto/Class/MethodMaker/MethodMaker.so: undefined symbol: Perl_Tstack_sp_ptr
The problem is that VMware ships binaries compiled against some older Perl version. This problem is easy to solve. Just install package perl-Class-MethodMaker provided with the F10 distribution.
Another error message you could see is:
Hiding the command line arguments: symbol lookup error: /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/auto/Crypt/SSLeay/SSLeay.so: undefined symbol: Perl_Tstack_sp_ptr
To reslove this error install distribution's package perl-Crypt-SSLeay.
Additional binaries on ESXi
This is not supported on ESXi, but it is certainly possible. There are posts on the Internet that claim that VMware does something special with their ESXi binaries, but this claim seems to be false. So, in order to install a binary, simply copy it to ESXi machine. There are several catches though:
- After a first reboot all those binaries will be removed. So, for a permanent installation you have to perform some additional steps described later.
- You can not use just any binary you wish. First, 64-bit binaries wont work. Second, it is high probability that anything compiled on the machine with the kernel 2.6 wont work either. Finally, all the tools that use /proc, /sys, /dev or similar filesystems wont work either as ESXi doesn't have them or has a heavily modified version.
- Your best bet is to use statically compiled binaries.
Some additional facts about ESXi binaries:
- GCC version used to compile binaries was 3.3.1 (judging based on the strings found in libgcc_s.so.1)
- Binaries compiled against GLIBC versions 2.0, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.2, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.2.6, 2.3, 2.3.2 and 2.3.3.
- Compiling and trying to run dynamic binary from CentOS 4 (GCC 3.4.6
and GLIBC 2.3.4) wont work with the following error message:
# ./busybox ./busybox: /lib/libc.so.6: version `GLIBC_2.3.4' not found (required by ./busybox)
But statically compiled binaries work (at least BusyBox is working).
- Both, statically and dynamically compiled binaries on CentOS 3.9 work without modification on ESXi.
Note 20090709: I just figured out that the information about GLIBC version is available by downloading GPL covered sources from VMware's site. So I effectively discovered hot water. :) But then, better late then never. :)
SSH client
Well, SSH client is already on ESXi. :) What you have to do is to execute the following command before being able to use it:
~ # ln -s /sbin/dropbearmulti /bin/ssh
And that's it. :)
Replacing BusyBox
In ESXi 3.5U4 there is a BusyBox version 1.2.1. This is neither newest version nor compiled with all the applets. So, there could be reasons for someone to wish to upgrade BusyBox. At the time of this writing the newest (unstable) version was 1.14.0 which I tried to compile for ESXi.
Busybox 1.14.0 was compiled on CentOS 4.7 with a default configuration but the following modules (i.e. applets) were removed as they couldn't be compiled:
- ionice (CONFIG_IONICE)
- ifenslave (CONFIG_IFENSLAVE)
- acpid (CONFIG_ACPID)
On CentOS 3.9 additional applets had to be disabled:
- flash_eraseall (CONFIG_FLASH_ERASEALL)
I also removed the following applets because they don't work in ESXi:
- ps (CONFIG_PS)
- netstat (CONFIG_NETSTAT)
- df (CONFIG_DF)
Note that the easiest way to disable some applet is to open .config file, search for the configuration option given in the parentheses, and change it's value to n. You have to run make menuconfig at least once (and save configuration before exit) in order to have .config file. Furthermore, this file will be in the root of the BusyBox source distribution.
After you replaced BusyBox you'll have some additional (and necessary, IMHO) tools available:
- strings
- du
Adding tools not in BusyBox
The following tools are not provided by BusyBox, but they are essential too:
- rsync
ESXi initialization process
Boot process
When ESXi starts it's init process runs commands given in the /etc/inittab file labeled with sysinit keyword.
Shutdown process
Shutdown process invokes /sbin/shutdown.sh script. This script stops all the configured VMs, then stops all local services and finally backups local state.
Backuping local state
The local state is saved across boots by the script /sbin/backup.sh. To invoke the script use the following syntax:
/sbin/backup.sh <shutdown> [<bootbank>]
The first argument is a boolean value that is used to signal if the shutdown process in is progress (1) or not (0). The second argument is optional and points to the store where local state will be saved. Default value for this argument is /bootbank.
High availability on ESXi
ESXi doesn't have any high availability features that I'm aware of. This is important feature for any serious use of ESXi so here are some proposals/ideas on how to achieve it.
Single machine
With a single machine you can only have highly available disks using some form of RAID. There are two options here. The first one is to use hardware RAID if the machine's motherboard supports it. Software RAID is not supported by the ESXi but there is an option of building MD drivers from Linux kernel. I didn't try it yet, but I plan to. The other option is to two separate disks with manual synchronization between them.
Interesting facts about ESXi
In this section I'll list some facts that are interesting to know but for which I don't know if there is some practical use. Feel free to mail me if you disagree.
Disk devices and partitions
ESXi doesn't have devices, but there are files that represent whole disks and their partitions. They can be found in /dev/disks. For example, in my case they look like this:
/dev/disks # ls -l -rw------- 1 root root 8589934592 May 15 05:37 vmhba1:0:0:0 -rw------- 1 root root 782237696 May 15 05:37 vmhba1:0:0:1 -rw------- 1 root root 4293918720 May 15 05:37 vmhba1:0:0:2 -rw------- 1 root root 3509583872 May 15 05:37 vmhba1:0:0:3 -rw------- 1 root root 4177920 May 15 05:37 vmhba1:0:0:4 -rw------- 1 root root 50315264 May 15 05:37 vmhba1:0:0:5 -rw------- 1 root root 50315264 May 15 05:37 vmhba1:0:0:6 -rw------- 1 root root 115326976 May 15 05:37 vmhba1:0:0:7 -rw------- 1 root root 566214656 May 15 05:37 vmhba1:0:0:8
The last number is obviously a partition number, with partition zero representing the whole disk. It can be loaded into fdisk program which shows the following (note that I'm using updated busybox, not the stock one):
/dev/disks # /busybox fdisk -l vmhba1\:0\:0\:0
Disk vmhba1:0:0:0: 8589 MB, 8589934592 bytes
64 heads, 32 sectors/track, 8192 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Device Boot Start End Blocks Id System
vmhba1:0:0:0p1 5 750 763904 5 Extended
vmhba1:0:0:0p2 751 4845 4193280 6 FAT16
vmhba1:0:0:0p3 4846 8192 3427328 fb Unknown
vmhba1:0:0:0p4 * 1 4 4080 4 FAT16 <32M
vmhba1:0:0:0p5 5 52 49136 6 FAT16
vmhba1:0:0:0p6 53 100 49136 6 FAT16
vmhba1:0:0:0p7 101 210 112624 fc Unknown
vmhba1:0:0:0p8 211 750 552944 6 FAT16
Partition table entries are not in disk order
/dev/disks #
Additional links
vpxd.cfg Advanced Configuration
Building OSS drivers for ESXi
GPL Components - Build ESX Drivers?
Open Virtualization Drivers