This isn’t something typically done so it took a while to figure out. Here are my findings.
You can install ESX 4 inside ESX 4 but in this case I chose to install it inside of ESX 3.5 Update 4. Regardless, there are a few caveats:
1. If you try to snapshot the image, ESX will no longer boot- it gives error “You have entered the recovery shell … /bin/sh: can’t access tty; job control turned off” (sometimes you can recover from this error by shutting down and deleting all your snapshots).
2. If you give it less than 2GB of memory it will error at boot and say “not enough memory to load VMKernel. <current> KB of RAM was detected. We require 2065384 KB of RAM to boot. Aborting boot” (I figured out a fix for this, but you must give it 2GB initially).
3. If you don’t give it at least 9.5GB of hard drive space, it will act like the disk isn’t writable.
So here are my steps:
New Virtual Machine:
Guest OS RHEL5 64bit
at least 10 GB for hard drive, LSI Logic (recommend 12-15 if you want to patch it)
at least 2GB for memory (slim this down after the install)
2 CPUs (probably not required, but it runs slow as is)
During install select NO for “Install Custom Drivers?” and YES for “Load the system drivers?” I also changed the swap partition from 600MB to 2GB, probably not necessary. I added a user called ‘sshUser’ (alternately you can use root if you console and change /etc/ssh/sshd_config allowRootLogin = yes)
A virtual switch with promiscuous enabled (you can install without this but if you want networking to work you’ll need to enable promiscuous mode on the vSwitch on the Host, no reboot required).
Once installed, ssh using the user you created (sshUser), then open /etc/vmware/init/init.d/00.vmnix and change RequiredMemory to 516096.
Then you can boot with 768 memory.
You can try 512 but you’ll probably purple screen.
#UPDATE for installing ESX 4 inside of ESXi 5. Go into the console of your ESXi host and do this:
echo 'vhv.allow = "TRUE"' >> /etc/vmware/config