Motivation
It is undeniable that virtualization provide you with a lot of benefits; consolidation of servers will save you money. Maintaining your virtualized servers becomes a breeze, since they are all accessible from a single front-end. But implementing your environment on VMware comes at a price... even for the free version VMware Server; maintenance. You will still need to monitor your virtualization servers for performance and distribute your virtual machines among your servers or resource pools according to load. Although there are tools available to assist you, it is not the biggest issue.
VMware Server requires an additional operating system like Linux (or Windows). This introduces an added maintenance dependency; kernel modules which VMware uses need to be recompiled when the kernel gets updated. If a security exploit is found in the kernel, the virtualization host can be compromised and therefore jeopardize the continuity of your business. To prevent this, you update the kernel, but on reboot you notice that VMware does not start properly. On the command line you will need to perform a vmware-config.pl which build the modules. Unfortunately, your kernel headers are newer and either vmmon (virtual machine monitor) or vmnet (network component) does not build properly. Because of this, your whole infrastructure is offline until VMware provides new components or the release of the community patch package 'vmware-any-any'.
Users of VMware ESX will not have this issue, since the operating system and modules are provided in the form of the VMware's vmkernel. If a security update is needed, you only need to install an update. The previously mentioned issue has happened to me often enough and I I wanted to have the same solution as ESX, but with the convenience of using Un*x/Linux experience for maintenance without the dependency of a third-party product. Since RHEL 5.4 this is provided as an enterprise supported solution; KVM, the Kernel-based Virtual Machine, once developed by Qumranet and now owned by RedHat. And by using CentOS this is available to everyone without too much trouble.
There are some differences between VMware and KVM. KVM is provided as part of the kernel, so any Linux distribution can provide it as part of their offering. KVM supports different formats for a virtual hard disk, while VMware only uses their own VMDK format. VMware can run in a full virtualized mode and is therefore suitable for older hardware. KVM needs a processor that provides VT-x (Intel) or AMD-V (AMD) to operate. VMware can run x86_64 guests on a i686 host, while KVM can not.
Because of this last reason, I had to reinstall CentOS on one of my systems, since it still ran an i686 version of CentOS 5.4 (on a Core2Duo E6600). This provides a good overview of what needs to be done to run KVM on CentOS.
Note: since KVM uses qemu to emulate any hardware you can also choose to use qemu without KVM as the hypervisor, but instead use the kqemu kernel module.
Installation
You will need to download an installation disc of CentOS 5.4. Prefer to use the x86_64 version; CentOS-5.4-x86_64-bin-DVD.iso. During the installation you can keep the default settings and when asked for the 'additional tasks' you will perform, do not select the 'Virtualization' option. After the installation finished, you will need to update the system to the latest updates.
From the reinstall I noticed a minor error in the CentOS 5.4 x86_64 installation. Due to installing openssl.i686 it also installed a lot of other standard i386-arch libraries. You can find out if you also have those packages installed as described in a earlier post. If you also want to remove those packages, perform the following command line option before continuing.
$ rpm -e `rpm -qa --qf '%{name}.%{ARCH}\n' |grep i386` openssl.i686To update your system:
$ yum updateIf your system is up-to-date, continue to install the needed packages for KVM and Virt-Manager.
$ yum install bridge-utils kvm kvm-qemu virt-manager virt-viewer python-virtinstIn later versions of CentOS it might suffice to issue the command "yum groupinstall 'Virtualization'", but currently this installs a Xen enabled kernel. According to the used processor, you can test to load the needed kernel module and start the libvirt daemon.
$ modprobe kvm
$ modprobe kvm-intel (or kvm-amd)
$ service libvirtd startI normally use a different management workstation. And as an example I use a Fedora 12 installation to connect to the virtualization host (beibei). First from a tunneled connection to the host (with X11 forwarding).
$ ssh -X root@beibei
root@beibei's password: **************
# virt-managerThis will start the Virtual Machine Manager from which you can create new instances. You can of course use a Gnome desktop from the CentOS system to manage your virtual machines, but it pays off to try a seperate management workstation. This will allow you to try additional features, like live migrations between KVM virtualization servers.
This is all it needs to install KVM on a CentOS 5.4 installation. For an earlier version of CentOS, it is best to consult the wiki article about KVM.
Network settings
A default VMware Server installation will bridge the network card you have. You will probably want to have the same configuration for your KVM environment. For this you need standard Linux bridging utilities.
$ yum install bridge-utilsYou only need to edit (or create) some network-scripts to make a bridge work.
$ vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:1a:2b:3c:4d:5e
ONBOOT=yes
BRIDGE=br0And create the bridge
$ vi /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=BridgeWhen you start the network again, you will have a br0 and eth0 device.
$ service network restartYou will now be able to select a bridged network card inside the KVM virtual machine configuration. If you have more network card, create a bridge per network card.
Converting the virtual disk files
For the migration part I choose a virtual machine who once began it's life as a physical machine; forum.survion.net. It is a Linux installation once based on FC3, got migrated to CentOS... and then ran as a VMware virtual machine.
The VMware virtual machine consists of one or more hard disk files, with the extension .vmdk and a description file, with the extension .vmx. A listing of server-forum.vmwarevm looks like:
[root@beibei server-forum.vmwarevm]# ls -al total 29405144 drwxr-xr-x 2 root users 4096 Dec 12 20:53 . drwxr-xr-x 21 root root 4096 Dec 9 17:32 .. -rwxr-xr-x 1 root vmware 8684 Dec 7 21:09 nvram -rw------- 1 root vmware 0 Jan 3 2009 Red Hat Linux.vmsd -rwxr-xr-x 1 root vmware 2369 Dec 7 21:09 Red Hat Linux.vmx -rw-r--r-- 1 root vmware 268 Jan 3 2009 Red Hat Linux.vmxf -rw-r--r-- 1 root vmware 2147221504 Dec 7 22:22 storage-f001.vmdk -rw-r--r-- 1 root vmware 2147221504 Dec 7 22:20 storage-f002.vmdk -rw-r--r-- 1 root vmware 2147221504 Dec 7 22:21 storage-f003.vmdk -rw-r--r-- 1 root vmware 2147221504 Dec 7 22:20 storage-f004.vmdk -rw-r--r-- 1 root vmware 2147221504 Dec 7 22:22 storage-f005.vmdk -rw-r--r-- 1 root vmware 2147221504 Aug 30 14:45 storage-f006.vmdk -rw-r--r-- 1 root vmware 2147221504 Dec 7 22:22 storage-f007.vmdk -rw-r--r-- 1 root vmware 2147221504 Oct 25 19:18 storage-f008.vmdk -rw-r--r-- 1 root vmware 2147221504 Nov 30 04:05 storage-f009.vmdk -rw-r--r-- 1 root vmware 2147221504 Nov 30 04:04 storage-f010.vmdk -rw-r--r-- 1 root vmware 2621440 Jul 23 2008 storage-f011.vmdk -rw-r--r-- 1 root vmware 749 Dec 7 21:11 storage.vmdk -rw-r--r-- 1 root vmware 2147221504 Dec 7 22:22 system-f001.vmdk -rw-r--r-- 1 root vmware 2147221504 Dec 7 22:22 system-f002.vmdk -rw-r--r-- 1 root vmware 2147221504 Dec 7 22:22 system-f003.vmdk -rw-r--r-- 1 root vmware 2147221504 Dec 7 22:22 system-f004.vmdk -rw-r--r-- 1 root vmware 17659904 Jul 23 2008 system-f005.vmdk -rw-r--r-- 1 root vmware 616 Dec 7 21:11 system.vmdk
As you can see, there are two hard disks (system and storage) and several other files. The hard disk as shown here is a fully allocated file which is divided up in seperate 2Gb files (handy for use on a FAT32 filesystem for portability). We have to convert it back into a monolithic file for use on KVM. Luckily KVM understands the monolithic VMDK file format, so we will choose this for interoperability. To convert the file you can use a VMware provided tool called vmware-vdiskmanager (includes with VMware products).
$ vmware-vdiskmanager -r system.vmdk -t 0 forum-system.img
$ vmware-vdiskmanager -r storage.vmdk -t 0 forum-storage.imgAfter this you will have two files that need to be copied to the in the images folder of libvirt on the virtualization host (/var/lib/libvirt/images/). You can do this using scp, ftp or mount an NFS export. From here we can create a virtual machine to connect these disk files to. Connect to the virtualization host and start the Virtual Machine Manager.
Create the Virtual Machine
Select localhost and create a new virtual machine using the context menu. Give it a unique name that identifies it easily for you, as I did: server-forum. If KVM is installed properly, you can only select "Full virtualized' from the Virtualization Method. Select the CPU architecture that fits your former VMware virtual machine and as the hypervisor: kvm.
On the 'Installation Method' page choose 'Network boot (PXE)' so you don't have to select a installation disk image.
On the 'Storage' page you deselect the 'Allocate entire virtual disk now' since we will not use the create virtual hard disk image. Beware; don't let the size become 0, since this way libvirt will not create the disk file and will fail to create the virtual machine.
On the 'Network' page you will probably want to select the bridged network card. This gives you the same functionality as VMware did as default option. On the 'Memory and CPU Allocation page' you can adjust the settings which are most close to your former virtual machine. After the summary the virtual machine will start automatically. Immediately issue a 'Force Off' as Shut down option. Now select the 'Hardware' tab.
Remove the 'Disk hda' and add new hardware of the type 'Storage' and select the system file from the storage location. If you have selected the file, you will see the correct size. Do the same of the other disks you have.
Be sure to change the boot device in the Boot options of this VM. Else it will try to start using Network (PXE) instead of the virtual hard disk.
Now everything should be ok to start the virtual machine.
Issues you might encounter
The conversion will probably not be without problems. For instance, I had a common issue with the kernel of the virtual machine. Due to a kernel panic it did not start properly. Fortunately tools are available to perform actions of the virtual hard disk files. One of those tools is guestfish. On CentOS and Fedora you can install this tool using:
$ yum install guestfishFor instance, I had to download a newer kernel and try to install this inside the virtual machine. The following commands show what I did:
$ guestfish
fs# add /var/lib/libvirt/images/forum-system.img
fs# run
Could not open '/dev/kqemu' - QEMU acceleration layer not activated: No such file or directory
fs# mount /dev/sda2 /
fs# mount /dev/sda1 /boot
fs# upload /tmp/downloaded.rpm /tmp/kernel.rpm
fs# command "rpm -ivh /tmp/kernel.rpm"
fs# vi /boot/grub/grub.confThese commands will add the forum-system.img to a qemu instance. Inside the qemu instance you will then mount the mountpoints, upload the kernel RPM from the local path to the tmp inside the filesystem. From here nothing special... and then it will issue a command inside qemu to install the kernel package. This command will run inside the filesystem as if it performed the command from the commandline. Finally I verify the grub configuration and change it if necessary. From here I was able to start the virtual machine without any errors.
Note: If you convert Windows virtual machines you might encounter more issues. Those conversion steps are outside the scope of this document, but are not impossible to perform.
Fedora 12
The latest release of Fedora includes many improvements which are virtualization related. If you want to try KVM on a workstation, you should definitely try Fedora 12.
Below here you can see virt-manager. The interface has changed a little. As you can see from the screenshot, it is connected to the local machine and two remote hosts. The console is shown from the remote host (using ssh and vnc). For this no additional configuration was needed.
It is easy to monitor the performance of a virtual machine using the Virtual Machine Manager. Here you can see the server-forum instance with an overview of CPU/Memory usage and Disk/Network activity.
Just a few days ago SPICE was released. This is an alternative to vnc to view the console of a virtual machine. It is target towards the use of a Virtual Desktop. More information about this can be found on the SPICE website. A how-to for Fedora 12 is available.
