Paravirtualising an HVM CentOS VM

If you've installed a CentOS (or, for that matter, any other linux distro) on XenServer without using one of the templates, you may have found that you cannot install XenTools, meaning that you can't perform live migrations, and the VM may not perform as well as could be expected. This is because when it is installed in this way, it typically is not Paravirtualised, and is installed instead using Hardware virtualisation (HVM). Paravirtualising an HVM DomU is a pain, but it is definitely do-able, and here's how. There's a fair bit of documentation around on the net, but I found none of the techniques worked fully for me, so here's my step by step guide. YMMV.

The first step is to install a Xen-Aware kernel onto the new host. Assuming the server has Internet connectivity, you can do this simply with the following command (as root):

yum install kernel-xen

This will find, and prompt you to download, the latest version of CentOS' Xen aware kernel. Now we need to generate a new initrd file without some of this kernel's SCSI drivers. To do this, execute the following:

cd /boot/
mkinitrd --omit-scsi-modules --with=xennet --with=xenblk \
--preload=xenblk initrd-$(uname -r)-no-scsi.img $(uname-r)xen

This will make an initrd file without the SCSI drivers, but including the Xen networking and block device modules. If you get an error, uname might be returning an invalid version, so in place of $(uname -r) just insert the desired version string (ie 2.6.18-194.8.1.el5). You can verify this by running:

ls /boot/

and looking for a new file called something like 'initrd-2.6.18-194.8.1.el5xen-no-scsi.img'.

Now we need to make some changes to the GRUB bootloader. If you look at the file /boot/grub/menu.lst, you will see the Xen Kernel install has added a new option for us (as Option 0). It should be looking something like this:

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
#          initrd /initrd-version.img
#boot=/dev/hda
default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-194.8.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-194.8.1.el5
        module /vmlinuz-2.6.18-194.8.1.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-194.8.1.el5xen.img
title CentOS (2.6.18-164.15.1.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-164.15.1.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-164.15.1.el5.img
title CentOS (2.6.18-128.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-128.el5.img

You can see we have three kernel options here, and the first is the Xen aware one. There are a few changes we need to make to this file. The first is to change the line:

default=1

to:

default=0

This will tell the bootloader to default to the first option, or our Xen Kernel. There is a further slight change to be made to the entry for the Xen kernel to allow it to work with the XenServer pygrub bootloader. We need to remove the following line:

kernel /xen.gz-2.6.18-194.8.1.el5

And edit the line below to replace the word 'module' with 'kernel'. Then go to the line below this and replace 'module' with 'initrd'. You will also need to change the file name so that it reads 'initrd-2.6.18-194.8.1.el5xen-no-scsi.img'; the name of the initrd file we created earlier. Version numbers of course might vary. The file should now look like the following:

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
#          initrd /initrd-version.img
#boot=/dev/hda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-194.8.1.el5xen)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-194.8.1.el5xen ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-194.8.1.el5xen-no-scsi.img
title CentOS (2.6.18-164.15.1.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-164.15.1.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-164.15.1.el5.img
title CentOS (2.6.18-128.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-128.el5.img

We now need to make some changes to the XenServer configuration for this DomU.

Firstly, you will need to shutdown the VM.

Once this is done, click on the Dom0 server in XenCentre (or connect via SSH), and navigate to the Console pane. You will need to gather a chain of information as follows:

First, we need to get the UUID of the DomU. This is the Xen internal unique ID. You can get it with the following command:

xe vm-list name-label= --minimal

The output will be a single line containing the UUID, which will look something like this: c58fcc8a-4f5d-c695-d6a1-29c6063b9296

Once we have this, copy it to the clipboard and save it in a notepad window, as we'll use it in the next command. We need to find the VBD UUID for our DomU's disk. We can do this with the following:

xe vm-disk-list vm=

This will return something like the following:

Disk 0 VBD:
uuid ( RO)             : 55f9199b-a5ca-f6ff-67c2-6e1830547f0b
    vm-name-label ( RO): backup-01.egh
       userdevice ( RW): 0


Disk 0 VDI:
uuid ( RO)             : 998d344a-5ec6-4bf9-8f34-190b7ad12fb6
       name-label ( RW): 0
    sr-name-label ( RO): backup-01.egh
     virtual-size ( RO): 7945689497

The one we're interested in is the VBD, whose UUID is 55f9199b-a5ca-f6ff-67c2-6e1830547f0b in the above. Like before, save this to a notepad file for later.

Next step is to set this VBD to bootable (needs the VBD UUID from the last step):

xe vbd-param-set uuid= bootable=true

We need to turn off the HVM boot policy for this VM now we're paravirtualised (needs the VM UUID from the first step):

xe vm-param-set uuid= HVM-boot-policy=

The end after the = is blank intentionally, this has the effect of 'unsetting' the parameter. Next is to set up a new PV bootloader to use pygrub (needs the VM UUID from the first step):

xe vm-param-set uuid= PV-bootloader=pygrub

Finally we do some configuration of the PV to handle graphical output from screen and console:

xe vm-param-set uuid= PV-args="console=xvc0"

You should be able to start the VM up again now. You may need to restart XenCentre if keyboard input to the VM's console is not working - this can be a side effect of the conversion from HVM to PV.

Now that we're back up and Paravirtualised, we can install XenTools (recommended). To do this, log into the VM as root. From XenCentre, right click on the running VM and select 'Install XenTools' from the menu. You will see this loads a disk image called 'xs-tools.iso' to /dev/xvdd. We can now install XenTools with the following set of commands:

mkdir /media/cdrom
mount /dev/xvdd /media/cdrom
cd /media/cdrom/Linux
./install.sh

This will run the installer. Once complete, reboot the VM, and the install should be done. You can verify this is complete by clicking on the running VM in XenCentre, and going to the 'General' tab. Next to 'Virtualization State' you will see it says Optimized, showing that we're now fully paravirtualised.

One last thing. The Xen network drivers for some reason cause the newly paravirtualised VM to lose it's network connectivity, however it is very easy to restore. Log onto the machine as root, and issue the following commands:

cd /etc/sysconfig/network-scripts/
mv ifcfg-eth0.bak ifcfg-eth0
ifup eth0

Test with a couple of pings, and all should be well. If something does go horribly wrong, most of the time you can revert to booting in HVM with the following XenServer CLI command:

xe vm-param-set uuid= HVM-boot-policy="BIOS Order"

Once you've executed this, before you start the VM up again you'll need to go into the properties for the VM and tell it to boot from the Hard Disk, otherwise you'll just get a message in the console saying no bootable media found.

If you get a pygrub boot error, this may be down to omitting the command which sets the VBD paramater 'bootable' to true.

If during boot you get an error such as:

Kernel panic - not syncing VFS: Cannot open root device "VolGroup00/LogVol00"

This typically means that something went wrong when you generated your initrd file - it is still using the Xen Kernel with the original SCSI drivers. Try re-creating this, and make sure your GRUB menu.lst file points to the correct initrd file.