Collaborative Coding

Just a quick post this time, to spread the word about Ethercodes, which is essentially multiplayer notepad, with syntax highlighting. Very handy for those using agile methods, or just getting a bit of help when two (or more!) heads are better than one. Check it out, free and easy.

Paravirtualising an HVM CentOS VM

If you've installed a CentOS (or, for that matter, any other linux distro) on XenServer without using one of the templates, you may have found that you cannot install XenTools, meaning that you can't perform live migrations, and the VM may not perform as well as could be expected. This is because when it is installed in this way, it typically is not Paravirtualised, and is installed instead using Hardware virtualisation (HVM). Paravirtualising an HVM DomU is a pain, but it is definitely do-able, and here's how. There's a fair bit of documentation around on the net, but I found none of the techniques worked fully for me, so here's my step by step guide. YMMV.

The first step is to install a Xen-Aware kernel onto the new host. Assuming the server has Internet connectivity, you can do this simply with the following command (as root):

yum install kernel-xen

This will find, and prompt you to download, the latest version of CentOS' Xen aware kernel. Now we need to generate a new initrd file without some of this kernel's SCSI drivers. To do this, execute the following:

cd /boot/
mkinitrd --omit-scsi-modules --with=xennet --with=xenblk \
--preload=xenblk initrd-$(uname -r)-no-scsi.img $(uname-r)xen

This will make an initrd file without the SCSI drivers, but including the Xen networking and block device modules. If you get an error, uname might be returning an invalid version, so in place of $(uname -r) just insert the desired version string (ie 2.6.18-194.8.1.el5). You can verify this by running:

ls /boot/

and looking for a new file called something like 'initrd-2.6.18-194.8.1.el5xen-no-scsi.img'.

Now we need to make some changes to the GRUB bootloader. If you look at the file /boot/grub/menu.lst, you will see the Xen Kernel install has added a new option for us (as Option 0). It should be looking something like this:

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
#          initrd /initrd-version.img
#boot=/dev/hda
default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-194.8.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-194.8.1.el5
        module /vmlinuz-2.6.18-194.8.1.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-194.8.1.el5xen.img
title CentOS (2.6.18-164.15.1.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-164.15.1.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-164.15.1.el5.img
title CentOS (2.6.18-128.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-128.el5.img

You can see we have three kernel options here, and the first is the Xen aware one. There are a few changes we need to make to this file. The first is to change the line:

default=1

to:

default=0

This will tell the bootloader to default to the first option, or our Xen Kernel. There is a further slight change to be made to the entry for the Xen kernel to allow it to work with the XenServer pygrub bootloader. We need to remove the following line:

kernel /xen.gz-2.6.18-194.8.1.el5

And edit the line below to replace the word 'module' with 'kernel'. Then go to the line below this and replace 'module' with 'initrd'. You will also need to change the file name so that it reads 'initrd-2.6.18-194.8.1.el5xen-no-scsi.img'; the name of the initrd file we created earlier. Version numbers of course might vary. The file should now look like the following:

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
#          initrd /initrd-version.img
#boot=/dev/hda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-194.8.1.el5xen)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-194.8.1.el5xen ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-194.8.1.el5xen-no-scsi.img
title CentOS (2.6.18-164.15.1.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-164.15.1.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-164.15.1.el5.img
title CentOS (2.6.18-128.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-128.el5.img

We now need to make some changes to the XenServer configuration for this DomU.

Firstly, you will need to shutdown the VM.

Once this is done, click on the Dom0 server in XenCentre (or connect via SSH), and navigate to the Console pane. You will need to gather a chain of information as follows:

First, we need to get the UUID of the DomU. This is the Xen internal unique ID. You can get it with the following command:

xe vm-list name-label= --minimal

The output will be a single line containing the UUID, which will look something like this: c58fcc8a-4f5d-c695-d6a1-29c6063b9296

Once we have this, copy it to the clipboard and save it in a notepad window, as we'll use it in the next command. We need to find the VBD UUID for our DomU's disk. We can do this with the following:

xe vm-disk-list vm=

This will return something like the following:

Disk 0 VBD:
uuid ( RO)             : 55f9199b-a5ca-f6ff-67c2-6e1830547f0b
    vm-name-label ( RO): backup-01.egh
       userdevice ( RW): 0


Disk 0 VDI:
uuid ( RO)             : 998d344a-5ec6-4bf9-8f34-190b7ad12fb6
       name-label ( RW): 0
    sr-name-label ( RO): backup-01.egh
     virtual-size ( RO): 7945689497

The one we're interested in is the VBD, whose UUID is 55f9199b-a5ca-f6ff-67c2-6e1830547f0b in the above. Like before, save this to a notepad file for later.

Next step is to set this VBD to bootable (needs the VBD UUID from the last step):

xe vbd-param-set uuid= bootable=true

We need to turn off the HVM boot policy for this VM now we're paravirtualised (needs the VM UUID from the first step):

xe vm-param-set uuid= HVM-boot-policy=

The end after the = is blank intentionally, this has the effect of 'unsetting' the parameter. Next is to set up a new PV bootloader to use pygrub (needs the VM UUID from the first step):

xe vm-param-set uuid= PV-bootloader=pygrub

Finally we do some configuration of the PV to handle graphical output from screen and console:

xe vm-param-set uuid= PV-args="console=xvc0"

You should be able to start the VM up again now. You may need to restart XenCentre if keyboard input to the VM's console is not working - this can be a side effect of the conversion from HVM to PV.

Now that we're back up and Paravirtualised, we can install XenTools (recommended). To do this, log into the VM as root. From XenCentre, right click on the running VM and select 'Install XenTools' from the menu. You will see this loads a disk image called 'xs-tools.iso' to /dev/xvdd. We can now install XenTools with the following set of commands:

mkdir /media/cdrom
mount /dev/xvdd /media/cdrom
cd /media/cdrom/Linux
./install.sh

This will run the installer. Once complete, reboot the VM, and the install should be done. You can verify this is complete by clicking on the running VM in XenCentre, and going to the 'General' tab. Next to 'Virtualization State' you will see it says Optimized, showing that we're now fully paravirtualised.

One last thing. The Xen network drivers for some reason cause the newly paravirtualised VM to lose it's network connectivity, however it is very easy to restore. Log onto the machine as root, and issue the following commands:

cd /etc/sysconfig/network-scripts/
mv ifcfg-eth0.bak ifcfg-eth0
ifup eth0

Test with a couple of pings, and all should be well. If something does go horribly wrong, most of the time you can revert to booting in HVM with the following XenServer CLI command:

xe vm-param-set uuid= HVM-boot-policy="BIOS Order"

Once you've executed this, before you start the VM up again you'll need to go into the properties for the VM and tell it to boot from the Hard Disk, otherwise you'll just get a message in the console saying no bootable media found.

If you get a pygrub boot error, this may be down to omitting the command which sets the VBD paramater 'bootable' to true.

If during boot you get an error such as:

Kernel panic - not syncing VFS: Cannot open root device "VolGroup00/LogVol00"

This typically means that something went wrong when you generated your initrd file - it is still using the Xen Kernel with the original SCSI drivers. Try re-creating this, and make sure your GRUB menu.lst file points to the correct initrd file.

Extreme Networks, BGP and Route Policies

Setting up simple BGP topologies using Extreme Networks' XOS based switches is pretty straightforward, but I've found there to be a slight lack of documentation around anything simpler than route-reflectors and extended communities.

The basic topology I had was a SummitStack of two x480 switches, and a pair of Gigabit links connecting the switches to an MPLS VRF operated by another company. We want a simple active/passive setup, with one link acting as a backup, and the other carrying all traffic, unless an event occurs to take it offline. Here's a basic setup for the BGP config on the SummitStack (AS Numbers and IP Addresses have been changed to protect the innocent).

configure bgp AS-number 65001
configure bgp routerid 242.242.242.242
enable bgp fast-external-fallover
configure bgp add network 190.190.190.0/23
create bgp neighbor 194.194.194.193 remote-AS-number 65002
enable bgp neighbor 194.194.194.193
create bgp neighbor 242.242.242.241 remote-AS-number 65002
enable bgp neighbor 242.242.242.241
enable bgp

This should be sufficient to bring the BGP sessions up and to start advertising our 190.190.190.0/23 network, and learning whatever is sent back. Note that in order to advertise the 190.190.190.0/23 network there needs to be a route for this in the switch's route table - you may need to route it to a local loopback at first.

However we need a way to enforce the active/passive nature of these links, which can be done using a local preference, (as long as the other party will reciprocate). Local preference values for BGP routes default to 100, so all we need to do is increase the local preference for routes learned through our preferred primary circuit, and they will take precedence in the route table. We can do this using route policies. We can also use route policies to have a bit more control over what routes we send to our peers, and what routes we'll accept in return. In this specific case we're only interested in learning a default route (0.0.0.0/0), and advertising our 190.190.190.0/23 network. We can build some route policies that will enforce this for us. First, lets build the policies that will restrict what we learn, and set the local preference.

We'll create two policy files, one for the primary circuit called bgp_pri_policy.pol, and one for the backup, called bgp_sec_policy.pol.

Here's bgp_pri_policy.pol:

entry default_pri {
  if match all {
    nlri 0.0.0.0/0 exact;
    next-hop 194.194.194.193;
  } then {
    origin egp;
    local-preference 500;
    permit; 
  }
}

entry deny_other {
  if {
  } then {
  deny;
 }
}

Note that it matches on the next-hop (this determines its come from our primary peer), and on the NLRI, or the route we've learned. The first entry accepts 0.0.0.0/0 from 194.194.194.193 only, and sets its local preference to 500. This is higher than the default 100, so will be preferred. bgp_sec_policy.pol is very similar, only with a different next-hop and a lower local preference:

entry default_sec {
  if match all {
    nlri 0.0.0.0/0 exact;
    next-hop 242.242.242.241;
  } then {
    origin egp;
    local-preference 100;
    permit;
  }
}

entry deny_other {
  if {
  } then {
    deny;
  }
}

Note that local preference is being set to 100 (we could have left it, as 100 is default, but it is here for completeness' sake). We can now create a final policy to limit what we send out to our peers, good practice to prevent any unexpected route leaks. We will create a policy called bgp_exp_policy.pol with a single entry:

entry export_policy {
  if match all {
   nlri 190.190.190.0/23;
  } then {
   permit;
  } 
}

This uses an implicit deny to block anything that is not permitted. We now just need to apply these policies to our peers:

configure bgp neighbor 194.194.194.193 route-policy in bgp_pri_policy
configure bgp neighbor 194.194.194.193 route-policy out bgp_exp_policy

configure bgp neighbor 242.242.242.241 route-policy in bgp_sec_policy
configure bgp neighbor 242.242.242.241 route-policy out bgp_exp_policy

If you refresh the BGP now you should see the routes behaving as expected.

JDOM, XPath and Namespaces

The combination of JDOM and XPath seems to be the source of greatest woe for me of late - following my last blog post I spent (wasted) another day wrestling with these technologies on another sticking point. I'll not claim to be an XPath guru by any stretch (I am learning though!), but it does seem to throw some real curveballs. The latest was down to attempting to extract a List of JDOM sub-Elements from an Element using XPath.selectNodes(element). It continually refused to work, and seemed to be related to the fact that there was a namespace declaration in the root Element. After much head scratching and the help of the wider community, it came out that all that was needed was to wrap the Element in a Document. Can anyone with a deeper knowledge shed a light on why?

Here's the code that doesn't (but to my mind should) work:

Namespace ns = Namespace.getNamespace("srv","http://ess.com/ws/srv");   
XPath filterXpression = XPath.newInstance("//srv:ItemCost");
filterXpression.addNamespace(ns);   
List nodes = filterXpression.selectNodes(response);

And here's the working code:

Document build = new Document(response);
XPath filterXpression = XPath.newInstance("//srv:ItemCost");
List nodes = filterXpression.selectNodes(build);

You can get the full lowdown in this StackOverflow question.

JDOM & XPath Contexts

Second post today - its been a week of minor but frustrating issues so far and I want to publish as many solutions as I can find time for.

This issue had me scratching my head for a good few hours, and there didn't seem to be much solid documentation online to help out. I was using a combination of JDOM 1.0 and XPath to manipulate a short XML document (arriving via a SOAP interface). Here's the basic document structure that was the subject (Names have been changed to protect the innocent):

<srv:Root>
  <srv:Name>Overall Name</srv:Name>
  <srv:Items>
    <srv:Item>
      <srv:ItemName>Name 1</srv:ItemName>
      <srv:ItemPrice>20</srv:ItemPrice>
    </srv:Item>
    <srv:Item>
      <srv:ItemName>Name 2</srv:ItemName>
      <srv:ItemPrice>30</srv:ItemPrice>
    </srv:Item>
  </srv:Items>
</srv:Root>

The above conforms to the schema set out for the message, everything validates fine. I was using the following to extract the Items, then iterate through each Item and print out the ItemName and the ItemPrice:

XPath itemXpression = XPath.newInstance("//srv:Item"); 
itemXpression.addNamespace(ns); 
XPath itemXpression = XPath.newInstance("//srv:ItemName"); 
itemnameXpression.addNamespace(ns); 
XPath itemXpression = XPath.newInstance("//srv:ItemPrice"); 
itempriceXpression.addNamespace(ns); 

List nodes = itemXpression.selectNodes(document);
  
  // Ensure we have some items to iterate through
  if(!nodes.isEmpty()) {
   
    Iterator it = nodes.iterator();
   
    //Init list of Item
    List items = new ArrayList();
   
 //Iterate through all items
 while(it.hasNext()) {

   Element item = (Element)it.next();

   Item i = new Item();
    
   // Evaluate XPath expressions for values
   String itemname = itemnameXpression.valueOf(item);
   String itemprice = itempriceXpression.valueOf(item);
 
   System.out.println(itemname);
   System.out.println(itemprice);
   
   }
  }
}

However, the code always prints out the Name and Price of the first item twice, rather than the details of the first and then the details of the second.
I tried mixing it up with using pure JDOM to no avail, tried numerous different approaches, and ether got NPEs or that first item repeated twice. In the end, the solution was simply to adjust the XPath search expression from "//srv:Item..." to "/srv:Item...". It seems that despite the fact I'm explicitly passing in an Element containing only one Item (and using an XMLOutputter verifies this), preceeding the expression with // resets the search context to the overall document root. Strange behaviour, but there you go.

Useless Eclipse Error Messages

I had an absolute nightmare the other day with a web services project I was working on in Eclipse (actually Spring STS). I had written a WSDL file and accompanying Schema, and Eclipse was giving me a warning triangle at the very top of the document with the following useless error message:

WS-I: A problem occured while running the WS-I WSDL conformance check:
 org.eclipse.wst.wsi.internal.analyzer.WSIAnalyzerException:
 The conformance validation process failed. 

The only further detail available is:

Nested exception is: 
 java.lang.NullPointerException The WSDLAnalyzer was unable to 
 validate the given WSDL File.

No indication of where there might be an error, or what might be causing the problem. After much removing and replacing of segments of the file, and external validation of the Schema document, my advice to anyone else facing this error is to double, triple and quadruple check the spelling and capitalisation of the name fields in the operations within the definition. Changing a few 'c's to 'C's sorted the issue out for me, but methinks its time for some more informative error messages Eclipse Team!

What's the time?

I've been having issues with a number of CentOS hosts not holding the correct time - they seem to be drifting incessantly, despite the fact that they should be syncing to a number of NTP servers. Looking at the output of an ntpdate, I get the following error:

[root@nx-01 ~]# ntpdate
 8 Jun 09:29:09 ntpdate[4267]: no servers can be used, exiting

Although the log messages are showing that the server is quite happily synchronised to its NTP peers:

Jun  8 09:08:20 nx-01 ntpd[1698]: synchronized to LOCAL(0), stratum 10
Jun  8 09:09:25 nx-01 ntpd[1698]: synchronized to 77.75.105.150, stratum 2
Jun  8 09:13:41 nx-01 ntpd[1698]: synchronized to 62.84.188.34, stratum 2

Forcing ntp to update by shutting down the daemon and passing a server as an argument to ntpdate also has no effect - the drift remains.

After some head scratching, I've come across the root cause of this issue. The servers in question are actually DomU instances on a Xen host, and by default they attempt to synchronize their clocks with the Dom0. I've since updated my Dom0's to use correct NTP settings, but for reference if you need to break this dependency you can do it with the following command:

echo 1 > /proc/sys/xen/independent_wallclock

To make it permanent you can append that line to the bottom of /etc/rc.d/rc.local.