Todd Smith Org

April 18, 2009

Configure XenServer 5.0 Free for Software RAID 1

Filed under: Uncategorized — admin @ 4:35 pm

I used the notes from some guy on the Citrix forums. I wish that I had his name so I could give him credit but it’s not here on the stuff I printed out.

On with the show.

To make this work, you need to drives in your system. The second drive must be identical or larger than the first, and you must have installed XenServer to the first drive without selecting the second drive as part of the storage pool.

Install XenServer as usual. Do not select any extra drives as storage pools.

After installation, boot up, and login into console 3 (Alt-F3) as root

type fdisk -l to list the partitions:

Most likely your disks are represented by SCSI device names under linux / XenServer 5.0

in that case your boot disk would be /dev/sda,

To copy the partition table from /dev/sda to /dev/sdb you can use dd

dd if=/dev/sda of=/dev/sdb bs=512 count=1

Now set the partition table up on /dev/sdb the way it should be for Linux RAID. This means setting the partition types to 0xfd.

I used the following command:

echo -e "\nt\n1\nfd\nt\n3\nfd\nw\nx" | fdisk /dev/sdb

That tells says to fdisk, “tag partition 1 as type 0xfd, tag partition 3 as type 0xfd”

Check to make sure the /dev/md? devices are present

[ -e /dev/md0 ] || mknod /dev/md0 b 9 0
[ -e /dev/md1 ] || mknod /dev/md1 b 9 1

Startup the degraded RAID devices

mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb3

The following procedure is directly from the other guys notes. I’ve modified the commands to what I think works a little better.

pvcreate /dev/md1
volume_group=`vgscan | grep VG | awk -F \" '{print $2}'`
vgextend $volume_group /dev/md1
pvmove /dev/sda3 /dev/md1
# If this is a fresh install, then there won't be any data to move
vgreduce $volume_group /dev/sda3

Now we’re ready to copy the filesystem over to the RAID device /dev/md0

mkfs.ext3 /dev/md0
cd / && mount /dev/md0 /mnt && rsync -a --progress --exclude=/sys --exclude=/proc --exclude=/dev/shm --exclude=/dev/pts / /mnt
mkdir /mnt/sys
mkdir /mnt/proc
sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /mnt/etc/fstab

Now let’s setup initrd

mkdir /root/initrd && cd /root/initrd
zcat /boot/initrd-`uname -r`.img | cpio -i && \
cp /lib/modules/`uname -r`/kernel/drivers/md/raid1.ko lib

Now we have to edit the init file

q="echo Waiting for driver initialization."
sed -r -i "s,^${q}$,\n\necho Loading raid1.ko module\ninsmod /lib/raid1.ko\n${q}\n,g" init
q="resume /var/swap/swap.001"
sed -r -i "s,^${q}$,${q}\necho Running raidautorun\nraidautorun /dev/md0\nraidautorun /dev/md1,g" init
r=`grep mkroot /root/initrd/init`
sed -r -i "s|^${r}$|${r/sda1/md0}|g" init

Now we’ll copy the initial ramdisk to the /boot on the new RAID

find . -print | cpio -o -c | gzip -c > /boot/initrd-`uname -r`.img
sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /mnt/etc/fstab
sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /etc/fstab

And setup the boot loader

sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /mnt/boot/extlinux.conf
sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /boot/extlinux.conf
cat /usr/lib/syslinux/mbr.bin > /dev/sdb
cd /mnt && extlinux -i boot/
extlinux -i boot/

If you’ve done this remotely, you can try the following.

cp /mnt/boot/extlinux.conf /boot/
cp /mnt/boot/initrd-`uname -r`.img /boot

Unmount /dev/md0, sync, and reboot

cd ; umount /mnt || umount /dev/md0
sync
reboot

First we tag the partitions as type Linux raid, then we have to add /dev/sda to the RAID.

echo -e "\nt\n1\nfd\nt\n3\nfd\nw\nx" | fdisk /dev/sda
mdadm -a /dev/md0 /dev/sda1
mdadm -a /dev/md1 /dev/sda3

Troubleshooting

The first time I did this procedure I got stuck because my rsync didn’t copy /proc and /sys like I asked it not to. So I had to add the step of creating those two directories so they could be mounted at boot.

After messing around with it a little, I came up with this command line to boot XenServer 5 from extlinux.

mboot.c32 /boot/xen.gz dom0_mem=752M --- /boot/vmlinuz-2.6-xen root=/dev/md0  single --- /boot/initrd-2.6-xen.img

22 Comments »

  1. Not booting. Can’t find /dev/root etc.

    Comment by Ilja — May 26, 2009 @ 6:45 am

  2. Did you change your /etc/fstab to point to /dev/md?

    Also, you want to make sure to follow the steps to fixup the initrd to ensure that it is loading the correct kernel modules.

    Comment by admin — May 26, 2009 @ 7:00 am

  3. Ok, found the spot:

    This:
    find . -print | cpio -o -c | gzip -c > /boot/initrd-`uname -r`.img

    Should be:
    find . -print | cpio -o -c | gzip -c > /mnt/boot/initrd-`uname -r`.img

    This is done twise in your description. No need:
    sed -r -i ‘s,LABEL=root-\w+ ,/dev/md0 ,g’ /mnt/etc/fstab

    This is also done twise:
    extlinux -i boot/

    Comment by Ilja — May 26, 2009 @ 8:33 am

  4. Please let me know if this is working 100% because I wan’t to set this up in 2 production servers. Also, I need Xen to boot automatically from the other drive in case one of the them is damaged. And mon should send an email when this happens.

    Please let me know the correct steps on this.

    Keep up the good work :)

    Comment by organetic — June 10, 2009 @ 5:41 am

  5. It’s working for me in production environments. It’s Linux software RAID which is very good RAID code. Also this configuration boots off of the RAID device. So as long as at least one of your disks are good, the system should boot.

    Comment by admin — June 10, 2009 @ 6:48 am

  6. After following your tutorial including the adjustment pointed out by Ilja, I have a successful install of Xen w/ software RAID 1. At this point I am trying to re-build the array in another (idnetical box). When connecting using XenCenter I get an error saying that it cannot connect to both Xen boxes because one is a backup of the other. Do you know what I can change so the boxes can both be connected to simultaneously?

    Comment by tdb — June 22, 2009 @ 11:11 am

  7. Andrew,

    I’m sorry I don’t know what needs to be changed. I’m sure it’s some identifier that is unique to each installation. If you find out, please let me know.

    -Todd

    Comment by admin — June 23, 2009 @ 9:20 pm

  8. tdb, it could be that extlinux is written only on the first drive. When raid is built, you need to write “boot loader code” to the second HDD. Unfortunately I don’t know how to do it with extlinux, haven’t work with it much.

    With 2 HDD configuration using grub it would be:
    # grub
    grub> device (hd1) /dev/sdb
    device (hd1) /dev/sdb
    grub> root (hd1,0)
    root (hd1,0)
    Filesystem type is ext2fs, partition type 0xfd
    grub> setup (hd1)
    setup (hd1)
    Checking if “/boot/grub/stage1″ exists… no
    Checking if “/grub/stage1″ exists… yes
    Checking if “/grub/stage2″ exists… yes
    Checking if “/grub/e2fs_stage1_5″ exists… yes
    Running “embed /grub/e2fs_stage1_5 (hd1)”… 15 sectors are embedded.
    succeeded
    Running “install /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/grub.conf”… succeeded
    Done.
    grub> quit
    #

    Good luck.

    - Ilja

    Comment by Ilja — June 29, 2009 @ 12:23 pm

  9. Hi. Can you advise whether this works for Free XenServer 5.5

    Comment by David — July 18, 2009 @ 8:06 am

  10. Excellent howto!

    Just a suggestion, but perhaps instead of minimizing the amount of enter presses using && and |, suggest breaking the commands out individually and explaining what each one does. This would be useful in a future XS version if something changed, one could engineer around the changes if the steps were more discrete.

    Comment by Coop — August 14, 2009 @ 8:00 pm

  11. I think that I was the above mentioned guy with initial instructions on citrix forums :) Original how-to is here:

    http://dmit.lv/xs50raid/XenServer-on-RAID.txt

    Coop:
    Present how-to is a bit shorter and more “geek-friendly” then mine in terms of used instruments and commands. Though when I first published the whole process I tried to make it more explanatory and specific so anyone interested could understand the global idea and repeat the steps for upcoming versions of Xen Server.

    But though it is still possible with current 5.5 version I would not suggest doing this in the future. New licensing policy requires to obtain a free one year license which you should renew on annual basis. I’m afraid that upcoming license will not apply on the system without current patches installed. But if you apply any patch this will break your software RAID.

    Comment by Dmitry Komarov — August 19, 2009 @ 8:12 am

  12. The link above is not on-line right now (migrating to new XS) so here is the working one:

    http://core.bofh.lv/XenServer-on-RAID.txt

    Comment by Dmitry Komarov — August 20, 2009 @ 2:20 am

  13. Hi, I followed all the steps, the ones in the comments too, and it is booting ok and working ok, but I have a doubt, when I type fdisk -l in order to see the output it seems something wrong, I post it here so you can see too:
    Can you give me some advice?

    Thanks for your help, and if you could explain the steps it would be awesome, since I am just learning Linux.

    [root@virtual ~]# fdisk -l

    Disk /dev/sda: 160.0 GB, 160041885696 bytes
    255 heads, 63 sectors/track, 19457 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/sda1 * 1 499 4008186 fd Linux raid autodetect
    /dev/sda2 500 998 4008217+ 83 Linux
    /dev/sda3 999 19457 148271917+ fd Linux raid autodetect

    Disk /dev/sdb: 160.0 GB, 160041885696 bytes
    255 heads, 63 sectors/track, 19457 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/sdb1 * 1 499 4008186 fd Linux raid autodetect
    /dev/sdb2 500 998 4008217+ 83 Linux
    /dev/sdb3 999 19457 148271917+ fd Linux raid autodetect

    Disk /dev/md0: 4104 MB, 4104257536 bytes
    2 heads, 4 sectors/track, 1002016 cylinders
    Units = cylinders of 8 * 512 = 4096 bytes

    This doesn’t look like a partition table
    Probably you selected the wrong device.

    Device Boot Start End Blocks Id System
    /dev/md0p1 ? 402920848 433755839 123339962 78 Unknown
    Partition 1 does not end on cylinder boundary.
    /dev/md0p2 ? 54108890 151069367 387841909+ 10 OPUS
    Partition 2 does not end on cylinder boundary.
    /dev/md0p3 ? 233695321 473599079 959615034 8b Unknown
    Partition 3 does not end on cylinder boundary.
    /dev/md0p4 ? 404488193 405528591 4161595+ a OS/2 Boot Manager
    Partition 4 does not end on cylinder boundary.

    Partition table entries are not in disk order

    Disk /dev/md1: 151.8 GB, 151830331392 bytes
    2 heads, 4 sectors/track, 37067952 cylinders
    Units = cylinders of 8 * 512 = 4096 bytes

    Disk /dev/md1 doesn’t contain a valid partition table
    [root@virtual ~]#

    Comment by Jorge Zamit — September 23, 2009 @ 1:13 pm

  14. It was going so well with 5.5, I don’t know where I went wrong!

    I followed through all the steps, and the mirror came up, rebuilt and was running clean, yay!

    I assumed all was good, so spent a day creating my virtual machines.

    But then I rebooted the physical server to move it, and that was the last time it booted. :’(

    When you hit escape on the blue boot splash screen, this is what come up:

    > map: vt01 => fb0
    > could not find filesystem: ‘/dev/root’
    > setuproot: moving /dev failed – no such file or directory
    > setuproot: error mounting /proc – no such file or directory
    > setuproot: error mounting /sys – no such file or directory
    > switchroot: mount failed – no such file or directory
    > kernel panic – not syncing: attempt to kill init!

    So, I tried the troubleshooting step, which gets much further, but then panics anyway. I cant pause the screen, and everything goes by so fast. This is one of the errors that I saw:

    Extfs unreadable superblock

    BLEH. I guess I will go find a live boot CD of some sort, and see what else I can break. :(

    Comment by Cody — October 26, 2009 @ 2:47 pm

  15. Phwew!!! Got version 5.5 working again!!

    I used the text file mentioned above:
    http://core.bofh.lv/XenServer-on-RAID.txt
    another hard drive with a fresh install of xenserver
    and a centos live CD

    The problem was that I blindly copied and pasted this entire setup above hoping for the best. I really don’t know why it booted in the first place, and I frankly don’t care enough to figure it out.

    Here is what I did.

    made a fresh install of xenserver on another hard drive
    booted onto the centos liveCD with all 3 hard drives installed
    started at step 6 of the text instructions, creating a new initrd ramdisk.

    I mounted md0 to /mnt/disk/md0. /dev/sdc1 was already mounted to /mnt/disk/sdc1
    I used the fresh install to get the source files and replaced the files on the raid array.

    And it booted without a hitch.

    Comment by Cody — October 27, 2009 @ 1:48 pm

  16. Fantastic info!

    I’m been trying to get this to work in a 5.5 server and first reboot works fine. It seems as soon as I add the sda1 partition to md0 things fall apart on boot? It just keeps rebooting. I used the troubleshooting boot command above and I get a kernel panic and a message about I/O resource already in use.

    Any ideas?

    Comment by Jim — November 28, 2009 @ 2:04 pm

  17. I think your instructions are both based on the work of Harry de Jong and Stephan Groenewegen from correct.net. They posted their instructions for RAID 1 under XenEnterprise 3.1 on January 9th, 2007 in the XenSource forums. Thank you guys!

    Comment by Reginald Nagle — December 3, 2009 @ 2:33 am

  18. I just did the whole procedure, paying attention to Ilja’s comments and it worked like a charm!

    Only thing that worries me a bit is the remark about “boot loader code” written to one disk.

    Comment by Roel — December 3, 2009 @ 2:26 pm

  19. I just wanted to say thank you to the author and other contributors. I’ve used this guide at least a dozen times.

    For future readers: Be careful with the mount and then the copying over of the sda filesystem to the software raid device if you’ve already added the server to an attached storage repository. Without providing the proper rsync exclude it will attempt to copy over the contents of your storage repository and quickly fill your disk. I did this mindlessly once and decided I would just delete the extra files off the disk after unmounting the shared storage. Unfortunately, i forgot to unmount the shared storage. yeah, that sucked. :)

    Comment by Russ — December 8, 2009 @ 12:05 pm

  20. How to install xenserver 5.5 with Intel Embedded (LSI) raid 1 configuration?
    1. Need any drivers for that?
    2. Its not possible to install?

    Comment by raja — December 17, 2009 @ 12:07 pm

  21. Fucking great!

    tested on XenServer 5.5 and works flawlessly. I did most of the work by my self though… those sed and awk seemed to complex to type my hand… :p

    Comment by Kodomo — March 18, 2010 @ 7:59 am

RSS feed for comments on this post. TrackBack URL

Leave a comment

Powered by WordPress