I used the notes from some guy on the Citrix forums. I wish that I had his name so I could give him credit but it’s not here on the stuff I printed out.
On with the show.
To make this work, you need to drives in your system. The second drive must be identical or larger than the first, and you must have installed XenServer to the first drive without selecting the second drive as part of the storage pool.
Install XenServer as usual. Do not select any extra drives as storage pools.
After installation, boot up, and login into console 3 (Alt-F3) as root
type fdisk -l to list the partitions:
Most likely your disks are represented by SCSI device names under linux / XenServer 5.0
in that case your boot disk would be /dev/sda,
To copy the partition table from /dev/sda to /dev/sdb you can use dd
dd if=/dev/sda of=/dev/sdb bs=512 count=1
Now set the partition table up on /dev/sdb the way it should be for Linux RAID. This means setting the partition types to 0xfd.
I used the following command:
echo -e "\nt\n1\nfd\nt\n3\nfd\nw\nx" | fdisk /dev/sdb
That tells says to fdisk, “tag partition 1 as type 0xfd, tag partition 3 as type 0xfd”
Check to make sure the /dev/md? devices are present
[ -e /dev/md0 ] || mknod /dev/md0 b 9 0 [ -e /dev/md1 ] || mknod /dev/md1 b 9 1
Startup the degraded RAID devices
mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1 mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb3
The following procedure is directly from the other guys notes. I’ve modified the commands to what I think works a little better.
pvcreate /dev/md1
volume_group=`vgscan | grep VG | awk -F \" '{print $2}'`
vgextend $volume_group /dev/md1
pvmove /dev/sda3 /dev/md1
# If this is a fresh install, then there won't be any data to move
vgreduce $volume_group /dev/sda3
Now we’re ready to copy the filesystem over to the RAID device /dev/md0
mkfs.ext3 /dev/md0 cd / && mount /dev/md0 /mnt && rsync -a --progress --exclude=/sys --exclude=/proc --exclude=/dev/shm --exclude=/dev/pts / /mnt mkdir /mnt/sys mkdir /mnt/proc
sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /mnt/etc/fstab
Now let’s setup initrd
mkdir /root/initrd && cd /root/initrd zcat /boot/initrd-`uname -r`.img | cpio -i && \ cp /lib/modules/`uname -r`/kernel/drivers/md/raid1.ko lib
Now we have to edit the init file
q="echo Waiting for driver initialization."
sed -r -i "s,^${q}$,\n\necho Loading raid1.ko module\ninsmod /lib/raid1.ko\n${q}\n,g" init
q="resume /var/swap/swap.001"
sed -r -i "s,^${q}$,${q}\necho Running raidautorun\nraidautorun /dev/md0\nraidautorun /dev/md1,g" init
r=`grep mkroot /root/initrd/init`
sed -r -i "s|^${r}$|${r/sda1/md0}|g" init
Now we’ll copy the initial ramdisk to the /boot on the new RAID
find . -print | cpio -o -c | gzip -c > /boot/initrd-`uname -r`.img sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /mnt/etc/fstab sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /etc/fstab
And setup the boot loader
sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /mnt/boot/extlinux.conf sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /boot/extlinux.conf cat /usr/lib/syslinux/mbr.bin > /dev/sdb cd /mnt && extlinux -i boot/ extlinux -i boot/
If you’ve done this remotely, you can try the following.
cp /mnt/boot/extlinux.conf /boot/ cp /mnt/boot/initrd-`uname -r`.img /boot
Unmount /dev/md0, sync, and reboot
cd ; umount /mnt || umount /dev/md0 sync reboot
First we tag the partitions as type Linux raid, then we have to add /dev/sda to the RAID.
echo -e "\nt\n1\nfd\nt\n3\nfd\nw\nx" | fdisk /dev/sda mdadm -a /dev/md0 /dev/sda1 mdadm -a /dev/md1 /dev/sda3
Troubleshooting
The first time I did this procedure I got stuck because my rsync didn’t copy /proc and /sys like I asked it not to. So I had to add the step of creating those two directories so they could be mounted at boot.
After messing around with it a little, I came up with this command line to boot XenServer 5 from extlinux.
mboot.c32 /boot/xen.gz dom0_mem=752M --- /boot/vmlinuz-2.6-xen root=/dev/md0 single --- /boot/initrd-2.6-xen.img
Not booting. Can’t find /dev/root etc.
Comment by Ilja — May 26, 2009 @ 6:45 am
Did you change your /etc/fstab to point to /dev/md?
Also, you want to make sure to follow the steps to fixup the initrd to ensure that it is loading the correct kernel modules.
Comment by admin — May 26, 2009 @ 7:00 am
Ok, found the spot:
This:
find . -print | cpio -o -c | gzip -c > /boot/initrd-`uname -r`.img
Should be:
find . -print | cpio -o -c | gzip -c > /mnt/boot/initrd-`uname -r`.img
This is done twise in your description. No need:
sed -r -i ‘s,LABEL=root-\w+ ,/dev/md0 ,g’ /mnt/etc/fstab
This is also done twise:
extlinux -i boot/
Comment by Ilja — May 26, 2009 @ 8:33 am
Please let me know if this is working 100% because I wan’t to set this up in 2 production servers. Also, I need Xen to boot automatically from the other drive in case one of the them is damaged. And mon should send an email when this happens.
Please let me know the correct steps on this.
Keep up the good work
Comment by organetic — June 10, 2009 @ 5:41 am
It’s working for me in production environments. It’s Linux software RAID which is very good RAID code. Also this configuration boots off of the RAID device. So as long as at least one of your disks are good, the system should boot.
Comment by admin — June 10, 2009 @ 6:48 am
After following your tutorial including the adjustment pointed out by Ilja, I have a successful install of Xen w/ software RAID 1. At this point I am trying to re-build the array in another (idnetical box). When connecting using XenCenter I get an error saying that it cannot connect to both Xen boxes because one is a backup of the other. Do you know what I can change so the boxes can both be connected to simultaneously?
Comment by tdb — June 22, 2009 @ 11:11 am
Andrew,
I’m sorry I don’t know what needs to be changed. I’m sure it’s some identifier that is unique to each installation. If you find out, please let me know.
-Todd
Comment by admin — June 23, 2009 @ 9:20 pm
tdb, it could be that extlinux is written only on the first drive. When raid is built, you need to write “boot loader code” to the second HDD. Unfortunately I don’t know how to do it with extlinux, haven’t work with it much.
With 2 HDD configuration using grub it would be:
# grub
grub> device (hd1) /dev/sdb
device (hd1) /dev/sdb
grub> root (hd1,0)
root (hd1,0)
Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd1)
setup (hd1)
Checking if “/boot/grub/stage1″ exists… no
Checking if “/grub/stage1″ exists… yes
Checking if “/grub/stage2″ exists… yes
Checking if “/grub/e2fs_stage1_5″ exists… yes
Running “embed /grub/e2fs_stage1_5 (hd1)”… 15 sectors are embedded.
succeeded
Running “install /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/grub.conf”… succeeded
Done.
grub> quit
#
Good luck.
- Ilja
Comment by Ilja — June 29, 2009 @ 12:23 pm
Hi. Can you advise whether this works for Free XenServer 5.5
Comment by David — July 18, 2009 @ 8:06 am
Excellent howto!
Just a suggestion, but perhaps instead of minimizing the amount of enter presses using && and |, suggest breaking the commands out individually and explaining what each one does. This would be useful in a future XS version if something changed, one could engineer around the changes if the steps were more discrete.
Comment by Coop — August 14, 2009 @ 8:00 pm
I think that I was the above mentioned guy with initial instructions on citrix forums
Original how-to is here:
http://dmit.lv/xs50raid/XenServer-on-RAID.txt
Coop:
Present how-to is a bit shorter and more “geek-friendly” then mine in terms of used instruments and commands. Though when I first published the whole process I tried to make it more explanatory and specific so anyone interested could understand the global idea and repeat the steps for upcoming versions of Xen Server.
But though it is still possible with current 5.5 version I would not suggest doing this in the future. New licensing policy requires to obtain a free one year license which you should renew on annual basis. I’m afraid that upcoming license will not apply on the system without current patches installed. But if you apply any patch this will break your software RAID.
Comment by Dmitry Komarov — August 19, 2009 @ 8:12 am
The link above is not on-line right now (migrating to new XS) so here is the working one:
http://core.bofh.lv/XenServer-on-RAID.txt
Comment by Dmitry Komarov — August 20, 2009 @ 2:20 am
Hi, I followed all the steps, the ones in the comments too, and it is booting ok and working ok, but I have a doubt, when I type fdisk -l in order to see the output it seems something wrong, I post it here so you can see too:
Can you give me some advice?
Thanks for your help, and if you could explain the steps it would be awesome, since I am just learning Linux.
[root@virtual ~]# fdisk -l
Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 499 4008186 fd Linux raid autodetect
/dev/sda2 500 998 4008217+ 83 Linux
/dev/sda3 999 19457 148271917+ fd Linux raid autodetect
Disk /dev/sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 499 4008186 fd Linux raid autodetect
/dev/sdb2 500 998 4008217+ 83 Linux
/dev/sdb3 999 19457 148271917+ fd Linux raid autodetect
Disk /dev/md0: 4104 MB, 4104257536 bytes
2 heads, 4 sectors/track, 1002016 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
This doesn’t look like a partition table
Probably you selected the wrong device.
Device Boot Start End Blocks Id System
/dev/md0p1 ? 402920848 433755839 123339962 78 Unknown
Partition 1 does not end on cylinder boundary.
/dev/md0p2 ? 54108890 151069367 387841909+ 10 OPUS
Partition 2 does not end on cylinder boundary.
/dev/md0p3 ? 233695321 473599079 959615034 8b Unknown
Partition 3 does not end on cylinder boundary.
/dev/md0p4 ? 404488193 405528591 4161595+ a OS/2 Boot Manager
Partition 4 does not end on cylinder boundary.
Partition table entries are not in disk order
Disk /dev/md1: 151.8 GB, 151830331392 bytes
2 heads, 4 sectors/track, 37067952 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md1 doesn’t contain a valid partition table
[root@virtual ~]#
Comment by Jorge Zamit — September 23, 2009 @ 1:13 pm
It was going so well with 5.5, I don’t know where I went wrong!
I followed through all the steps, and the mirror came up, rebuilt and was running clean, yay!
I assumed all was good, so spent a day creating my virtual machines.
But then I rebooted the physical server to move it, and that was the last time it booted. :’(
When you hit escape on the blue boot splash screen, this is what come up:
> map: vt01 => fb0
> could not find filesystem: ‘/dev/root’
> setuproot: moving /dev failed – no such file or directory
> setuproot: error mounting /proc – no such file or directory
> setuproot: error mounting /sys – no such file or directory
> switchroot: mount failed – no such file or directory
> kernel panic – not syncing: attempt to kill init!
So, I tried the troubleshooting step, which gets much further, but then panics anyway. I cant pause the screen, and everything goes by so fast. This is one of the errors that I saw:
Extfs unreadable superblock
BLEH. I guess I will go find a live boot CD of some sort, and see what else I can break.
Comment by Cody — October 26, 2009 @ 2:47 pm
Phwew!!! Got version 5.5 working again!!
I used the text file mentioned above:
http://core.bofh.lv/XenServer-on-RAID.txt
another hard drive with a fresh install of xenserver
and a centos live CD
The problem was that I blindly copied and pasted this entire setup above hoping for the best. I really don’t know why it booted in the first place, and I frankly don’t care enough to figure it out.
Here is what I did.
made a fresh install of xenserver on another hard drive
booted onto the centos liveCD with all 3 hard drives installed
started at step 6 of the text instructions, creating a new initrd ramdisk.
I mounted md0 to /mnt/disk/md0. /dev/sdc1 was already mounted to /mnt/disk/sdc1
I used the fresh install to get the source files and replaced the files on the raid array.
And it booted without a hitch.
Comment by Cody — October 27, 2009 @ 1:48 pm
Fantastic info!
I’m been trying to get this to work in a 5.5 server and first reboot works fine. It seems as soon as I add the sda1 partition to md0 things fall apart on boot? It just keeps rebooting. I used the troubleshooting boot command above and I get a kernel panic and a message about I/O resource already in use.
Any ideas?
Comment by Jim — November 28, 2009 @ 2:04 pm
I think your instructions are both based on the work of Harry de Jong and Stephan Groenewegen from correct.net. They posted their instructions for RAID 1 under XenEnterprise 3.1 on January 9th, 2007 in the XenSource forums. Thank you guys!
Comment by Reginald Nagle — December 3, 2009 @ 2:33 am
I just did the whole procedure, paying attention to Ilja’s comments and it worked like a charm!
Only thing that worries me a bit is the remark about “boot loader code” written to one disk.
Comment by Roel — December 3, 2009 @ 2:26 pm
I just wanted to say thank you to the author and other contributors. I’ve used this guide at least a dozen times.
For future readers: Be careful with the mount and then the copying over of the sda filesystem to the software raid device if you’ve already added the server to an attached storage repository. Without providing the proper rsync exclude it will attempt to copy over the contents of your storage repository and quickly fill your disk. I did this mindlessly once and decided I would just delete the extra files off the disk after unmounting the shared storage. Unfortunately, i forgot to unmount the shared storage. yeah, that sucked.
Comment by Russ — December 8, 2009 @ 12:05 pm
How to install xenserver 5.5 with Intel Embedded (LSI) raid 1 configuration?
1. Need any drivers for that?
2. Its not possible to install?
Comment by raja — December 17, 2009 @ 12:07 pm
Fucking great!
tested on XenServer 5.5 and works flawlessly. I did most of the work by my self though… those sed and awk seemed to complex to type my hand… :p
Comment by Kodomo — March 18, 2010 @ 7:59 am