Buildslave Osnabrück

The Buildslave Osnabrück is one of two Xen virtual machines hosted on... well, a Xen host. It's compiling ReactOS debug builds.

Current Status

Running. The debug buildslave currently resides together with the release buildslave on Christophs Server. At the moment, this is for testing purposes and can be reverted if required; if I successfully solve all remaining problems, then it will stay this way for a few days to give me time to reinstall the Xen server. --Martin von Wittich 23:13, 24 November 2007 (CET)

Future

The current configuration of the whole server has proved problematic. Therefore there are some modifications planned:

• Gentoo instead of Fedora Core as operating system. Well, it's a personal preference of mine, I'm just not too happy with FC. Plus, emerging will be a pleasure with 8 cores :-)
• Kernel update. The current kernel wastes plenty power because it doesn't feature dynamic ticks and such.
• ccache. Should improve compile times greatly. done, I was right.
• New hard disk configuration. The current RAID5 wastes power too because no hard disk can spin down. I will reinstall the whole system on the first disk, using the other two as a backup storage in RAID1.
• No VM for ReactOS anymore. I was thinking about changing to KVM because it's supported in vanilla kernels, but after looking at a few benchmarks I guess it's best to abandon such gadgetry.
• ISO creation on the download server. At the moment, the ISO files are generated on the build servers, and then rsynced onto the download server. That's better than uploading via FTP, but only transferring single files and creating & zipping the ISO files on the download server would be even better.

Downtimes

Start date (UTC)	End date (UTC)	Reason
21.11.2007 15:20	21.11.2007 17:10	Oh my, moving the whole buildbot folder onto the RAM disk was a bad idea actually. We were repeatedly surprised after restarts that our script changes were missing (changed them on the RAM disk and forgot to copy them back to the hard disk), this time the uploadiso script was reverted, leading to corrupted ISO files on the download server. I moved the folder back on the hard disk for the sake of stability; obj, output and now ccache remain on the RAM disk. ccache is now installed on the debug server too, reducing the build time from ~5 minutes to ~2 minutes.
16.11.2007 18:10	16.11.2007 18:2519:20	Reboot. Oh my, the expected new hard disk didn't show up. Investigating, expect another reboot soon. Fixed it, took me almost a whole hour longer than expected. Looks like only 4 out of 6 SATA ports work atm. Enabling AHCI support in the BIOS should fix this, but Fedora doesn't seem to support this (kernel panic). Anyway, I'm going to install Gentoo sooner or later, than I will do this properly.
01.10.2007 ~15:00	01.10.2007 ~21:00	Repeated power failures led to a server shutdown, and I had forgotten to a) configure the initscript to rebuild the RAM disk after boot and b) copy some new scripts from the RAM disk back to the hard disk. Took me some time to notice the buildbot wasn't coming back online and then to fix these issues.
05.09.2007 07:55	05.09.2007 08:30	I'll try to move the whole buildbot onto the RAM disk, not only the object files. Only the buildbot will have ocassionally downtimes, the VM will stay online.
04.09.2007 21:05	04.09.2007 21:10	The RAM disk is not working as expected. I will setup a tmpfs instead and reboot the VM, so it can use all available memory.
25.08.2007 16:00	25.08.2007 17:00	Setting up a RAM disk for the Reactos VM.
28.06.2007 15:00	28.06.2007 18:00	Fixed remaining problems with the RAID configuration. The new configuration reduced the build time from 9:00 to 7:40.
27.06.2007 21:30	28.06.2007 15:00	Installed a UPS and apcupsd. Upgraded the Reactos VM to FC7. Configured a serial console for the Xen host to simplify future maintenance work. Changed the RAID0 configuration of the Reactos VM. Cleaned the fans.

Legend:

    scheduled downtime that has not occured yet

    scheduled downtime that has occured

    unscheduled downtime that is occuring right now

    unscheduled downtime that has occured

Hardware

Mainboard	Asus DSBF-DE
CPU	2x Intel Xeon 5310
Memory	Kingston ValueRAM FB-DIMM 8 GB DDR2-667
Hard disks	3x 305GiB, 1x 27GiB /dev/md0: RAID5 (3x 297GiB); /dev/md1: RAID0 (3x 1027MiB)
Host operating system	FC7
VM operating system	FC7

History

• May 2007: We finally got a replacement mainboard from Alternate, not the one we sent in, but a very similar one, so the server is now back up. The object files are now placed on a RAID0 with three disks, improviving the build time from ~15 to ~10 minutes.
• March 2007: As long as the new server doesn't work, the old server is back online.
• February 2007: Martin comes up with the idea to put plenty new SATA hard disks into the server, but the old mainboard has no SATA support, so the whole server needs to be replaced. The ReactOS team decides to co-fund the upgrade and instead of consumer level hardware, a full-blown Xeon QuadCore server is bought. Due to problems with the online shop, it took a few weeks to get the new server up and running - only to find out that the mainboard sometimes simply won't boot and has to be sent back for replacement.
• Long ago: Christoph installs the buildbot on Martin's private server, then an old AMD 800MHz with 512 MB RAM. The RAM is later upgraded to 1024MB, the CPU is replaced with a Sempron 2500+.