Difference between revisions of "Buildslave Osnabrück"

From ReactOS Wiki
Jump to: navigation, search
m
 
(19 intermediate revisions by 4 users not shown)
Line 1: Line 1:
__NOTOC__
+
The '''[[Buildbot|Buildslave]] [[wikipedia:Osnabrück|Osnabrück]]''' compiles ReactOS debug builds. Contrary to the name, it is now actually located in [[wikipedia:Braunschweig|Braunschweig]] (hosted by the company I'm working for, IServ)... I should consider to rename it :)
The '''[[Buildbot|Buildslave]] [[wikipedia:Osnabrück|Osnabrück]]''' is one of two Xen virtual machines hosted on... well, a Xen host. It's compiling ReactOS debug builds.
 
  
 
== Current Status ==
 
== Current Status ==
<del>Running.</del>
+
Running as [[Buildslave Lammhult]].
The debug buildslave currently resides together with the release buildslave on Christophs Server. At the moment, this is for testing purposes and can be reverted if required; if I successfully solve all remaining problems, then it will stay this way for a few days to give me time to reinstall the Xen server. --[[User:Martin von Wittich|Martin von Wittich]] 23:13, 24 November 2007 (CET)
 
 
 
== Future ==
 
The current configuration of the whole server has proved problematic. Therefore there are some modifications planned:
 
* Gentoo instead of Fedora Core as operating system. Well, it's a personal preference of mine, I'm just not too happy with FC. Plus, emerging will be a pleasure with 8 cores :-)
 
* Kernel update. The current kernel wastes plenty power because it doesn't feature dynamic ticks and such.
 
* <del>ccache. Should improve compile times greatly.</del> done, I was right.
 
* New hard disk configuration. The current RAID5 wastes power too because no hard disk can spin down. I will reinstall the whole system on the first disk, using the other two as a backup storage in RAID1.
 
* No VM for ReactOS anymore. I was thinking about changing to KVM because it's supported in vanilla kernels, but after looking at a [http://www.phoronix.com/scan.php?page=article&item=623&num=3 few benchmarks] I guess it's best to abandon such gadgetry.
 
* ISO creation on the download server. At the moment, the ISO files are generated on the build servers, and then rsynced onto the download server. That's better than uploading via FTP, but only transferring single files and creating & zipping the ISO files on the download server would be even better.
 
  
 
== Downtimes ==
 
== Downtimes ==
{| class="prettytable"
+
{| class="wikitable"
 
|-
 
|-
 
! Start&nbsp;date&nbsp;(UTC)
 
! Start&nbsp;date&nbsp;(UTC)
 
! End&nbsp;date&nbsp;(UTC)
 
! End&nbsp;date&nbsp;(UTC)
 
! Reason
 
! Reason
 +
 +
|- style="background-color: silver;"
 +
| 21 October 2011
 +
| 2 November 2011
 +
|
 +
* Relocation to Lammhult.
 +
 +
|- style="background-color: gray;"
 +
| 3 June 2008
 +
| 4 June 2008
 +
|
 +
* The buildslave daemon was running twice for unknown reasons; both instances were interfering with each other and spammed the BuildBot status page. I killed all running instances and restarted the daemon.
 +
 +
|- style="background-color: silver;"
 +
| 30 March 2008&nbsp;11:20
 +
| 31 March 2008&nbsp;15:00
 +
|
 +
* Relocation to Braunschweig.
 +
 +
|- style="background-color: silver;"
 +
| 6 December 2007&nbsp;15:30
 +
| 6 December 2007&nbsp;<del>16:30</del> <del>18:30</del> <del>19:30</del> 20:00
 +
|
 +
* Kernel update -> linux-2.6.24-gentoo, and a few minor other things. Will see whether I can fix that buildbot chaos my brother has made... EDIT: need more time...
 +
 +
EDIT: still need more time. The new kernel is running fine, I've even correctly configured grub for falling back to an older kernel when a kernel update should fail. Now I still gotta fix that buildbot setup. I'll try to leave it running most of the time, though.
 +
 +
EDIT: I'm done now. I've fixed the buildbot mess, it's installed via portage again now. Wasn't able to fix the non-working serial console in grub though :/
 +
 +
YET ANOTHER EDIT: fixed some permissions problems my brother had caused.
 +
 +
|- style="background-color: silver;"
 +
| 6 December 2007&nbsp;10:25
 +
| 6 December 2007&nbsp;10:30
 +
|
 +
* Kernel update -> linux-2.6.23-gentoo-r3
  
 
|- style="background-color: gray;"
 
|- style="background-color: gray;"
| 21.11.2007&nbsp;15:20
+
| 5 December 2007&nbsp;21:00
| 21.11.2007&nbsp;17:10
+
| 5 December 2007&nbsp;21:45
 +
|
 +
* A server reboot caused the domain name resolver to malfunction; Gentoo overwrites the /etc/resolv.conf file on each reboot, and thereby removed my nameserver setting. I put the nameserver address into /etc/conf.d/net so Gentoo will now generate corrent resolv.conf files.
 +
 
 +
|- style="background-color: silver;"
 +
| 1 December 2007&nbsp;20:10
 +
| 1 December 2007&nbsp;<del>20:40</del>22:30
 +
|
 +
* <del>Going to test if ReactOS will now build on 64 bit Gentoo.</del>Oh my, we were watching a film and I forgot that I took the buildslave offline... it's now back online, running as 64bit :)
 +
 
 +
|- style="background-color: gray;"
 +
| 21 November 2007&nbsp;15:20
 +
| 21 November 2007&nbsp;17:10
 
|
 
|
 
* Oh my, moving the whole buildbot folder onto the RAM disk was a bad idea actually. We were repeatedly surprised after restarts that our script changes were missing (changed them on the RAM disk and forgot to copy them back to the hard disk), this time the uploadiso script was reverted, leading to corrupted ISO files on the download server. I moved the folder back on the hard disk for the sake of stability; obj, output and now ccache remain on the RAM disk. ccache is now installed on the debug server too, reducing the build time from ~5 minutes to ~2 minutes.
 
* Oh my, moving the whole buildbot folder onto the RAM disk was a bad idea actually. We were repeatedly surprised after restarts that our script changes were missing (changed them on the RAM disk and forgot to copy them back to the hard disk), this time the uploadiso script was reverted, leading to corrupted ISO files on the download server. I moved the folder back on the hard disk for the sake of stability; obj, output and now ccache remain on the RAM disk. ccache is now installed on the debug server too, reducing the build time from ~5 minutes to ~2 minutes.
  
 
|- style="background-color: silver;"
 
|- style="background-color: silver;"
| 16.11.2007&nbsp;18:10
+
| 16 November 2007&nbsp;18:10
| 16.11.2007&nbsp;<del>18:25</del>19:20
+
| 16 November 2007&nbsp;<del>18:25</del>19:20
 
|
 
|
 
* <del>Reboot.</del> Oh my, the expected new hard disk didn't show up. <del>Investigating, expect another reboot soon.</del> Fixed it, took me almost a whole hour longer than expected. Looks like only 4 out of 6 SATA ports work atm. Enabling AHCI support in the BIOS should fix this, but Fedora doesn't seem to support this (kernel panic). Anyway, I'm going to install Gentoo sooner or later, than I will do this properly.
 
* <del>Reboot.</del> Oh my, the expected new hard disk didn't show up. <del>Investigating, expect another reboot soon.</del> Fixed it, took me almost a whole hour longer than expected. Looks like only 4 out of 6 SATA ports work atm. Enabling AHCI support in the BIOS should fix this, but Fedora doesn't seem to support this (kernel panic). Anyway, I'm going to install Gentoo sooner or later, than I will do this properly.
  
 
|- style="background-color: grey;"
 
|- style="background-color: grey;"
| 01.10.2007&nbsp;~15:00
+
| 1 October 2007&nbsp;~15:00
| 01.10.2007&nbsp;~21:00
+
| 1 October 2007&nbsp;~21:00
 
|
 
|
 
* Repeated power failures led to a server shutdown, and I had forgotten to a) configure the initscript to rebuild the RAM disk after boot and b) copy some new scripts from the RAM disk back to the hard disk. Took me some time to notice the buildbot wasn't coming back online and then to fix these issues.
 
* Repeated power failures led to a server shutdown, and I had forgotten to a) configure the initscript to rebuild the RAM disk after boot and b) copy some new scripts from the RAM disk back to the hard disk. Took me some time to notice the buildbot wasn't coming back online and then to fix these issues.
 
|- style="background-color: silver;"
 
|- style="background-color: silver;"
| 05.09.2007&nbsp;07:55
+
| 5 September 2007&nbsp;07:55
| 05.09.2007&nbsp;08:30
+
| 5 September 2007&nbsp;08:30
 
|
 
|
 
* I'll try to move the whole buildbot onto the RAM disk, not only the object files. Only the buildbot will have ocassionally downtimes, the VM will stay online.
 
* I'll try to move the whole buildbot onto the RAM disk, not only the object files. Only the buildbot will have ocassionally downtimes, the VM will stay online.
 
|- style="background-color: silver;"
 
|- style="background-color: silver;"
| 04.09.2007&nbsp;21:05
+
| 4 September 2007&nbsp;21:05
| 04.09.2007&nbsp;21:10
+
| 4 September 2007&nbsp;21:10
 
|
 
|
 
* The RAM disk is not working as expected. I will setup a tmpfs instead and reboot the VM, so it can use all available memory.
 
* The RAM disk is not working as expected. I will setup a tmpfs instead and reboot the VM, so it can use all available memory.
 
|- style="background-color: silver;"
 
|- style="background-color: silver;"
| 25.08.2007&nbsp;16:00
+
| 25 August 2007&nbsp;16:00
| 25.08.2007&nbsp;17:00
+
| 25 August 2007&nbsp;17:00
 
|
 
|
 
* Setting up a RAM disk for the Reactos VM.
 
* Setting up a RAM disk for the Reactos VM.
 
|- style="background-color: silver;"
 
|- style="background-color: silver;"
| 28.06.2007&nbsp;15:00
+
| 28 June 2007&nbsp;15:00
| 28.06.2007&nbsp;18:00
+
| 28 June 2007&nbsp;18:00
 
|
 
|
 
* Fixed remaining problems with the RAID configuration. The new configuration reduced the build time from 9:00 to 7:40.
 
* Fixed remaining problems with the RAID configuration. The new configuration reduced the build time from 9:00 to 7:40.
 
|- style="background-color: silver;"
 
|- style="background-color: silver;"
| 27.06.2007&nbsp;21:30
+
| 27 June 2007&nbsp;21:30
| 28.06.2007&nbsp;15:00
+
| 28 June 2007&nbsp;15:00
 
|
 
|
 
* Installed a UPS and apcupsd.
 
* Installed a UPS and apcupsd.
Line 70: Line 107:
 
|}
 
|}
 
Legend:
 
Legend:
<div><span style="background-color: yellow">&nbsp;&nbsp;&nbsp;</span> scheduled downtime that has not occured yet</div>
+
<div><span style="background-color: yellow">&nbsp;&nbsp;&nbsp;</span> scheduled downtime</div>
 
<div><span style="background-color: silver">&nbsp;&nbsp;&nbsp;</span> scheduled downtime that has occured</div>
 
<div><span style="background-color: silver">&nbsp;&nbsp;&nbsp;</span> scheduled downtime that has occured</div>
 
<div><span style="background-color: red">&nbsp;&nbsp;&nbsp;</span> unscheduled downtime that is occuring right now</div>
 
<div><span style="background-color: red">&nbsp;&nbsp;&nbsp;</span> unscheduled downtime that is occuring right now</div>
Line 76: Line 113:
  
 
== Hardware ==
 
== Hardware ==
{| class="prettytable"
+
{| class="wikitable"
 
|-
 
|-
 
! Mainboard
 
! Mainboard
Line 85: Line 122:
 
|-
 
|-
 
! Memory
 
! Memory
| Kingston ValueRAM FB-DIMM 8 GB DDR2-667
+
| Kingston ValueRAM FB-DIMM 8&nbsp;GB DDR2-667
 
|-
 
|-
 
! Hard disks
 
! Hard disks
| 3x 305GiB, 1x 27GiB<br/>/dev/md0: RAID5 (3x 297GiB); /dev/md1: RAID0 (3x 1027MiB)
+
| plenty
 
|-
 
|-
 
! Host operating system
 
! Host operating system
| FC7
+
| Gentoo
|-
 
! VM operating system
 
| FC7
 
 
|}
 
|}
  
 
== History ==
 
== History ==
 +
* '''May 2008''': Relocation to Braunschweig, Germany; now using a 100 MBit Internet connection.
 
* '''May 2007''': We finally got a replacement mainboard from Alternate, not the one we sent in, but a very similar one, so the server is now back up. The object files are now placed on a RAID0 with three disks, improviving the build time from ~15 to ~10 minutes.
 
* '''May 2007''': We finally got a replacement mainboard from Alternate, not the one we sent in, but a very similar one, so the server is now back up. The object files are now placed on a RAID0 with three disks, improviving the build time from ~15 to ~10 minutes.
 
* '''March 2007''': As long as the new server doesn't work, the old server is back online.
 
* '''March 2007''': As long as the new server doesn't work, the old server is back online.
* '''February 2007''': Martin comes up with the idea to put plenty new SATA hard disks into the server, but the old mainboard has no SATA support, so the whole server needs to be replaced. The ReactOS team decides to co-fund the upgrade and instead of consumer level hardware, a full-blown Xeon QuadCore server is bought. Due to problems with the online shop, it took a few weeks to get the new server up and running - only to find out that the mainboard sometimes simply won't boot and has to be sent back for replacement.
+
* '''February 2007''': Martin comes up with the idea to put plenty new SATA hard disks into the server, but the old mainboard has no SATA support, so the whole server needs to be replaced. The ReactOS team decides to co-fund the upgrade and instead of consumer level hardware, a full-blown Xeon QuadCore server is bought. Due to problems with the online shop, it took a few weeks to get the new server up and running only to find out that the mainboard sometimes simply won't boot and has to be sent back for replacement.
* '''Long ago''': Christoph installs the buildbot on Martin's private server, then an old AMD 800MHz with 512 MB RAM. The RAM is later upgraded to 1024MB, the CPU is replaced with a Sempron 2500+.
+
* '''Long ago''': Christoph installs the buildbot on Martin's private server, then an old AMD 800&nbsp;MHz with 512&nbsp;MB RAM. The RAM is later upgraded to 1024&nbsp;MB, the CPU is replaced with a Sempron 2500+.
 +
 
 +
{{DEFAULTSORT:Buildslave Osnabruck}}
 +
[[Category:ReactOS Foundation]]
 +
[[Category:Community]]
 +
[[Category:Hardware]]
 +
 
 +
__NOTOC__

Latest revision as of 15:45, 15 May 2014

The Buildslave Osnabrück compiles ReactOS debug builds. Contrary to the name, it is now actually located in Braunschweig (hosted by the company I'm working for, IServ)... I should consider to rename it :)

Current Status

Running as Buildslave Lammhult.

Downtimes

Start date (UTC) End date (UTC) Reason
21 October 2011 2 November 2011
  • Relocation to Lammhult.
3 June 2008 4 June 2008
  • The buildslave daemon was running twice for unknown reasons; both instances were interfering with each other and spammed the BuildBot status page. I killed all running instances and restarted the daemon.
30 March 2008 11:20 31 March 2008 15:00
  • Relocation to Braunschweig.
6 December 2007 15:30 6 December 2007 16:30 18:30 19:30 20:00
  • Kernel update -> linux-2.6.24-gentoo, and a few minor other things. Will see whether I can fix that buildbot chaos my brother has made... EDIT: need more time...

EDIT: still need more time. The new kernel is running fine, I've even correctly configured grub for falling back to an older kernel when a kernel update should fail. Now I still gotta fix that buildbot setup. I'll try to leave it running most of the time, though.

EDIT: I'm done now. I've fixed the buildbot mess, it's installed via portage again now. Wasn't able to fix the non-working serial console in grub though :/

YET ANOTHER EDIT: fixed some permissions problems my brother had caused.

6 December 2007 10:25 6 December 2007 10:30
  • Kernel update -> linux-2.6.23-gentoo-r3
5 December 2007 21:00 5 December 2007 21:45
  • A server reboot caused the domain name resolver to malfunction; Gentoo overwrites the /etc/resolv.conf file on each reboot, and thereby removed my nameserver setting. I put the nameserver address into /etc/conf.d/net so Gentoo will now generate corrent resolv.conf files.
1 December 2007 20:10 1 December 2007 20:4022:30
  • Going to test if ReactOS will now build on 64 bit Gentoo.Oh my, we were watching a film and I forgot that I took the buildslave offline... it's now back online, running as 64bit :)
21 November 2007 15:20 21 November 2007 17:10
  • Oh my, moving the whole buildbot folder onto the RAM disk was a bad idea actually. We were repeatedly surprised after restarts that our script changes were missing (changed them on the RAM disk and forgot to copy them back to the hard disk), this time the uploadiso script was reverted, leading to corrupted ISO files on the download server. I moved the folder back on the hard disk for the sake of stability; obj, output and now ccache remain on the RAM disk. ccache is now installed on the debug server too, reducing the build time from ~5 minutes to ~2 minutes.
16 November 2007 18:10 16 November 2007 18:2519:20
  • Reboot. Oh my, the expected new hard disk didn't show up. Investigating, expect another reboot soon. Fixed it, took me almost a whole hour longer than expected. Looks like only 4 out of 6 SATA ports work atm. Enabling AHCI support in the BIOS should fix this, but Fedora doesn't seem to support this (kernel panic). Anyway, I'm going to install Gentoo sooner or later, than I will do this properly.
1 October 2007 ~15:00 1 October 2007 ~21:00
  • Repeated power failures led to a server shutdown, and I had forgotten to a) configure the initscript to rebuild the RAM disk after boot and b) copy some new scripts from the RAM disk back to the hard disk. Took me some time to notice the buildbot wasn't coming back online and then to fix these issues.
5 September 2007 07:55 5 September 2007 08:30
  • I'll try to move the whole buildbot onto the RAM disk, not only the object files. Only the buildbot will have ocassionally downtimes, the VM will stay online.
4 September 2007 21:05 4 September 2007 21:10
  • The RAM disk is not working as expected. I will setup a tmpfs instead and reboot the VM, so it can use all available memory.
25 August 2007 16:00 25 August 2007 17:00
  • Setting up a RAM disk for the Reactos VM.
28 June 2007 15:00 28 June 2007 18:00
  • Fixed remaining problems with the RAID configuration. The new configuration reduced the build time from 9:00 to 7:40.
27 June 2007 21:30 28 June 2007 15:00
  • Installed a UPS and apcupsd.
  • Upgraded the Reactos VM to FC7.
  • Configured a serial console for the Xen host to simplify future maintenance work.
  • Changed the RAID0 configuration of the Reactos VM.
  • Cleaned the fans.

Legend:

    scheduled downtime
    scheduled downtime that has occured
    unscheduled downtime that is occuring right now
    unscheduled downtime that has occured

Hardware

Mainboard Asus DSBF-DE
CPU 2x Intel Xeon 5310
Memory Kingston ValueRAM FB-DIMM 8 GB DDR2-667
Hard disks plenty
Host operating system Gentoo

History

  • May 2008: Relocation to Braunschweig, Germany; now using a 100 MBit Internet connection.
  • May 2007: We finally got a replacement mainboard from Alternate, not the one we sent in, but a very similar one, so the server is now back up. The object files are now placed on a RAID0 with three disks, improviving the build time from ~15 to ~10 minutes.
  • March 2007: As long as the new server doesn't work, the old server is back online.
  • February 2007: Martin comes up with the idea to put plenty new SATA hard disks into the server, but the old mainboard has no SATA support, so the whole server needs to be replaced. The ReactOS team decides to co-fund the upgrade and instead of consumer level hardware, a full-blown Xeon QuadCore server is bought. Due to problems with the online shop, it took a few weeks to get the new server up and running – only to find out that the mainboard sometimes simply won't boot and has to be sent back for replacement.
  • Long ago: Christoph installs the buildbot on Martin's private server, then an old AMD 800 MHz with 512 MB RAM. The RAM is later upgraded to 1024 MB, the CPU is replaced with a Sempron 2500+.