Page 1 of 1

Tools for automated OS testing on real hardware?

Posted: Sat Sep 17, 2011 10:36 pm
by SomeGuy
I was just tossing around some thoughts about how to automatically test ReactOS on real hardware.

It is probably still a bit early for it to really be needed, but ReactOS will eventually need some kind of automated system testing that individual users can run on their own hardware and report results back to ReactOS. I'm talking about kernel and driver related tests.

This way if enough people were to do it, devs could more quickly know if a change causes problems with certain chipsets, storage controllers, third party drivers, or other specific configurations (and there are plenty of configurations out there). If there were frequent enough testes, there would be less need for messy regression testing on real hardware.

There are, however, some interesting problems here. :) How do you automatically download and write a disk image to a real hard drive if there is no OS present? How do you tell it to automatically boot/reboot when it is ready? How do you send debug logs back if the network card driver doesn't work? How do you automatically capture logs from a com port without any other computer? How do you get any meaningful results back if the OS doesn't even load the kernel at all?!

The ReactOS TestBot has it easy because VirtualBox has so many scriptable features.

Is there any kind of system or hardware add on that can facilitate this kind of testing?

I am aware of some devices that are used for remote monitoring servers and can manage a machine even when no OS is running. Can something like those be adapted for scriptable fully automated OS testing?

Re: Tools for automated OS testing on real hardware?

Posted: Sun Sep 18, 2011 12:30 am
by Heis Spiter
Hi,

we (the ReactOS sysadmins) are already thinking about such solution. It would be based on a computer having either AMT or IPMI. And then the OS would be installed through PXE.
The point of sending debug has not been discussed yet. But it would likely go through serial to another computer.

Regards,
Pierre

Re: Tools for automated OS testing on real hardware?

Posted: Sun Sep 18, 2011 8:15 am
by Murmur
Can ROS install though PXE or capture an image yet?

Re: Tools for automated OS testing on real hardware?

Posted: Sun Sep 18, 2011 10:30 am
by Pisarz
Murmur,
First, proof of concept:
http://www.youtube.com/watch?v=mm9VIawx__Y

And there is some info on wiki already:
http://www.reactos.org/wiki/Building_PXE-ready_ReactOS

Re: Tools for automated OS testing on real hardware?

Posted: Sun Sep 18, 2011 9:22 pm
by Bblaauw
That video looks like PXELinux --> Memdisk --> HDDimage --> ReactOS
That implies the following should be possible as well: Syslinux -> Memdisk --> HDDimage --> ReactOS
However the documentation doesn't mention if any FreeLDR modifications are needed. Likely some "ram()\reactos" statement instead of "multi()disk()boot()\reactos"

As for the HDDimage, I'd guess Bochs (or QEMU?) was used to install ReactOS into this image file.

A first step for testing ReactOS compatibility with real hardware (or reverse, if you like) would be:
syslinux -> memdisk -> LiveCD.ISO