Tools for automated OS testing on real hardware?
Posted: Sat Sep 17, 2011 10:36 pm
I was just tossing around some thoughts about how to automatically test ReactOS on real hardware.
It is probably still a bit early for it to really be needed, but ReactOS will eventually need some kind of automated system testing that individual users can run on their own hardware and report results back to ReactOS. I'm talking about kernel and driver related tests.
This way if enough people were to do it, devs could more quickly know if a change causes problems with certain chipsets, storage controllers, third party drivers, or other specific configurations (and there are plenty of configurations out there). If there were frequent enough testes, there would be less need for messy regression testing on real hardware.
There are, however, some interesting problems here. How do you automatically download and write a disk image to a real hard drive if there is no OS present? How do you tell it to automatically boot/reboot when it is ready? How do you send debug logs back if the network card driver doesn't work? How do you automatically capture logs from a com port without any other computer? How do you get any meaningful results back if the OS doesn't even load the kernel at all?!
The ReactOS TestBot has it easy because VirtualBox has so many scriptable features.
Is there any kind of system or hardware add on that can facilitate this kind of testing?
I am aware of some devices that are used for remote monitoring servers and can manage a machine even when no OS is running. Can something like those be adapted for scriptable fully automated OS testing?
It is probably still a bit early for it to really be needed, but ReactOS will eventually need some kind of automated system testing that individual users can run on their own hardware and report results back to ReactOS. I'm talking about kernel and driver related tests.
This way if enough people were to do it, devs could more quickly know if a change causes problems with certain chipsets, storage controllers, third party drivers, or other specific configurations (and there are plenty of configurations out there). If there were frequent enough testes, there would be less need for messy regression testing on real hardware.
There are, however, some interesting problems here. How do you automatically download and write a disk image to a real hard drive if there is no OS present? How do you tell it to automatically boot/reboot when it is ready? How do you send debug logs back if the network card driver doesn't work? How do you automatically capture logs from a com port without any other computer? How do you get any meaningful results back if the OS doesn't even load the kernel at all?!
The ReactOS TestBot has it easy because VirtualBox has so many scriptable features.
Is there any kind of system or hardware add on that can facilitate this kind of testing?
I am aware of some devices that are used for remote monitoring servers and can manage a machine even when no OS is running. Can something like those be adapted for scriptable fully automated OS testing?