Periodic testing of ReactOS on real hardware.

Here you can discuss ReactOS related topics.

Moderator: Moderator Team

Post Reply
amber
Posts: 21
Joined: Fri Apr 19, 2013 7:39 pm

Periodic testing of ReactOS on real hardware.

Post by amber »

Hi there! When testing ReactOS in VBox, I recently encountered a virtual machine crash. This led me to think: how good is software virtualization today? Can the virtual machine be 100% trusted? Hence my question: Whether testing ReactOS on real oborudovanii and how often? I understand that debugging on real hardware can be a very difficult task. But trusting tests that run only in a virtual environment is reckless. Produced by ReactOS team testing on real hardware and how often? Thanks for understanding.

hbelusca
Developer
Posts: 1173
Joined: Sat Dec 26, 2009 10:36 pm
Location: Zagreb, Croatia

Re: Periodic testing of ReactOS on real hardware.

Post by hbelusca »

amber wrote:
Tue Dec 25, 2018 7:07 pm
Can the virtual machine be 100% trusted?
Not completely; you can find that your OS works 100% fine on VMs but is unable to boot on different real hardware configurations: this is what happens with ReactOS.
amber wrote:
Tue Dec 25, 2018 7:07 pm
Produced by ReactOS team testing on real hardware and how often?
We don't make periodic (and automatic) real-hardware testing; for that first we would have to fix the most obvious problems encountered there, namely, fixing the USB code (which is currently work-in-progress by Vadim Galyant alone) and certainly other things I don't have on top of my head, and then, we would have to use two or three mostly different kind of hardware that would "cover" the HW one could find out there.
So far we rely on you people for testing ReactOS on real HW until we can do it better...

(Disclaimer: This answer is only my own and not an official ReactOS team answer.)

jelabarre59
Posts: 1
Joined: Thu Jan 03, 2019 12:27 am

Re: Periodic testing of ReactOS on real hardware.

Post by jelabarre59 »

For that matter, I *have* a 32bit laptop I don't need for any other project (probably ever)which I would set up as a hardware test machine, if there were some test suite I could periodically run on it (something like an acceptance test or build validation). I asked elsewhere and got the impression the devs don't want to even bother testing on real hardware. It's a Dell Inspiron 6000, and I grabbed all the MSWin7 drivers for it by installing an eval version of MSWin7, pulled down the updates and the Dell drivers, then backed them up using DoubleDriver.

Maybe would have to go back to an XP install instead to get some that have better compatibility with ROS, but this requires some feedback on what is expected to work, and how they would like feedback.

middings
Posts: 1028
Joined: Tue May 07, 2013 9:18 pm
Location: California, USA

Re: Periodic testing of ReactOS on real hardware.

Post by middings »

jelabarre59 wrote:
Thu Jan 03, 2019 12:36 am
I ... got the impression the devs don't want to even bother testing on real hardware.
Running ReactOS on a virtual machine is convenient for the developers (devs). Virtual machines provide the devs fast design-code-build-test cycles. Creating a CD-ROM to load and install ReactOS on a PC introduces a long, distracting delay in the test step.

Also, the devs generally wish to make fixes and new features that will benefit a large number of possible ReactOS users. Most ReactOS failures on hardware are linked to driver software. Some of the API infrastructure for drivers is not yet implemented. Some drivers have bugs that the hardware maker never fixed.

Having said all that, the devs would like to see ReactOS run on more hardware. But they do not have a large variety of PCs to test ReactOS on. The closest the ReactOS devs have to 'reference hardware' are the Dell D531 laptop PCs (Turion X2 processor @ 2GHz, 2GiB RAM, 15.4-inch display) that many devs have. Devs interested in improving ReactOS's support for drivers require the assistance of testers who have the appropriate hardware, are available to build and test new ReactOS code, and provide the results with all the debugger traces and memory dumps that the developers require.

The best PCs to test ReactOS with are, in my opinion, PCs from market-leading manufacturers (such as Dell and HP/Compaq) that were sold in reasonably large numbers and could be purchased new with factory-installed copies of Microsoft Windows XP SP3 or Microsoft Server 2003. Copies of all necessary hardware driver software compatible with Windows XP or Server 2003 must still be available, preferably by download from the PC's original manufacturer's web site.
...if there were some test suite I could periodically run... (something like an acceptance test or build validation).
Such a test suite is a good idea. The best people to create and document such an test suite are the people who test ReactOS on hardware. Unfortunately my own computer that I use to test ReactOS (my "ROS rig") has been broken for quite some time. When I get it fixed or replace it with another PC that I can dedicate to testing ReactOS, I would like to participate with you and other hardware testers to make a test suite.
...this requires some feedback on... how they (the ReactOS developers) would like feedback.
The ReactOS developer mailing list or developer IRC channel are probably the best ways to explore that with the devs.

Ancient
Posts: 81
Joined: Tue Mar 27, 2018 11:32 pm

Re: Periodic testing of ReactOS on real hardware.

Post by Ancient »

Until a beta is out, it won't make much difference. Better to work on the beta as a VM only project, until near complete then start testing on real hardware. As it is, no real hardware will be 32 bit single core in a modern real machine. Implementation of multi core, HDMI media, real boot, and a ton of other necessary stuff (eg USB C) will delay finishing the 32 bit single core system. For alpha it may be best to assume VM only to avoid distraction. This is how bad Windows can be on real hardware - https://www.youtube.com/watch?v=M2LOMTp ... ture=share ... note half the performance vs Linux with 32 real cores. Bad performance even with 16 cores. Even using Linux subsystem (WSL) indicates Windows is unable to appropriately manage Ryzen. Windows is a well evolved product, and it has serious issues with real hardware. Stuff like this would be a tremendous distraction.

ThFabba
Developer
Posts: 291
Joined: Sun Jul 11, 2010 11:39 am

Re: Periodic testing of ReactOS on real hardware.

Post by ThFabba »

Well, to disagree with what everyone else is saying... I think testing on hardware is super useful, actually.
The reason why we don't do automated hardware testing is: it's relatively hard to set up.
The reason lots of hardware bugs don't get fixed is: we don't have the hardware to reproduce it and it's hard to diagnose remotely -- especially if the reporter is not very responsive or the bug report is not of sufficient quality.

With that said, I would highly encourage hardware tests, and here are the tests that I think make the most sense:
* Make sure that the OS installs correctly and your basic input devices work
* Try to install XP/2003 drivers for any hardware components -- in particular ethernet cards (should mostly work, and it's important to watch for regressions) and video drivers (lots of these don't work, but it's important to have good bugs filed for them in Jira
* Either using a special ISO with vgal's USB drivers, or once the new USB stack lands in master, check if your USB controller works correctly and if simple "driverless" devices like mice/keyboards/thumb drives work as expected

Some notes to make testing worthwhile, however:
* Use a machine that has a serial port (and make sure you have a way to receive the output -- usually a null modem cable plus a USB-serial adapter). Currently this is the only reliable way to debug ROS, and it will be much harder to get any bugs fixed if you can't get a debug log
* Make sure to file high-quality Jira reports. In particular, use the search function to avoid duplicates, always include debug logs and a textual description of the problem as well as screen shots/photos where appropriate, and always mention your hardware specs (including the log from 1st stage setup will help with this, as it enumerates PCI devices)
* Be available to work with developers to solve the problem. In particular you should be able to test patches -- so have a working build environment and a CD-RW to burn new images to.
* If the OS installation already fails, it's going to be hard to find a solution, so be prepared for a lot of trial and error. Don't try to boot from USB, that's not supported. If the BootCD doesn't work, see if the LiveCD is any different. In many cases it will be a problem with UniATA, so getting an UniATA debug log may be needed (there's a commented line in uniata's CMakeLists.txt)

User avatar
dizt3mp3r
Posts: 1606
Joined: Mon Jun 14, 2010 5:54 pm

Re: Periodic testing of ReactOS on real hardware.

Post by dizt3mp3r »

Personally, I think testing on real hardware is a waste of time. I am unsure as to whether ReactOS will ever run on real hardware by the time it is ready to reach beta, let alone live 1.0 as the the vast majority of current hardware at that point may have no usable drivers for ReactOS, we shall see but I am not pinning my hopes on it. My personal hope is for a stable windows-like o/s running extremely quickly on virtual hardware when multi (16,32+) core systems become cheap enough for general use. Then I will run Kubuntu as the stable supported host and have as many instances of Windows compatible ReactOS as I choose, each doing a single task until ReactOS is stable enough for multi-tasking in a real work environment.

I believe, no I think, or perhaps I intelligently suggest that this will be the future use of ReactOS well before ReactOS is ever run on real modern hardware. Old XP/2003 era hardware will still continue to be tested with ReactOS by hobbyists but it will be more of an academic exercise than a real-use set up becoming increasingly irrelevant to real-world users that want freedom from Microsoft and Windows application compatibility on modern and exotic hardware. If there exists a slim host with perfect virtualising then there is absolutely no need for real hardware compatibility.

In my real life work, I install/run/tune VMS systems on perfectly virtualised hardware, no-one in their right mind would want to run VMS on real obsolete hardware except as an academic exercise. I am employed solely to move applications away from obsolete hardware onto whichever hardware is current that the client chooses, that is the beauty of virtualisation, you are free from hardware constraints.

As with VMS on ancient hardware, the same will be with ReactOS by the time it is finally ready. IF you've been around the ReactOS forums for a long time you become aware of how much has been achieved by the devs, the trouble is it soon becomes apparent how much work is still to be done, there is a huge amount to do before ReactOS can even reach Beta. Real hardware support is such along way down the line that I am sorry to say I think you are wasting your time.
Skillset: VMS sysadmin 20 years, fault Tolerance, cluster, Vax, Alpha, ftSparc. DCL, QB45, VB6, NET, PHP, JS, CMS, Graphics, Project Manager, DOS, Windows admin from 1985. Quad Electronics. Classic cars & motorbikes. Artist watercolours. Historian.

PurpleGurl
Posts: 1788
Joined: Fri Aug 07, 2009 5:11 am
Location: USA

Re: Periodic testing of ReactOS on real hardware.

Post by PurpleGurl »

I don't see it as a waste of time....just need those with dev experience doing the tests or at least "riding shotgun" on the testing with a tester who has enough experience to relay useful information.

The compatibility problems are in the kernel, and "enabling" it by using a VM and hiding its broken behavior won't help.

justincase
Posts: 441
Joined: Sat Nov 15, 2008 4:13 pm

Re: Periodic testing of ReactOS on real hardware.

Post by justincase »

Honestly, I think what would probably help ReactOS the most would be to figure out either a better automated method of gathering bug reports, or a better way to teach people to make good bug reports.

If I remember correctly, there was some discussion a while back about making the BSOD display a QR code with debugging information. If such a QR code could contain the last (however far back is usually both useful & fits easily in a QR code) lines of a debug log and/or a backtrace, perhaps that could help more people make better bug reports, as then we could request reporters to scan the QR code with their phone instead of the 'Do you have a null modem cable? Can you get one? Nevermind, it looks like your PC doesn't have a serial port, come back later to see if we happen to have fixed your issue by chance.' threads I've seen happen a number of times here in the forums.
dizt3mp3r wrote:
Sat Jan 05, 2019 2:15 am
Personally, I think testing on real hardware is a waste of time.
You're entitled to your opinion, but as long as a user gives us a half-decent bug report when they experience an issue, and responds appropriately to any questions the devs have, any testing (any at all) is helpful, and therefore (as long as you also emphasize that ReactOS is not ready for daily use, and data loss will occur during real-HW testing {during installation, and also possibly as the result of bugs encountered during use}) should be encouraged.
hbelusca wrote:
Tue Dec 25, 2018 7:19 pm
So far we rely on you people for testing ReactOS on real HW
ThFabba wrote:
Fri Jan 04, 2019 11:20 pm
I think testing on hardware is super useful
Sounds like you two are in agreement at least. And I'm glad to hear you guys voice this sentiment. Hopefully the fact that some of the developers feel positively about users testing on real hardware will encourage some users to run (and report on) real hardware tests.
PurpleGurl wrote:
Sat Jan 05, 2019 6:37 am
The compatibility problems are in the kernel, and "enabling" it by using a VM and hiding its broken behavior won't help.
Exactly. And even if the issues that cause most of the major problems on real hardware are caused by the HAL (which may or may not be true, IDK), testing on many different (real & virtual) hardware configurations can hep us find severe bugs (anywhere in the code) that might not appear except in specific, unusual cases otherwise.

As a hypothetical example of such a bug, imagine a race condition, or an unprotected buffer, that would normally produce an error when triggered, but if certain parts of memory are manipulated correctly, could instead allow privilege escalation, and can be triggered if you're aware of it by supplying an unusually large amount of data to some internal ReactOS function (such bugs have been known to exist, and later be patched, in Windows), if this bug is more likely to occur accidentally when running under a VM, then the devs will likely notice it during their tests and fix it, however if it is more likely to be triggered accidentally while running on real hardware (especially if it's not likely to happen accidentally unless running on some specific hardware), and nobody does real hardware testing, then this bug would very likely remain in the code long after ReactOS is thought to be stable, and then could be discovered and exploited by some hacker to do, basically whatever they want with a users computer, meanwhile if we test ReactOS on as many (real and virtual) hardware configurations as we can during the earlier testing phases, this kind of bug is much more likely to be remedied before such an exploit gets 'into the wild' and reaches end users.
I reserve the right to ignore any portion of any post if I deem it not constructive or likely to cause the discussion to degenerate.

User avatar
dizt3mp3r
Posts: 1606
Joined: Mon Jun 14, 2010 5:54 pm

Re: Periodic testing of ReactOS on real hardware.

Post by dizt3mp3r »

Bear in mind that my opinion is only engendered by the speed of development of ReactOS and if ReactOS was more advanced I'd be doing the same on the few bits of compatible hardware I have here.

Consider this, consider the likely response to users finding an almost-beta ReactOS in three/five years time - because this is a realistic time frame. IF I was desperate to run that almost beta o/s (due to MS' next unholy feck-up) then I can imagine myself (and many others like me) resorting to running ReactOS in a VM - if that is a good stable combination - in the manner previously described rather than going out and looking for ancient and unreliable hardware and finding that it cannot access the latest technology (SSDs, multi cores, what ever is new in 2023 &c &c) and is not compatible with the tech. that is available off-the-shelf.

I think that hardware testing is a great thing to do but testing on compatible 2000-2010 kit in 2023 is going to be harder and harder to obtain and less and less realistic an option/target for most everyday users by the time ReactOS is ready. I'm not sure it makes any sense in the context of eventual target usage.

The likely target audience for ReactOS is going to be for key applications, multiple instances and I'd suggest 95-99% virtualised. the only variance from this where ReactOS could receive a natural home is on small, cheap ARM devices but for the moment that is modern, exotic hardware for ReactOS. That is perhaps where we should be focussing our hardware-testing efforts in the future.
Skillset: VMS sysadmin 20 years, fault Tolerance, cluster, Vax, Alpha, ftSparc. DCL, QB45, VB6, NET, PHP, JS, CMS, Graphics, Project Manager, DOS, Windows admin from 1985. Quad Electronics. Classic cars & motorbikes. Artist watercolours. Historian.

ThFabba
Developer
Posts: 291
Joined: Sun Jul 11, 2010 11:39 am

Re: Periodic testing of ReactOS on real hardware.

Post by ThFabba »

I think people tend to overestimate the architectural differences between older and newer machines.
There are three major changes: AHCI, UEFI and xHCI.
* I believe we've got a handle on AHCI: it's generally supported by UniATA, just needs some bug fixes (see: high-quality bug reports). We also have the storahci driver that could end up being more reliable (and follows a more modern architecture).
* As far as I am aware, most hardware today still fully supports booting in BIOS mode, so UEFI shouldn't be a problem yet. There's also work in progress on a compatible loader.
* xHCI is going to be important for USB boot, and of course input devices on modern hardware, but mostly doesn't affect whether ROS can be installed. We also have work in progress on a driver to support it.

I know there are several other important aspects we need to support -- x64 and SMP in particular -- but those are relevant for virtual machines as well, so I don't consider them super relevant to this discussion.

So, the point I'm trying to make here is: almost every fix to support old hardware will also help ROS run on newer hardware. So I really don't see this as wasted effort.

User avatar
dizt3mp3r
Posts: 1606
Joined: Mon Jun 14, 2010 5:54 pm

Re: Periodic testing of ReactOS on real hardware.

Post by dizt3mp3r »

ThFabba wrote:
Sat Jan 05, 2019 1:17 pm
So, the point I'm trying to make here is: almost every fix to support old hardware will also help ROS run on newer hardware. So I really don't see this as wasted effort.
That's very positive and I like that but within the timescale we are talking about there will will almost certainly be something new from Microdolts (sic) that will make users want to jump ship from Windows and onto the ReactOS life raft. If they attempt to install on their current hardware we will warn them not to, it won't install or it will crash at some important point. On a VM it is much more likely to just work.

All those users will do what I propose to do, you can boot windows and then run ReactOS, you can boot Linux and then run ReactOS. So in all likelihood this will be the majority use-case for ReactOS until it reaches 1.0. From that point onward virtualisation will still remain the largest target, I have no doubt. I suggest usage will explode once ReactOS becomes generally stable and usable in a VM.

My worry is that by testing with real hardware the public have high expectations that it will also be usable on real hardware. ReactOS is always raising people's expectations only to have them crushed shortly after.

I will modify my point with my normal approach, lower your expectations of what testing with real hardware is likely to achieve for you. It is merely contributing to ReactOS to identify issues and in that context it useful but in the context of getting it to run in a stable fashion on your hardware - lower your expectations!
Skillset: VMS sysadmin 20 years, fault Tolerance, cluster, Vax, Alpha, ftSparc. DCL, QB45, VB6, NET, PHP, JS, CMS, Graphics, Project Manager, DOS, Windows admin from 1985. Quad Electronics. Classic cars & motorbikes. Artist watercolours. Historian.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 3 guests