Overview: ReactOS testing in a VM

Got a ReactOS tutorial to share? Drop it in here

Moderator: Moderator Team

Post Reply
SomeGuy
Posts: 586
Joined: Mon Nov 29, 2004 9:48 am
Location: Marietta, GA

Overview: ReactOS testing in a VM

Post by SomeGuy »

The ReactOS project can use your help! All you have to do is periodically test an application or two and let us know when something breaks.

This is a high-level overview of what to test, how to test it, and how to let the ReactOS team know when something goes wrong.

Supported environments for running ReactOS are VirtualBox, VMWare, and QEMU. (Other environments are a whole different tutorial)

What to test:
* ReactOS itself - ReactOS has more functionality than one user will likley touch. As such things can break and may go unnoticed. There is much that is unimiplimented, so make sure what you test worked before.
* GoldenApps- These are specific versions of specific applications that have been chosen by the developers in order to test certian parts of ReactOS, and at one time or another have run almost flawlessly. Regressions in these applications are treated with priority. http://www.reactos.org/wiki/Testing_Central#GOLDENAPPS
* Open source Win32 software - This is the best kind of software to test against ReactOS. When something goes wrong in an open source program, it is easy for developers to inspect the application's source code to find out what it is trying to do.
* Downloadable closed source Win32 software - This kind of software can be helpful for exposing some kinds of bugs. However, debugging the application can be more difficult or impossible.
* Restricted commercial software - In general it is not useful to test software if others can not freely reproduce your results. Be prepared to do debugging yourself.
* Non-Win32 software - At present it is not generally helpful to test software that requires a runtime such as Java, Dot Net, Python, Etc, unless it is a test of the runtime itself.

When is a good time to test?
Select any period of time (weekley, monthly, yearly) you are comfortable with. The more often you test, the fewer builds you may have to go back through when something breaks.
There is usually a "call for testing" prior to an official release.
Testing a finalized official release is great for show, but use an SVN Debug build for reporting bugs.
Builds are more likely to be broken and untestable immediately after an official release.

How to test:
Download a current SVN BootCD debug (-dbg or -windbg) build, and install it in to a virtual drive image. Note that release builds do not contain the needed debugging information to create a crash report.

Typically you should test a single application at a time, starting from a clean install each time.

Hint: It can be quite useful to have a secondary drive in your VM to hold applications and data that does not get erased each time ReactOS is re-installed. There are a variety of ways to do this and tools to manipulate them.

Did it work? Great!

You may share your successes (or failures) using the ReactOS Compatibility Database. A detailed report of what you tested can be a great help to site visitors and other testers. Testers can also benefit by seeing what application features you tested, that they might not think to test.

http://www.reactos.org/compat/

Do NOT use this to report bugs. ReactOS developers do NOT read the Compatibility Database.

Found a bug? Here is what to do:

The complete, official rules for filing bug reports are here:
http://www.reactos.org/wiki/File_Bugs

And for all gory details on debugging see:
http://www.reactos.org/wiki/Debugging

You should review these before submitting a bug, but the following provides an overview. If you have any questions, feel free to discuss them on the forum.

Getting a crash log:

If a ReactOS crash / BSOD occurs, provide a crash log.

ReactOS has an interactive kernel debugger that is used to provide technical details about the crash.
Here is a simple way to get a typical crash log:

* In VirtualBox or VMWare, set your ReactOS VM's virtual COM1 to output to a file.
* Start ReactOS in Debug mode.
* Run your application to the point where you would get a BSOD.
* Instead of seeing a BSOD, ReactOS will appear to "freeze".
* At this point it is waiting for input at the interactive debugger but you will not see the prompts.
* In the ReactOS VM window, type "bt" and hit enter several times. (some backtraces are long and pause at each "screen")
* Close the ReactOS VM.
* Open the resulting text file and confirm it has the log and backtrace.

In some cases you may need to preform more complicated interactions that require a terminal program. Consult the Wiki page for your VM for details.

Regression testing:
If the bug you encountered is something that previously worked, then you should attempt to determine specifically which build revision was the first to exhibit the bug. This is very helpful as it is a simple but tedious and time consuming thing to do.

1: Start by noting the revision number of the current build you tested. (And double check it is still the latest and the problem persists)
2: Then locate an older build where this worked. Install, and confirm it worked.
3: Locate a build half way between the last one that the bug exists in and the last one that it doesn't.
4: Install, and note if the bug exists or not in this revision.
5: Go back to step 3 until it is narrowed down to a single revision.

Reporting a bug:
First, do your best to search through existing bugs in the Bugzilla database to ensure that your issue is not already reported.

What to include in a bug report:
The most important thing to include is sufficient information for someone else to replicate the bug.

*The specific ReactOS revision and build you tested.
*The environment (VirtualBox/VMWare/QEMU) including the version and any custom settings.
*The exact name and version of the tested application.
*If the application is not well known, provide a link to where others can obtain the application.
*A description of the actions taken to consistantly encounter the problem.
*A detailed description of the problem you are encountering.
*The problem should be very specific.
*If this is something that worked in a previous build, then specify which build first exhibited the problem.
*If the application is creating a ReactOS crash (BSOD), then *attach* your log with a backtrace. DO NOT COPY/PASTE THE ENTIRE LOG IN TO THE BODY OF THE BUG REPORT!
*To aid future searches you MAY copy and paste the last several lines before the crash in to the body of the bug report if they seem relevant.

What to expect:

Don't be disappointed if your bug sits around for months or even years with no activity. There are only so many developers with only so much time. Your bug report can still be valuable down the road - *especially* if you have narrowed down a specific build revision where something has broken.

Don't be disappointed if your bug is marked as a duplicate. A root cause of a bug can manifest itself in many different ways. More than likely you have still provided a unique, and therefore valuable test case.

Periodically retest to see if the bug is fixed. Often general rewrites and improvements will fix bugs without any knowledge of your specific bug report.

Devs or other ReactOS members may ask you to perform additional testing or provide additional information related to your bug. Some requests might seem overwhelming, please feel free to ask for explanations or assistance.
Last edited by SomeGuy on Thu Jul 12, 2012 4:50 pm, edited 1 time in total.
milon
Posts: 969
Joined: Sat Sep 05, 2009 9:26 pm

Re: Overview: ReactOS testing in a VM

Post by milon »

Bravo, SomeGuy! This is a great overview!
SomeGuy wrote:Typically you should test a single application at a time, starting from a clean install each time.

Hint: It can be quite useful to have a secondary drive in your VM to hold applications and data that does not get erased each time ReactOS is re-installed. There are a variety of ways to do this and tools to manipulate them.
I find it's very simple to just make a fresh install of ROS and then clone it for each test. That way you get a clean environment with no install delays.
DeadDude
Posts: 68
Joined: Thu Dec 28, 2006 7:53 pm

Re: Overview: ReactOS testing in a VM

Post by DeadDude »

Greatly appreciated!

I do have a few questions though...
Getting a crash log:

If a ReactOS crash / BSOD occurs, provide a crash log.

ReactOS has an interactive kernel debugger that is used to provide technical details about the crash.
Here is a simple way to get a typical crash log:

* In VirtualBox or VMWare, set your ReactOS VM's virtual COM1 to output to a file.
* Start ReactOS in Debug mode.
* Run your application to the point where you would get a BSOD.
* Instead of seeing a BSOD, ReactOS will appear to "freeze".
* At this point it is waiting for input at the interactive debugger but you will not see the prompts.

* In the ReactOS VM window, type "bt" and hit enter several times. (some backtraces are long and pause at each "screen")
* Close the ReactOS VM.
* Open the resulting text file and confirm it has the log and backtrace.
So, whenever ReactOS 'BSOD's, and the VM software did NOT say the CPU was disabled by the guest... I can hit bt and 'enter' repeatedly and get more informative log files?

Next question... is it possible to invoke a similar effect while the GUI is still responsive? ie. I can't find the path the VM SVGA Driver wizard is searching; is there a more informative debug log option than simple 'debug' ? What about RosDbg? What are the other boot menu options for? Might not be the right place to answer that question, but I don't know....
SomeGuy
Posts: 586
Joined: Mon Nov 29, 2004 9:48 am
Location: Marietta, GA

Re: Overview: ReactOS testing in a VM

Post by SomeGuy »

DeadDude wrote:Next question... is it possible to invoke a similar effect while the GUI is still responsive? ie. I can't find the path the VM SVGA Driver wizard is searching; is there a more informative debug log option than simple 'debug' ? What about RosDbg? What are the other boot menu options for? Might not be the right place to answer that question, but I don't know....
There are ways to wring more information out of the debug log such as the "debugchannel" command or recompiling modules with additional debug information. Those are beyond the scope of this tutorial, but see the wiki pages: http://www.reactos.org/wiki/Debugging http://www.reactos.org/wiki/Kdbg
DeadDude
Posts: 68
Joined: Thu Dec 28, 2006 7:53 pm

Re: Overview: ReactOS testing in a VM

Post by DeadDude »

Thank you!
vicmarcal
Test Team
Posts: 2733
Joined: Mon Jul 07, 2008 12:35 pm

Re: Overview: ReactOS testing in a VM

Post by vicmarcal »

SomeGuy: +1 Cool woork!!Really encouraging :)
Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests