Debugging

This page describes different methods of debugging ReactOS and the steps necessary to debug ReactOS.

Introduction

To be able to help ReactOS development, whether this be participating in the development of the source code or taking part in crucial testing, you are going to need knowledge of how to generate useful debug logs.

Useful debug logs are essential pieces of information which the developer needs to quickly pinpoint and identify exactly what the operating system is doing. Many people know how to get default debug output from the operating system, but this is generally not particularly useful for locating problems, esspecially bugs.

This article aims to give users knowledge not only on how to generate a debug log, but on how to generate a useful debug log which can be used directly to asses what the operating system is doing.

Available debugging methods

There are various methods to debug ReactOS, some require more knowledge than others. These are listed below.

Debugging through text messages

This is the easiest method for receiving debug information from ReactOS.

Serial Port

The serial port is the most common method used for receiving debug messages from ReactOS. The method used for receiving data from the serial port depends on whether you run ReactOS in a virtual machine or on a real computer.

QEMU Virtual Machine: Redirect to a file

If you're using the QEMU virtual machine and want to redirect the serial output to a file, add the following to your QEMU command line:

-serial file:FILENAME

Replace FILENAME with the name of a file, into which the debug output shall be written.

QEMU Virtual Machine: Redirect to the console

The method above redirects the output to a file. If you want to redirect it to the console, there are two different methods. They depend on the operating system you run QEMU under.

• Under Windows, add the following to your QEMU command line:

  -serial file:CON

• Under a Unix-based operating system, add the following to your QEMU command line:

  -serial stdio

VMware Virtual Machine: Redirect to a file

If you're using a VMware product and want to redirect the serial output to a file, click the Edit virtual machine settings link.

If your virtual machine does not yet have a virtual serial port, add one using the Add button. Then select it and in the Connection box, choose the option Use output file and enter the name of the file, into which the debug output shall be written.

VMware Virtual Machine: Redirect using a named pipe

You can use the same methode as with VirtualBox, below.

VirtualBox Virtual Machine: Redirect using a named pipe on a Win32 Host

You need to download the VmwareGateway application. Start it with /r option to make it run as a service. Next you need to start the service, using SC command:

sc start vmwaregateway

Sort out any firewall popup if applicable. Finally, use your favourite telnet client to connect to localhost on port no. 567.

To configure your VM, set it to Host Pipe with the following pipe adress:

\\.\pipe\vmwaredebug

Debug output should appear in your telnet client.

VirtualBox Virtual Machine: Redirect to a Console on a Linux Host

You have to use some application like the Win32 Hyperterminal, such as Minicom. Firstly install it

sudo apt-get install minicom

Then you will have to configure a pipe. Type Ctrl-A, then O. Select Configure serial ports, and in the device type something like

unix#pipe_path

.

pipe_path is a file which will be used as pipe between the VM and minicom. For example, in my pc i use /tmp/vboxlog

Please note you must type pipe_path exactly on both minicom and VirtualBox

In VirtualBox, create a new Virtual Machine, then go to the Serial Ports configuration. Select Host pipe, type the pipe_path and select Create Pipe.

When debugging, make sure to run VirtualBox before minicom

VirtualBox Virtual Machine: Redirect to a File on a Linux Host

When running minicom, type Ctrl-A then L. Type the name of the capture file.

Real computer: Physical serial cable

You will need a physical serial cable if you want to receive debug messages from a real computer through the serial port. This method also requires two computers (one on which you test ReactOS and another one for receiving the debug messages).

The cable needed for this debugging method is a Null-Modem serial cable. You should find it in many computer shops for less than 10 dollars. If you don't have one ready, you can also build one like this:

DTE1_______________________________________________DTE 2

9pol 25pol (female)_________________________________25pol 9pol (female)
5    7  ---GND---------------------GND-------  7   5

2    3  ---RxD--------. ,----------RxD-------  3   2
                       X
3    2  ---TxD--------' `----------TxD-------  2   3

7    4  ---RTS--------. ,----------RTS-------  4   7
                       X
8    5  ---CTS--------' `----------CTS-------  5   8

4   20  ---DTR--------. ,----------DTR------- 20   4
                       X
6    6  ---DSR--o-----' `-------o--DSR-------  6   6
                |               |
1    8  ---DCD--'               `--DCD-------  8   1

Connect the cable to the first serial port of both computers.

Then use a Terminal application like PuTTY or Windows HyperTerminal on the computer for receiving the debug messages. Set it up to listen to the first serial port (COM1).

After that, boot ReactOS on the test computer and you should receive debug messages.

Debug text output to file

<add detail>

Debug text output to screen

<add detail>

KDBG

kdbg

<add detail>

GDB

Needed Items:

• ReactOS Build Environment
• Qemu
• gdb

The ReactOS build environment now includes gdb by default. In order to use it you must create config.rbuild from config.template.rbuild and change it so that:

<property name="KDBG" value="0" />
<property name="DBG" value="1" />
<property name="GDB" value="1" />

After you recompile(specially, kdcom and ntoskrnl), start Qemu as you normally would, but add the following command line parameters:

-s -S

This is done so that Qemu starts in the STOPPED state, and allow you to connect using gdb. Now its time to get GDB of the ground

• (Assuming you are in the rosbe command line), enter “gdb” to start gdb.
• Enter “file ./output-i386/ntoskrnl/ntoskrnl.exe” to tell gdb where to load information about the kernel.
• Enter “set disassembly-flavor intel” for good measure.
• Enter “target remote localhost:1234” to connect gdb to what we are debugging.
• Enter “c” (for “continue“) to have gdb instruct Qemu to start/continue execution of the emulation.
• To manually pause execution, make sure your gdb window has focus and simply enter <CTRL>+<C>

Knowing what debug output to generate

<add detail>

Turning on debugging in key modules

<add detail>

Turning on at compile time

<add detail>

Turning on at runtime

<add detail>

Forcing a bugcheck

Bug checks (or system crashes as they are also known) occur when the operating system can no longer operate safely and to avoid corrupting data, it halts operation. This will normally throw up a Blue Screen Of Death, but if you have the kernel debugger activated, it will drop you into a gdb prompt giving you access to explore the system.

There are two reasons for forcing a bugchecks, each reason employing a different method:

Dynamic

If you want to halt the system for any given reason and break into the the kernel debugger, you can force a bugcheck from the keyboard.

If you are not using a DBG build, you first need to add the following value in the registry:

Path:    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt
Keyname: CrashOnCtrlScroll
Type:    REG_DWORD
Value:   1

Then, when ReactOS is running, hold down the right CTRL key, and press SCROLL LOCK key twice.

Static

This is useful when you want to halt the operating system when it hits a particular area of code. This is esspecially useful as you can get an immediate back trace to see where the code flow came before the bugcheck was forced.

For people who are seeing a particular bug, and who have the ability to follow the source code leading up to this, a forced bugcheck can be interted into the code.

This is done in kernel mode by calling the KeBugCheck or KeBugCheckEx, but it is more easily forced by adding: ASSERT(FALSE); into the code.

Again, if you have the kernel debugger activated, you will be dropped into a gdb session whereby you can issue the 'bt' command to get a back trace of the code flow previous to this bugcheck. This is esspecially useful to developers as it helps to pinpoint the buildup to the bugcheck and this can be followed in the source to gain a better understanding of what is happening.

Generating a backtrace

Before a backtrace can be initiated, you will first need to have broken into the kernel debugger. See the section forcing a bugcheck for directions on how to do this.

Once you are in the kernel debugger, the 'kdb:>' symbol will appear in your debug output. This is a prompt to recieve commands for the debugger.

If you are using a virtual machine, ensure that the VM still has keyboard focus as this is the machine in which kdb is running.

Enter 'bt' and you should see the letters appear in your debug output. If so, hit <enter> and you should see something similar to the following:

(drivers\filesystems\vfat\rw.c:809) <\ReactOS\system32\kernel32.dll>
Entered debugger on embedded INT3 at 0x0008:0x800935f2.
kdb:> bt
Eip:
<ntoskrnl.exe:935f3 (lib\rtl\i386\debug_asm.S:31 (DbgBreakPoint@0))>
Frames:
<vfatfs.sys:97de (drivers/filesystems/vfat/misc.c:111 (VfatDispatchRequest))>
<vfatfs.sys:9b25 (drivers/filesystems/vfat/misc.c:167 (VfatBuildRequest))>
<ntoskrnl.exe:3ab23 (ntoskrnl/io/iomgr/irp.c:1088 (IofCallDriver))>
<ntoskrnl.exe:36206 (ntoskrnl/io/iomgr/iofunc.c:686 (IoSynchronousPageWrite))>
<ntoskrnl.exe:59daa (ntoskrnl/mm/section.c:6330 (MmspWriteDataSectionPages))>
<ntoskrnl.exe:244c6 (ntoskrnl/ex/work.c:162 (ExpWorkerThreadEntryPoint))>
<ntoskrnl.exe:70e90 (ntoskrnl/ps/thread.c:134 (PspSystemThreadStartup))>
<ntoskrnl.exe:7b142 (ntoskrnl\ke\i386\ctxswitch.S:258 (KiThreadStartup@156))>
kdb:> 

Examining this in more detail, we can see that the bugcheck occured via the INT3 operation which then dropped us into kdb. The next line shows us Eip which is the instruction pointer, and this points to the last address before the system halted.

Following on from that are the frames. This is the important part of generating our backtrace and it contains all the function addresses in the buildup to the bugcheck. This is the crutial information developers need to understand the codeflow before the bugcheck.

Translating Addresses

Occasionally, there will come a time when you will need to manually translate addresses. When kdbg is not enabled and a bugcheck occurs, you will be presented with a stack trace similar to the following:

(subsystems\win32\csrss\win32csr\conio.c:1101) Console_Api Ctrl-C 

*** Fatal System Error: 0x00000001
                       (0x80079279,0x00000000,0x0000FFFF,0x00000000) 

<\SystemRoot\System32\NTOSKRNL.EXE: 29bb>
<\SystemRoot\System32\HAL.DLL: 4749>
<\SystemRoot\System32\NTOSKRNL.EXE: 54cb4>
<\SystemRoot\System32\NTOSKRNL.EXE: 582bf>
<\SystemRoot\System32\NTOSKRNL.EXE: 583fd>
<\SystemRoot\System32\NTOSKRNL.EXE: 89956>
<\SystemRoot\system32\drivers\videoprt.sys: 2417>
<\SystemRoot\system32\drivers\vbemp.sys: 17f5>
<\SystemRoot\system32\drivers\vbemp.sys: 19cf>
<\SystemRoot\system32\drivers\videoprt.sys: 1c48>
<\SystemRoot\System32\NTOSKRNL.EXE: 34c17>
<\SystemRoot\System32\NTOSKRNL.EXE: 21e0>
<\SystemRoot\System32\NTOSKRNL.EXE: 2908>
<\SystemRoot\System32\NTOSKRNL.EXE: 29bb>
<\SystemRoot\System32\NTOSKRNL.EXE: 85fa8>

As you can see, this is largely the same as the information presented when issuing a 'bt' command in kdbg. The problem here however, is that only the addresses are given. As these addresses are different for everyones builds, this information is useless for anyone trying to follow what events occured in the lead up towards to bugcheck.

The solution here is to translate the addresses into human readable function names. This is done via a tool named 'raddr2line' which is a modified version of the unix tool 'addr2line'. This tool will translate the addresses given to it into file names and line numbers. It does this by using debug information in the executable files to accociate the address with this human friendly info and outputs it into the console. This information can be pasted into the above debug log alongside the addresses providing the developers with a detailed stack trace.

radr2line is included in the Reactos Build Environment. It is invoked in the following way :

raddr2line <file> <address>

So taking the bottom address in the above stack trace :

C:\Users\Ged\MyFiles\ReactOS\clean_source>raddr2line ntoskrnl.exe 85fa8

C:\Users\Ged\MyFiles\ReactOS\clean_source\output-i386\ntoskrnl\ntoskrnl.exe
obj-i386\ntoskrnl\ex\zw.S:253 (ZwClearEvent)

we can see here that the address translation for 0x85fa8 is line 253 in file ntoskrnl\ex\zw.S (this will differ if you try it on your build)

This information can now be added into the above stack trace as follows :

<\SystemRoot\System32\NTOSKRNL.EXE: 29bb>    <enter next one here>
<\SystemRoot\System32\NTOSKRNL.EXE: 85fa8>   obj-i386\ntoskrnl\ex\zw.S:253 (ZwClearEvent)