This page describes different methods of debugging ReactOS and the steps necessary to debug ReactOS.
- 1 Introduction
- 2 Available debugging methods
- 2.1 Debugging through text messages
- 2.1.1 Serial Port
- 2.1.2 Debug text output to file
- 2.1.3 Debug text output to screen
- 2.1.4 Changing the BAUD rate
- 2.1.5 Changing the serial port address
- 2.2 KDBG
- 2.3 GDB
- 2.4 WinDbg
- 2.1 Debugging through text messages
- 3 Generating even more output
- 3.1 Turning on verbosity at compile time
- 3.2 Turning on verbosity at runtime
- 3.3 Breaking into the built-in kernel debugger
- 3.4 Generating a backtrace
- 3.5 Translating Addresses
- 3.6 Enabling Kernel Tracing
- 3.7 Debug Page Heap (DPH)
- 3.8 How to read/debug BugCheck messages
To be able to help ReactOS development, whether this be participating in the development of the source code or taking part in crucial testing, you are going to need knowledge of how to generate useful debug logs.
Useful debug logs are essential pieces of information which the developer needs to quickly pinpoint and identify exactly what the operating system is doing. Many people know how to get default debug output from the operating system, but this is generally not particularly useful for locating problems, especially bugs.
This article aims to give users knowledge not only on how to generate a debug log, but on how to generate a useful debug log which can be used directly to assess what the operating system is doing.
Available debugging methods
There are various methods to debug ReactOS, some require more knowledge than others. These are listed below.
Debugging through text messages
This is the easiest method for receiving debug information from ReactOS.
The serial port is the most common method used for receiving debug messages from ReactOS. The method used for receiving data from the serial port depends on whether you run ReactOS in a virtual machine or on a real computer. If you plan to use virtual machine, you might want to consider using Com0com instead of named pipe for connecting with virtual serial port.
How to handle serial output from virtual machines can be found on the VM specific debugging pages:
Real computer: Physical serial cable
You will need a physical serial cable if you want to receive debug messages from a real computer through the serial port. This method also requires two computers (one on which you test ReactOS and another one for receiving the debug messages).
The cable needed for this debugging method is a Null-Modem serial cable. You should find it in many computer shops for less than 10 dollars. If you don't have one ready, you can also build one:
DTE1_______________________________________________DTE 2 9pol 25pol (female)__________________________25pol 9pol (female) 5 7 ---GND---------------------GND------- 7 5 2 3 ---RxD--------. ,----------RxD------- 3 2 X 3 2 ---TxD--------' `----------TxD------- 2 3 7 4 ---RTS--------. ,----------RTS------- 4 7 X 8 5 ---CTS--------' `----------CTS------- 5 8 4 20 ---DTR--------. ,----------DTR------- 20 4 X 6 6 ---DSR--o-----' `-------o--DSR------- 6 6 | | 1 8 ---DCD--' `--DCD------- 8 1
Connect the cable to the first serial port of both computers.
Then use a Terminal application like PuTTY or Windows HyperTerminal on the computer for receiving the debug messages. Set it up to listen to the first serial port (COM1 [3F8/IRQ4]) and a baud rate of 115200.
After that, boot ReactOS (Debug) on the test computer and you should receive debug messages. If this doesn't work, check your hardware and your freeldr.ini configuration.
Serial hardware can be tricky to get right, but be persistent. There are a few things to remember:
- Plan which connections are DTE and which DCE, and which gender each has. Know which serial port (1 or 2) you're connecting on each computer.
- If you use a PCI serial card, it could be necessary to pass the COM-adress to the kernel (see below).
- Get the right kind of null modem. There are a few ways to make them and not all are the same.
- Use shorter cables as much as possible.
- Use a serial terminal program such as HyperTerminal or Minicom to observe the remote computer. If you don't see data you can recognize, then something is wrong.
- GDB remote commands start with $ and end with ;. You'll be able to recognize them that way.
- Note: use these settings (In HyperTerminal)
- Bits per second: 115200
- Data bits: 8
- Parity: None
- Stop bits: 1
- Flow control: Hardware
How to start Serial terminal on Linux
- Firstly you need to have installed cu terminal program, for rpm-based systems cu is in uucp-V.V.V.rpm package. ** (For Fedora do: sudo yum -y install uucp)
- Then run:
sudo cu -s 115200 --parity=none -l /dev/ttyS0
- /dev/ttyS0 is your COM port name, you could find its name by reading dmesg command output.
- also "-e -o" keys could be used instead of --parity=none
Serial terminal through FreeBSD
cu goes preinstalled in FreeBSD.
Run in console:
sudo cu -s 115200 -e -o -t -l /dev/cuau0
here /dev/cuau0 is your serial port (COM) device name, find right name of your COM port in in the output of dmesg command. And for other keys:
- "-e -o" options together mean no-parity
- -l /dev/Xdev specifies COM device name
- -t denotes connection is hardwired to a host on a dial-up line (not sure is this key really needed).
And to collect log output into a file for sending, run this script:
DATE=`date +"%F_%H%M%S"` screen -dmS ROSlogger script MyMachine1-ROS-debug-$DATE.log sudo cu -s 115200 -e -o -t -l /dev/cuau0
It will write log into file named MyMachine1-ROS-debug-$DATE.log, here $DATE will be the time when script was started. You could change here MyMachine1 to your machine name.
Debug text output to file
Choose ReactOS (Log file) in the boot menu. The debug messages will go to a file called debug.log. This method has some limitations. Fatal system error messages will not appear in the log file. To redirect the output into another file, edit the kernel parameter /DEBUGPORT=FILE in freeldr.ini. For example:
Options=/DEBUG /DEBUGPORT=FILE:\Device\Harddisk0\Partition1\debug.log /SOS
Options=/DEBUG /DEBUGPORT=FILE:\ArcName\multi(0)disk(0)fdisk(0)\debug.log /SOS
Debug text output to screen
Choose ReactOS (Screen) in the boot menu.
Or edit freeldr.ini to contain an entry like the following:
[ReactOS_Debug] BootType=ReactOS SystemPath=multi(0)disk(0)rdisk(0)partition(1)\ReactOS Options=/DEBUG /DEBUGPORT=SCREEN /SOS
Advanced option: Debugging the debug logger
Sometimes things go wrong and it becomes necessary to debug the debug logger, say SCREEN logger. To do this, it is possible to turn on more than one logger, by specifying it on kernel command line options like this: Edit freeldr.ini to contain an entry like the following:
[ReactOS_Debug2] BootType=ReactOS SystemPath=multi(0)disk(0)rdisk(0)partition(1)\ReactOS Options=/DEBUG /DEBUGPORT=SCREEN /DEBUGPORT=COM1 /SOS
Changing the BAUD rate
If you think that 115200 is way too slow and your serial connection supports higher speeds, like virtual com ports do, you can change it.
1. Open the freeldr.ini in the reactos installation's root folder.
2. Locate the "[ReactOS_Debug]" section
3. Change setting to something like "/BAUDRATE=921600" (tested to work with hyperterminal and putty)
4. Save file.
5. Change your terminals BAUD rate.
Changing the serial port address
Edit the kernel parameter /DEBUGPORT=COM. This could be necessary, if you use a PCI serial card on real hardware. For example:
Options=/DEBUG /DEBUGPORT=COM:0xCC00 /BAUDRATE=115200 /SOS
See kdbg command reference for more information about the built-in kernel debugger.
To use GDB as a kernel debugger, see GDB.
Start QEMU as you normally would, but add the following command line parameters:
This is done so that QEMU starts in the STOPPED state, and allow you to connect using GDB. Now it's time to get GDB off the ground.
- (Assuming you are in the RosBE command line), enter “gdb” to start GDB.
- Enter “file ./output-i386/ntoskrnl/ntoskrnl.exe” to tell GDB where to load information about the kernel.
- Enter “set disassembly-flavor intel” if you prefer Intel syntax.
- Enter “target remote localhost:1234” to connect GDB to QEMU.
- Enter “c” (for “continue“) to have GDB instruct QEMU to start/continue execution of the emulation.
- To manually pause execution, make sure your GDB window has focus and simply enter <CTRL>+<C>
Main article: WinDBG
WinDbg support is still limited. On the x86 build you need a Windows 2003 version of kdcom.dll to make it work. You also need to compile with WINKD=1 switch. On reactos64 builds it's the main debugging method and already works quite well with the current x64 version of kdcom.dll.
Generating even more output
In order to get meaningful debug output it is sometimes necessary to enable extra verbosity.
Turning on verbosity at compile time
Nearly all ReactOS modules use the built in "ReactOS style" debugging functionality. This style is characterized by:
- Verbosity level is usually defined per file.
- Only 2 message levels:
- always enabled (DPRINT1)
- only enabled when NDEBUG is not defined (DPRINT)
Files that follow this style can easily be spotted by this code:
#define NDEBUG #include <debug.h>
To enable full verbosity just comment out the "#define NDEBUG", and remember to uncomment it when submitting patches.
Adding own debug messages
Be sure that you included debug.h
And use DPRINT / DPRINT1, both work like printf, but have some different codes.
sample line to enable debug channels in usermode applications, in file:
; Debug channels HKLM,"SYSTEM\CurrentControlSet\Control\Session Manager\Environment","DEBUGCHANNEL",0x00020000,"+ole,+rpc"
Turning on verbosity at runtime
The easiest way to turn on debug verbosity on any particular component is to use DEBUGCHANNEL environment variable. For example, to get all debug messages from MSI, simply run in CMD:
Then, run the app you wish to test. You can add debug messages from multiple components, for example:
You can find a complete list of the debuggable components here.
After closing down CMD, debug verbosity will change to default.
<Describe details on how to set verbosity level, turn on debug from all components.>
Breaking into the built-in kernel debugger
Bugchecks occur when the operating system can no longer operate safely and to avoid corrupting data, it halts operation. This will normally throw up a Blue Screen Of Death, but if you have the kernel debugger activated, it will drop you into the prompt giving you access to explore the system state. By default ReactOS debug builds have the integrated kernel debugger (kdbg) enabled. Release builds do NOT have this feature enabled and will only display a blue screen.
There are two ways for forcing a bugcheck, each one employing a different method:
If you have a debug build and want to halt the system for any given reason and break into the the kernel debugger, you can force a bugcheck from the keyboard by simply typing:
Remember that kdbg output goes out through the serial port, but it receives input from the keyboard by default.
To allow input through the serial port as well start with ReactOS with the boot option "ReactOS (RosDbg)" or add the command /KDSERIAL to your freeldr.ini boot options.
Breaking on user mode Exceptions
For each type of exception known by KDB you can set the condition when KDB should be entered individually for first and last chance. The possible settings for the conditions are never, umode, kmode and always.
- never: kdbg won't be entered when exceptions are raised
- umode: kdbg will be entered when the exception was raised in usermode
- kmode: kdbg will be entered when the exception was raised in kernel mode
- always kdbg will be entered on every exception
To change the condition to enter KDB on all exceptions to "always" (default is "kmode"), enter the debugger and type:
set condition * first always
Type "cont" to continue normal execution.
This is useful when you want to halt the operating system when it hits a particular area of code you might be debugging. This is especially useful as you can get an immediate back trace to see where the code flow came before the bugcheck was forced.
This is done by using KeBugCheck() or ASSERT() in the code.
Generating a backtrace
In order to generate a backtrace you must break into the KDBG prompt.
Enter 'bt' and hit <return>, you should see something similar to the following:
(drivers\filesystems\vfat\rw.c:809) <\ReactOS\system32\kernel32.dll> Entered debugger on embedded INT3 at 0x0008:0x800935f2. kdb:> bt Eip: <ntoskrnl.exe:935f3 (lib\rtl\i386\debug_asm.S:31 (DbgBreakPoint@0))> Frames: <vfatfs.sys:97de (drivers/filesystems/vfat/misc.c:111 (VfatDispatchRequest))> <vfatfs.sys:9b25 (drivers/filesystems/vfat/misc.c:167 (VfatBuildRequest))> <ntoskrnl.exe:3ab23 (ntoskrnl/io/iomgr/irp.c:1088 (IofCallDriver))> <ntoskrnl.exe:36206 (ntoskrnl/io/iomgr/iofunc.c:686 (IoSynchronousPageWrite))> <ntoskrnl.exe:59daa (ntoskrnl/mm/section.c:6330 (MmspWriteDataSectionPages))> <ntoskrnl.exe:244c6 (ntoskrnl/ex/work.c:162 (ExpWorkerThreadEntryPoint))> <ntoskrnl.exe:70e90 (ntoskrnl/ps/thread.c:134 (PspSystemThreadStartup))> <ntoskrnl.exe:7b142 (ntoskrnl\ke\i386\ctxswitch.S:258 (KiThreadStartup@156))> kdb:>
The bt command will show a backtrace of the currently attached thread, so it may be necessary to use the 'thread attach' command, please refer to the kdbg manual for more details.
Examining this in more detail, we can see that the bugcheck occured via the INT3 operation which then dropped us into kdb. The next line shows us Eip which is the instruction pointer, and this points to the last address before the system halted.
Following on from that are the frames. This is the important part of generating our backtrace and it contains all the function addresses in the buildup to the bugcheck. This is the crutial information developers need to understand the codeflow before the bugcheck.
Occasionally, there will come a time when you will need to manually translate addresses. When kdbg is not enabled and a bugcheck occurs, you will be presented with a stack trace similar to the following:
(subsystems\win32\csrss\win32csr\conio.c:1101) Console_Api Ctrl-C
*** Fatal System Error: 0x00000001 (0x80079279,0x00000000,0x0000FFFF,0x00000000)
<\SystemRoot\System32\NTOSKRNL.EXE: 29bb> <\SystemRoot\System32\HAL.DLL: 4749> <\SystemRoot\System32\NTOSKRNL.EXE: 54cb4> <\SystemRoot\System32\NTOSKRNL.EXE: 582bf> <\SystemRoot\System32\NTOSKRNL.EXE: 583fd> <\SystemRoot\System32\NTOSKRNL.EXE: 89956> <\SystemRoot\system32\drivers\videoprt.sys: 2417> <\SystemRoot\system32\drivers\vbemp.sys: 17f5> <\SystemRoot\system32\drivers\vbemp.sys: 19cf> <\SystemRoot\system32\drivers\videoprt.sys: 1c48> <\SystemRoot\System32\NTOSKRNL.EXE: 34c17> <\SystemRoot\System32\NTOSKRNL.EXE: 21e0> <\SystemRoot\System32\NTOSKRNL.EXE: 2908> <\SystemRoot\System32\NTOSKRNL.EXE: 29bb> <\SystemRoot\System32\NTOSKRNL.EXE: 85fa8>
As you can see, this is largely the same as the information presented when issuing a 'bt' command in kdbg. The problem here however, is that only the addresses are given. As these addresses are different for everyone's builds, this information is useless for anyone trying to follow what events occurred in the lead up towards to bugcheck.
The solution here is to translate the addresses into human readable function names. This is done via a tool named 'raddr2line' which is a modified version of the Unix tool 'addr2line'. This tool will translate the addresses given to it into file names and line numbers. It does this by using debug information in the executable files to associate the address with this human friendly info and outputs it into the console. This information can be pasted into the above debug log alongside the addresses providing the developers with a detailed stack trace.
raddr2line is included in the Reactos Build Environment. It is invoked in the following way :
raddr2line <file> <address>
So taking the bottom address in the above stack trace :
C:\Users\Ged\MyFiles\ReactOS\clean_source>raddr2line ntoskrnl.exe 85fa8 C:\Users\Ged\MyFiles\ReactOS\clean_source\output-i386\ntoskrnl\ntoskrnl.exe obj-i386\ntoskrnl\ex\zw.S:253 (ZwClearEvent)
we can see here that the address translation for 0x85fa8 is line 253 in file ntoskrnl\ex\zw.S (this will differ if you try it on your build)
This information can now be added into the above stack trace as follows :
<\SystemRoot\System32\NTOSKRNL.EXE: 29bb> <enter next one here> <\SystemRoot\System32\NTOSKRNL.EXE: 85fa8> obj-i386\ntoskrnl\ex\zw.S:253 (ZwClearEvent)
Enabling Kernel Tracing
Refer to this article for instructions on how to enable kernel tracing.
Debug Page Heap (DPH)
Based on functionality of Windows Page Heap Verification, this mechanism is useful for more in-depth debugging of usermode Heap issues (crashes in ntdll:heap functions). It can be enabled per-application or globally (for the whole system).
To enable DPH per particular application, gflags.exe is needed. This program is a part of few packages, namely Debugging Tools for Windows and Windows Support Tools. Please make sure you download 32bit package. GFlags.exe must be executed from ReactOS with the following syntax: "gflags /p /enable application.exe /full" and the application executed afterwards. Debug can be disabled by rerunning gflags with /disable switch or by reboot.
[Need paragraph on system-wide DPH enabling]
How to read/debug BugCheck messages
- BugCheckCode parameter hex values are defined in:
\include\reactos\mc\bugcodes.mc or, generated .h-version for i386: \obj-i386\include\reactos\bugcodes.h
- Some messages have useful instructions