KmtestsHowto

Building and Running Tests

• Check out rostests from http://svn.reactos.org/reactos/trunk/rostests (View in browser) and put it into the ReactOS source folder as modules\rostests.
• Fire up your favorite build system (the test suite should build with RBuild/GCC, CMake/GCC, CMake/MSVC) and build the kmtest_all target (not available for RBuild).

 make kmtest_all
 nmake/nologo kmtest_all

• Put the resulting files (kmtest_.exe, kmtest_drv.sys, example_drv.sys and any other *_drv.sys files) into the same folder accessible by your ReactOS or Windows Server 2003 installation (a ReactOS Boot-CD will put them in ReactOS\bin) and run kmtest_.exe from a command prompt in that folder to see the available tests and options. Running a test might look as follows:

 kmtest_ Example

Test structure

Test naming and categories

Test must be sorted into categories, according to the module (usually ntoskrnl/hal) they test, and possibly which part of the module. Each category has its own folder for the test sources to reside in, and a prefix to be appended to each test name.

The following categories already exist:

Example tests

• Folder: example, Prefix: none

Functions from ntoskrnl, hal and corresponding macros or inline functions

• Folder: ntos_ex, Prefix: Ex – Executive library functions
• Folder: ntos_fsrtl, Prefix: FsRtl – File-System Runtime Library routines
• Folder: ntos_io, Prefix: Io – I/O-Manager support functions
• Folder: ntos_ke, Prefix: Ke – Core kernel support routines
• Folder: ntos_mm, Prefix: Mm – Memory manager support functions
• Folder: ntos_ob, Prefix: Ob – Object manager functions

Runtime Library functions and corresponding macros or inline functions

• Folder: rtl, Prefix: Rtl

Simple kernel-mode tests

Simple tests are part of the main kmtest driver. They should be placed in a single source file named like the test in the appropriate category sub-folder. An example is provided by the Example test.

The steps required for such a test are:

• add the source file to kmtests\CMakeLists.txt to be included in KMTEST_DRV_SOURCE
• add the source file to kmtests\kmtest_drv.rbuild
• add the test to kmtests\kmtest_drv\testlist.c in alphabetic (ASCII) order
• in the source file (testname.c), include <kmt_test.h>, then wrap your test in the START_TEST macro

START_TEST(Minimal)
{
    /* test goes here */
}

Test parts in separate drivers

For tests which are unsuitable for the main kmtest driver (such as tests directly involving the driver object, or dispatch routines), special-purpose drivers can be used. kmtest_drv\kmtest_standalone.c provides a framework for such drivers. The test must provide a TestEntry and TestUnload functions and can register handles for different events with the framework. See the example_drv driver for an example.

The steps required for such a test are:

• create a CMakeLists.txt file for the driver
• add the driver to the kmtest_drivers target in the kmtests root CMakeLists.txt file.
• create a testname_drv.rbuild file for the driver
• add the directory containing the driver to kmtests\CMakeLists.txt
• add the path to the driver's rbuild file to kmtests\directory.rbuild
• for rbuild, add the driver to reactos.dff so it goes on the bootcd
• in the source file (testname_drv.c), include <kmt_test.h>, then wrap your test in the START_TEST macro as with kernel-mode tests.
• create a user-mode part that calls the driver

User-mode test parts

Some tests require some interaction with a user-mode application (simple examples would be CreateFile or DeviceIoControl). This can be achieved using user-mode test parts, which are included in the kmtest.exe application.

In the presence of a user-mode test, that test part will always be called instead of the kernel-mode test with the same name. The user-mode part is then responsible for using KmtRunKernelTest as required to run the kernel-mode part.

User-mode tests are also responsible for loading and starting any tests in separate drivers. The Example_user test demonstrates the functions available for that purpose.

The steps required for such a test are:

• add the source file to kmtests\CMakeLists.txt to be included in KMTEST_SOURCE
• add the source file to kmtests\kmtest.rbuild
• add the test to kmtests\kmtest\testlist.c in alphabetic (ASCII) order
• in the source file (testname_user.c), include <kmt_test.h>, then wrap your test in the START_TEST macro as with kernel-mode tests.

RTL-type tests

Since Rtl functions can be used from both user and kernel mode, tests for these (or similar) can run in either mode. The special preprocessor symbol KMT_EMULATE_KERNEL should be defined in such tests to emulate any functions that are unavailable in user mode (such as IRQ-Level functions). The RtlMemory test is an example of such a test.

The steps required for this type of test are:

• add the source file to kmtests\CMakeLists.txt to be included in both KMTEST_SOURCE and KMTEST_DRV_SOURCE.
• add the source file to kmtests\kmtest.rbuild
• add the source file to kmtests\kmtest_drv.rbuild
• add the test to kmtests\kmtest\testlist.c with the normal test name (alphabetic (ASCII) order)
• add the test to kmtests\kmtest_drv\testlist.c (alphabetic (ASCII) order) with KM appended to the test name string to indicate the kernel-mode version
• in the source file (testname.c), define KMT_EMULATE_KERNEL, then include <kmt_test.h>, and wrap your test in the START_TEST macro as usual.

Hidden tests

By prefixing a test name with a minus (-) in testlist.c, it will become "hidden", meaning a simple call to kmtest or kmtest --list will not show it. This is useful for tests that might crash or show other unexpected behavior (such as requiring user interaction) and are thus unsuitable for automated testing runs. A test with both user- and kernel-mode components must be hidden in both test lists, or in neither. For the test list sorting order, ignore the leading minus

Test syntax

Testing a condition

The basic primitive for writing tests is a condition check using the ok macro. The macro takes a condition that must be true for the test to succeed (similar to assert), and a message to be displayed on failure.

Size = 1024 * 1024 * 1024 * 2;
Pointer = ExAllocatePoolWithTag(Size, NonPagedPool, 'xxxx');
ok(Pointer == NULL, "Allocating %lu bytes of non-paged pool succeeded unexpectedly! Pointer = %p\n", Size, Pointer);

ok will automatically increase the count of tests performed, and also count failures. The source file name and line number of the ok invocation will be added to the failure message, so it need not contain any information included in that line. It should, however include any information against which the test failed (such as a return or variable value) and non-obvious values such as loop counters.

It is preferred that no tests ever fail on Windows.

Convenience macros

Several ok_* convenience macros for the most common conditions are also available.

ok_irql(PASSIVE_LEVEL);
ok_bool_false(KeAreAllApcsDisabled(), "KeAreAllApcsDisabled returned");
ok_eq_pointer(Something->ListEntry.Flink, &Something->ListEntry);
ok_eq_ulong(KeGetCurrentProcessorNumber(), 0);
ok_eq_int(Irp->RequestorMode, UserMode);
ok_eq_hex(Status, STATUS_ACCESS_VIOLATION);
ok_eq_str(AnsiName, "Ansi Name");
ok_eq_wstr(UnicodeName, L"Unicode Name");

Warning: Most of these macros will evaluate the expression passed to them twice! Do not call functions that have side-effects inside these macros!

ok_eq_long(InterlockedIncrement(&Variable), 5);

Instead, add a variable for the return value:

Ret = InterlockedIncrement(&Variable);
ok_eq_long(Ret, 5);

The ok_bool_true and ok_bool_false macros are notable exceptions that are safe to use.

Adding informational output

For adding any additional information that may be useful (for instance when debugging why a test fails), the trace macro can be used.

trace("Registry Path: %wZ\n", RegistryPath);

Note that it is preferred to do condition checks whenever possible, as these will provide results which can be automatically parsed.

Skipping tests

If some condition prevents specific tests from running, these tests can be skipped using the skip() macro. This is preferred over executing tests that might otherwise crash.

Pointer = ExAllocatePoolWithTag(PAGE_SIZE, PagedPool, 'xxxx');
ok(Pointer != NULL, "Out of memory\n");

if (!skip(Pointer != NULL, "Allocation failed\n"))
{
    /* do stuff that uses the memory */
    ExFreePoolWithTag(Pointer, 'xxxx');
}

skip() should be passed a condition which must be TRUE in order for the following test(s) to succeed. It will then return whether to skip the test(s).

An approach in line with the Winetest version can also be taken to prevent nesting:

Pointer = ExAllocatePoolWithTag(PAGE_SIZE, PagedPool, 'xxxx');

if (skip(Pointer != NULL, "Allocation failed\n"))
    goto done;

/* do stuff that uses the memory */

done:
if (Pointer)
    ExFreePoolWithTag(Pointer, 'xxxx');

Checking for OS build type

As many tests go near the border of what is "acceptable" behavior when calling API functions, the behavior differs between debug and release builds of the operating system. A checked build will have much stricter checks, and issue bugchecks for behavior that a free build will accept without any problem. To avoid system crashes and still allow these tests to be run on release builds, the offending pieces of code can be wrapped in an if statement checking the variable KmtIsCheckedBuild. This variable is available to all kernel-mode tests, and will be 1 for a checked (debug) build and 0 for a free (release) build at runtime.

The behavior of some synchronization techniques also varies greatly between build types. Synchronization is usually much simpler in a uniprocessor kernel compared to a multiprocessor kernel. The variable KmtIsMultiProcessorBuild can be used to check the kernel build type at runtime.

The KernelType test demonstrates the use of these variables.

Guarded Memory Allocations

To detect buffer overruns in the functions to be tested, guarded memory allocations can be used. A memory area allocated using KmtAllocateGuarded has a non-accessible page directly behind it, so that any buffer overrun will result in an access violation. Such an area can be freed using KmtFreeGuarded.

The GuardedMemory test is an example (and a test) for using guarded allocations.