Writing tests

From ReactOS Wiki
Jump to: navigation, search
This page is probably outdated

The information on this page might not be valid for the current state of ReactOS.
A Wiki Administrator should look at this page and decide or discuss what to do with it.


Overview

In order to be useful to users, ReactOS needs to be at least as good today as it was yesterday. Unfortunately, due to the complexity of software as large as ReactOS, this can be difficult to achieve. Automated tests have proven to be an effective method of spotting undesired side effects of changes made to the software early.

Writing a simple test

This section assumes that the component being tested has been prepared for testing. See the section named Test-enabling a component for information about how to do this.

Choose a filename for the test

Group related tests under the same base filename and postfix the base filename with - (dash) and a number and a .c extension. For instance, file-1.c and file-2.c are related tests.

Create the new file for the test

Include the necessary headers and also "regtests.h" which is the interface to the testing framework.

Implement the test dispatcher

The name of the test dispatcher is derived from the filename chosen for the test. The first letter in the test dispatcher function name is the capital version of the first letter in the test filename. All other letters in the test filename up till, but not including the .c extension are copied to the test dispatcher function name. The letters are all converted to lower case. If a letter in the test filename is not one of a-z, A-Z or 0-9, it is converted to an underscore (_). Finally Test is appended to the test dispatcher function name. So file-1.c gives us a test dispatcher function name File_1Test and myTest-1.c gives Mytest_1Test.

A test dispatcher function has the following prototype:

void (TestDispatcherFunction)(int Command);

Currently, Command can assume one of two values which are listed below. Tests should return TS_FAILED for commands they don't support or understand. The dispatcher returns data to the framework in the global char array variable _Buffer.

TESTCMD_TESTNAME

If Command is TESTCMD_TESTNAME, the testing framework asks the test for a textual description of itself. The test returns the description in _Buffer.

TESTCMD_RUN

If Command is TESTCMD_RUN, the testing framework asks the test to execute itself. If the test is executed successfully, the test returns TS_OK. If something went wrong during the test, the test returns TS_FAILED and it also returns a human readable description of what went wrong in _Buffer. This information is written to the console when the tests are executed. Usually the test dispatcher function will have a structure similar to the following:

void MyTestDispatcher(int Command)
{
	switch (Command)
	{
		case TESTCMD_RUN:
			RunTest();
			break;
		case TESTCMD_TESTNAME:
			strcpy(_Buffer, "My test");
			break;
		default:
			_Result = TS_FAILED;
			break;
	}
}

Here RunTest() is an internal function that will execute the test:

static void RunTest()
{
	_AssertEqualValue(1, abs(1));
	_AssertEqualValue(0, abs(0));
	_AssertEqualValue(1, abs(-1));
}

Here the testing framework function _AssertEqualValue() is used to check that certain conditions are true. In this case the conditions are that abs(1) should return 1, abs(0) should return 0 and abs(-1) should return 1. There are more testing framework function to perform assertions and they will all be explained later in the section called Assertions. Since most test dispatchers are not more complicated than the above test dispatcher, a shorter version of the dispatcher is:

_Dispatcher(MyTestDispatcher, "My test")

Please use this helper macro if possible.

Some tests need to set up some state in the component they test before the tests can be performed. The framework provides a macro named _SetupOnce() to help declare this function. It is used like this:

_SetupOnce()
{
	/* This code is executed only once before the first test is run */
	DllMain(NULL, DLL_PROCESS_ATTACH, NULL);
}

Please use this helper macro if possible. For DLL's, DllMain is usually called with reason DLL_PROCESS_ATTACH as shown above. DllMain() must be declared before use like this:

extern BOOL
STDCALL
DllMain(HANDLE hInstDll,
        ULONG dwReason,
        LPVOID lpReserved);

The above set up code is usually put in a file called setup.c. Please do this for all your tests. The complete setup.c file could look like this:

#include <windows.h>
#include "regtests.h"

extern BOOL
STDCALL
DllMain(HANDLE hInstDll,
        ULONG dwReason,
        LPVOID lpReserved);

_SetupOnce()
{
	DllMain(NULL, DLL_PROCESS_ATTACH, NULL);
}

or it could just do nothing if no state is needed to be set up in the component (eg. for a simple .exe component). The complete test for abs() is listed below.

#include <stdlib.h>
#include <windows.h>
#include "regtests.h"

static void RunTest()
{
	_AssertEqualValue(1, abs(1));
	_AssertEqualValue(0, abs(0));
	_AssertEqualValue(1, abs(-1));
}

_Dispatcher(AbsTest, "Abs")

The test can be executed by running “make module” in the top-level directory. _AssertEqualValue() is used to assert that abs() behaves correctly when passed certain values in its first and only parameter. Next, the possible assertions are explained.

Assertions

Assertions are used to verify that certain conditions are true. If one or more conditions are found to be false then the test will not pass and will return a status of TS_FAILED to the framework. Below is a list of possible assertions.

_AssertTrue

_AssertTrue(Condition). Asserts that the value of the parameter Condition is true.

_AssertFalse

_AssertFalse(Condition). Asserts that the value of the parameter Condition is false.

_AssertEqualValue

_AssertEqualValue(Expected, Actual). Asserts that the value of the parameter Actual of value type (int, float, double etc.) is equal to the value of the parameter Expected of value type.

_AssertEqualWideString

__AssertEqualWideString(Expected, Actual). Asserts that the value of the parameter Actual of wide string type (wchar_t) is equal to the value of the parameter Expected of wide string type. The comparison is case sensitive.

_AssertNotEqualValue

_AssertEqualValue(Expected, Actual). Asserts that the value of the parameter Actual of value type (int, float, double etc.) is not equal to the value of the parameter Expected of value type.

Simulating functionality

Sometimes a function depend on other functions. This can make the function harder to test since it may not even be possible to execute these functions in the testing environment. When developers face this problem in object-oriented languages, they use a concept called mock objects. Basicly, part of the code is replaced with a simpler piece of code (the mock object) that simulates the real code and asserts that it is used correctly by the code being tested. The code being tested is also referred to as the domain code. In ReactOS, the same concept is used, but the implementation is tailored to procedural languages like C.

The functions can be divided into two categories. Internal functions and external functions. Internal functions are functions that are located in the same component as the function that is to be tested. External functions are functions that are located in another component (usually a DLL). Here it is discussed how to test functions with a dependency on external functions. In the next section it is discussed how to test functions with a dependency on internal functions.

Each function in the external components must have a stub provided. See the section named Test-enabling a component for information about how to do this. Then, in the test file, an array of hooks is declared.

static HOOK Hooks[] =
{
	{"CloseHandle", MockCloseHandle},
	{NULL, NULL}
};

The first field of each entry is the name of the function to replace (the hooked function). The second field is a reference to a function with the same signature as the hooked function. This is the mock function and it should contain checks to validate that it is used as expected.

Important note: The last entry in the hook table must be {NULL, NULL} otherwise strange results occur when running the tests.

The hook array is registered with the testing framework by calling _SetHooks() like this:

_SetHooks(Hooks);

Until _UnsetHooks() is called like this:

_UnsetHooks(Hooks);

control will be redirected to the mock function whenever the hooked function is referenced in the code. In more complex tests, several hook tables can be used. If two or more hook tables are registered at the same time, then the union of the all the tables are registered. If two or more hook tables containing the same function name are registered at the same time, then the hook for that particular function that is located in the last registered table wins. This means that all previous registrations of the function are overwritten. To remove all registrations no matter how many hook tables are registered, _UnsetAllHooks() can be called like this:

_UnsetAllHooks();

Below is a complete example of a test for WSACloseEvent(). This function is supposed to use the OS function CloseHandle() to perform the hard work so the test makes sure that it does exactly this. In this example, CloseHandle() is the hooked function and MockCloseHandle() is the mock function.

#include <windows.h>
#include <winsock2.h>
#include "regtests.h"

#define TestHandle (HANDLE) 1

static BOOL CloseHandleCalled = FALSE;

static BOOL STDCALL
MockCloseHandle(HANDLE hObject)
{
	CloseHandleCalled = TRUE;
	_AssertEqualValue(TestHandle, hObject);
	return TRUE;
}

static HOOK Hooks[] =
{
	{"CloseHandle", MockCloseHandle},
	{NULL, NULL}
};

static void
TestWSACloseEvent()
{
	_SetHooks(Hooks);
	WSACloseEvent(TestHandle);
	_AssertTrue(WSACloseEvent(TestHandle));
	_AssertEqualValue(NO_ERROR, WSAGetLastError());
	_AssertTrue(CloseHandleCalled);
	_UnsetAllHooks();
}

static void
RunTest()
{
	WSADATA WSAData;

	WSAStartup(MAKEWORD(2, 0), &WSAData);
	TestWSACloseEvent();
}

_Dispatcher(WinsockeventTest, "Winsock 2 event")

Setup.c must look like the following:

#include "regtests.h"

extern BOOL
STDCALL
DllMain(HANDLE hInstDll,
        ULONG dwReason,
        LPVOID lpReserved);

_SetupOnce()
{
	DllMain(NULL, DLL_PROCESS_ATTACH, NULL);
}

Writing tests for more complex functions

Not all functions are as simple as abs() which was tested earlier. The abs() function is simple. When passed a value it returns another value. It does not call other functions which may not able to be executed in the build environment. Sometimes functions do exactly that, so they have to be rewritten in order for it to be testable. In order to be testable, a function need only to obey one rule - it must not call another function in the same component directly. The test for the function must be able to intercept control when it is passed to other functions wether they are located in the same component or in another. In this section it is discussed how to call other functions in the same component and still enable the test to have the control it needs in order to test the function.

KeAreApcsDisabled() is a function that is a little more complex than abs(). It calls another function, KeGetCurrentThread(), that is responsible for providing a pointer to the KTHREAD object for the the current thread. In the testing environment this is not possible so it needs to be rewitten.

BOOLEAN STDCALL
KeAreApcsDisabled(VOID)
{
	return KeGetCurrentThread()->KernelApcDisable ? TRUE : FALSE;
}

Splitting the function in two, it becomes possible to test this functionality. One function (KeAreApcsDisabled) is responsible for calling KeGetCurrentThread() and then pass the result of that function to another function (KiAreApcsDisabled) which is responsible for returning wether APCs are disabled or not. The result is this:

BOOLEAN STDCALL
KiAreApcsDisabled(PKTHREAD Thread)
{
	return Thread->KernelApcDisable ? TRUE : FALSE;
}

BOOLEAN STDCALL
KeAreApcsDisabled(VOID)
{
	return KiAreApcsDisabled(KeGetCurrentThread());
}

Now a test can be written for KiAreApcsDisabled(). The full test is listed below.

#include <ntoskrnl.h>
#include "regtests.h"

static void RunTest()
{
	KTHREAD Thread;

	Thread.KernelApcDisable = FALSE;
	_AssertFalse(KiAreApcsDisabled(&Thread));

	Thread.KernelApcDisable = TRUE;
	_AssertTrue(KiAreApcsDisabled(&Thread));
}

_Dispatcher(ApcTest, "APC")

Sometimes it is not possible to extract the call to a dependent function from the function being tested. It might for instance be passed some values in its parameters which are computed in the function being tested. The same principle that was used in the earlier example can still be used however as the next example will show. The following function will be tested:

VOID STDCALL
NtQueueApcRundownRoutine(PKAPC Apc)
{
	ExFreePool(Apc);
}

The direct call to ExFreePool() need to be replaced with an indirect call so the function can be tested. The first step is to pass a pointer to the ExFreePool() function as a parameter to NtQueueApcRundownRoutine(). Since the signature is changed, the function is renamed to NtiQueueApcRundownRoutine() in the process and another function with the same name will replace it. The new function named NtQueueApcRundownRoutine() will call the internal version of the function and pass the correct parameters.

typedef VOID STDCALL
PExFreePool(PVOID Block);

VOID STDCALL
NtiQueueApcRundownRoutine(PKAPC Apc,
	PExFreePool pExFreePool)
{
	pExFreePool(Apc);
}

VOID STDCALL
NtQueueApcRundownRoutine(PKAPC Apc)
{
	NtiQueueApcRundownRoutine(Apc,
		ExFreePool);
}

The function definition PExFreePool is used by both the domain code and the test so it should be placed in a central header. Note the conventions here are to prefix the type definition with a large P (for pointer) and prefix the name of the function parameter with a small p. The function which can be tested is prefixed with an additional i for Internal. Please use these conventions when creating new testable functions. The test for the internal function is listed below.

#include <ntoskrnl.h>
#include "regtests.h"

static KAPC TestApc;
static BOOLEAN MockExFreePoolCalled = FALSE;

static VOID STDCALL
MockExFreePool(PVOID Block)
{
	_AssertFalse(MockExFreePoolCalled);
	_AssertEqualValue(&TestApc, Block);
	MockExFreePoolCalled = TRUE;
}

static void RunTest()
{
	NtiQueueApcRundownRoutine(&TestApc,
		MockExFreePool);
	_AssertTrue(MockExFreePoolCalled);
}

_Dispatcher(ApcTest, "APC")

It is verified that ExFreePool() is called only once and that the correct parameter is passed to it.

Test-enabling a component

For a component to integrate with the testing framework, a few modifications to it are needed. The following is a walk-through of test-enabling kernel32.dll. The following directory structure is needed:

<component>\tests

where <component> is the path of the component that is to be test-enabled. In this example it is lib\kernel32. The file:

<component>\tests\kernel32.xml

should be filled with:

<module name="kernel32_test" type="test">
	<include base="rtshared">.</include>
	<define name="_DISABLE_TIDENTS" />
	<define name="_SEH_NO_NATIVE_NLG" />
	<define name="WINVER">0x0500</define>
	<library>rtshared</library>
	<library>regtests</library>
	<library>kernel32_base</library>
	<library>pseh</library>
	<library>rosrtl</library>
	<library>ntdll</library>
	<library>msvcrt</library>
	<linkerflag>-lgcc</linkerflag>
	<linkerflag>-nostartfiles</linkerflag>
	<linkerflag>-nostdlib</linkerflag>
	<directory name="tests">
		<file>CreateFile.c</file>
	</directory>
	<file>setup.c</file>
	<xi:include href="stubs.xml" />
</module>

The important things are explained here. The module type is “test” and the module name is the name of the module with the domain code (the code being tested) postfixed with _test. Please follow this convention as it makes it clearer that ”module_test” contain the tests for ”module”. The defines are domain code specific. The two libraries ”rtshared” and ”regtests” are always needed. The rest of the libraries and the linkerflags are also domain code specific. The directory ”tests” is not required, but it helps to know which files are actual tests and which are setup or other helper code for the tests. Setup.c contains the setup code that is executed only once before the first test is run. Finally the stubs.xml file is included to let the testing framework know for which symbols it should create stubs to allow hooking of functions located in external components.

First you need to split the code in the domain code module out into a new module of type ”objectlibrary”. This is so both the domain code module and the test module can share the code. Name the new module of type ”objectlibrary” the same as the domain code module, but with ”_base” postfix. So for the kernel32 module, the shared code will be in a module named kernel32_base. When the source code files are moved from the kernel32 module til the kernel32_base module, then import the kernel32_base library into the kernel32 module.

Setup.c should look like the following:

#include "regtests.h"

_SetupOnce()
{
	// This code is executed only once before the first test is run.
}

Be careful when importing libraries. The tests need to be run in the build environment and some DLL's may not be available then. Instead provide stubs for the functions in the DLLs so mock functions can be used instead. In order to transfer control to a mock function instead of the hooked function, the framework need to intercept the call. This is done by providing a small piece of code (a stub) for each function that needs to be hooked. The stubs are auto-generated during build from a file (usually called stubs.xml) which contain descriptions of the stubs. Stubs.xml is usually located at:

<component>\tests\stubs.xml

For each unresolved reference in the component, a stub need to be added to this file. The format is like this:

<component name="ntdll.dll">
	<symbol>NtClose@4</symbol>
	<symbol newname=”RtlAllocateHeap”>HeapAlloc@12</symbol>
</component>

There can be more than one component and symbol elements in the file. The value of the name attribute of the component element is the name of the DLL in which the hooked function is located. The value of the symbol element is the name of the function as seen by the linker and is also the name the linker complains about not being able to find in case of an unresolved reference. The symbol is the decorated function name that includes the calling convention and size of parameters in bytes. Note that each parameter normally takes up 4 bytes of space. Examples of symbols include NtCreateProcess@34 (STDCALL), @InterlockedIncrement@4 (FASTCALL) and printf (CDECL). The newname attribute is optional. It specifies that the name of the hooked function is different from the undecorated symbol. This can happen when the component being tested use functions of it's own and those functions are exports forwarded to a function in another DLL under a different name. Let us take a look at the two examples above.

The first example tell the framework that it should create a stub for the function called NtClose with STDCALL calling convention and 1 parameter (4/4=1). The hooked function is located in the DLL called ntdll.dll.

The second example tell the framework that it should create a stub for the function called HeapAlloc@12 with STDCALL calling convention and 3 parameters (12/4=3). The hooked function is located in the DLL called ntdll.dll, but it has a different name. The name of the hooked function is RtlAllocateHeap, not HeapAlloc.

Finally, include the build system file for the module tests in the build system file for the module. This is done as usual:

<directory name="tests">
	<xi:include href="tests/kernel32.xml" />
</directory>

Now you can type “make kernel32_test” at the top-level directory to build and run the tests.