Shell Documentation

From ReactOS Wiki
Revision as of 19:42, 17 March 2016 by PurpleGurl (talk | contribs) (Undo revision 41199 by PurpleGurl (talk) - Undo accidental overwrite.)
Jump to: navigation, search
Shellclasses 2.png


The CMenuSite class and the window it creates act as an intermediary between a BaseBar and a MenuBand, forwarding the events and messages either to the child band, or to the parent bar, as necessary. It handles the sizing of the child band, to adapt it to changes in the available space. It also provides certain services related to the positioning and sizing of the child bar, which it handles by forwarding some requests to the child, and the rest to the parent.

This implementation was done within rshell, a DLL created by Giannis that is used to provide alternative implementations of the shell32 classes, which can be referenced instead of the real ones while constructing the Start Menu. With rshell, the code can be tested directly from within Visual Studio, using all of the debugging features of the IDE.

Gigaherz continued by writing a wrapper for CMenuBand that logs the calls, with parameters and return values, done to a real CMenuBand from Windows. This allowed for seeing the essential features needed to implement the basic displaying of content in a CMenuBand, in the way expected by Windows' Start Menu and Menu Site.

The rough, partial implementation of CMenuBand was ported from ReactOS' shell32 into rshell. Allow the start menu to use this CMenuBand required implementing support for QueryService and Exec methods. Although the behavior of the Exec system is still largely unknown, there's enough information from the call logs and stack traces to realize that command id 16 is used to set "large icons" mode, and id 19 is used in the Popup function, and requires returning S_FALSE.

The band currently assumes that a null mask means "give me everything", and it requests the optimal size from the toolbar to provide a value different than 0. This makes the CMenuBand at least partially functional.

Analyzing CMenuBand allowed for adding rudimentary support for a custom CMenuBand which shows both a shell folder and a static menu. The callbacks are also used to obtain the icons for the static menu items.

Gigaherz continued by writing a wrapper for CMenuDeskBar, in the same style used for CMenuBand. The partial implementation CMenuDeskBar was ported from shell32 to rshell. Merging the code from CBaseBar into it simplified the implementation. The combined log from the wrappers for CMenuDeskBar and CMenuBand allowed seeing that the SetIconSize from CMenuDeskBar is the one that is supposed to notify the CMenuBand of the icon size choice. To achieve this, it calls the Exec method with cmd=16 and opts=2. It appears that opts=2 means “big icons”, and calling this method with opts=0 keeps the icons small.

The Popup system is supposed to calculate the optimal size of the start menu, and notify the different objects that it are about to be displayed.

Clicking on Menu Items

Clicking on menu items involves sending a callback notification for the currently selected item, asking the callback to execute the appropriate action of the item.

Some of the items are not meant to be clicked on and are supposed to show a sub menu instead. Implementing these required subclassing the toolbar windows to add a timer on hover and handle the opening of the submenu in the resulting WM_TIMER event, which would be received by the toolbar, but needed to be handled by the code.

The OnSelect behavior seems to be something close to this:

  • The DeskBar’s SubMenu parent is the IMenuPopup interface of the DeskBar’s “owner” Site, except for the top-level start menu, which does not have a submenu parent.
  • The DeskBar’s SubMenu child is the contained Band obtained through the contained MenuSite by using the SID_SMenuPopup service.
  • The Band’s SubMenu parent is the DeskBar obtained through the parent MenuSite which is itself obtained through the SID_SMenuBandChild.
  • When a user clicks a toolbar item and the item executes an action, the Band sends an OnSelect(Execute) notification to its SubMenu parent.
  • When a user presses the alt key, the Band sends an OnSelect(FullCancel) notification to its parent.
  • When a different window activates, the DeskBar sends an OnSelect(FullCancel) to itself.
  • When the DeskBar receives an OnSelect notification that results in closing of a submenu, it notifies the child using OnSelect(CancelLevel).
  • When the DeskBar receives a notification other than CancelLevel, it notifies its parent menu. This is not mutually exclusive with the previous “rule”: Some notifications are forwarded to the child and parent.
  • When the Band receives a cancellation event it cancels its current child, if any.
  • When the Band receives a keyboard navigation event it either notifies the parent or, in the case of SelectRight, it opens the submenu of the currently selected item.
  • SelectLeft is equivalent to CancelLevel except that it is notified to the parent also so it can select the appropriate item.

This implementation requires a focus manager class, which takes care of hooking the application’s message flow to intercept keyboard and mouse events and do the appropriate actions in each case.

Since the focus manager is unable to process the WM_ACTIVATION messages. a handler was added to the DeskBar class, which will send OnSelect notifications when a window gets deactivated in favour of a foreign one. To be able to prevent the menu from closing if the window being shown is a submenu, the GetSite method of the contained Band was repurposed to query the SubMenu child of the band. This is probably not the way Windows does it, but it serves its purpose until a better method is found.

CBandSite and COM Aggregation

The CBandSite class of browseui is the class used by the taskbar as a container for the taskbar toolbars, including the toolbar the hosts the application window icons. The issue was not a lack of implementation, but the lack of support for a COM programming concept known as “Aggregation”, where one COM class can use another COM class’ interfaces as if they were their own. Supporting Aggregation requires some new classes in the ATL library of ReactOS.

ReactOS RShell

Previously, the ReactOS menu code (which was based on Wine’s) misused the MF_POPUP flag as an indicator of the menu having a submenu attached to it, while it’s clear from the MSDN description of the MENUITEMINFO structure that the hSubMenu member already has enough information: this member is NULL if the menu does NOT have a submenu.

The ancient method of adding items to a menu involved using AppendMenu and/or InsertMenu. These two functions use the MF_POPUP flag to indicate that the ID parameter will not be an id, and instead will be repurposed as a submenu handle. In modern versions of windows (Win2k and up), the InsertMenuItem function was introduced, together with some other related functions, which work with a MENUITEMINFO structure. In this structure, the MF_POPUP flag is not used anywhere, and it has separate fields for the id (wID), and the submenu handle (hSubMenu). Because of the wrong assumption that the item ID was reused as the submenu handle, the resource loading code was wrongly setting the submenu handle in the wID field. Fixing those made the functions behave as expected, both in how they store the popup information, and how they build the menus when they are read from an application’s resource data.

Tests confirm that Windows does return the same pointer in SHGetImageList as in Shell_GetImageLists, which is known to return the global, unsafe, image list pointers.

Focus Manager and Capture System

For navigation to work correctly, even with the keyboard, the parent menu has to be aware of when the child menu is opening/closing, not just when it was created.

Improving the behavior of hot-tracking across different level menus required adding a new feature to the focus manager, where it also Captures the input, so that using the keyboard prevents the mouse from resetting the hot item, while still allowing mouse movements from resuming the mouse-based hot-tracking.

Gigaherz had to improve the way the shell menus work without activating their windows. This meant that the title bar of the parent window does not go grey when a shell menu is displayed. But when the logic that calculates when and how to cancel the menu relies on activation events, suddenly that system stops working, since the windows are never activating.

Redesigning the input captures system make it track mouse events and manages what receives those events. The Focus Manager class now truly manages the focus of the menu, arbitrating what receives the input at any given time. But alas, it can never be that simple.

This makes it possible to switch between menus in the horizontal menubar, simply by moving the mouse to a new item. A dual hook system was implemented, where shell menus use a GetMessage hook, and system popups use a MessageFilter hook. Moving the mouse to a new item causes the old item to be cancelled, and the new item to be activated. Or that was the theory. Because of the complex flow of messages and method calls going between the different windows and classes, once a shell menu is open, moving to a system popup behaves wrongly.

It seemed that all the failed attempts at fixing tracking bugs were always traced back to the same place: TBN_HOTITEMCHANGE. Then Gigaherz asked, "Wouldn’t the toolbar use the existing TrackMouseEvent feature as a means to decide when the mouse moves away from its rectangle?" If so, it would mean the toolbar would receive a WM_MOUSELEAVE event, which could be used instead of the captures.

Horizontal keyboard navigation needs to achieve 2 goals. The first is to be able to handle moving left and right while a system-managed popup is open, and the second is to handle left and right while a shell menu is open. The former required modifying the already existing message filter of the focus manager to handle the keys in a similar way. Some logic was needed to decide if a submenu was open, and if the current item was supposed to open a submenu, to decide if pressing left or right would cause the active menu to change.

It seems that Windows uses a flag in the Popup method of the deskbar in an Exec call to the band to tell a submenu to open with an item already selected.

Gripper Option

Alignment differences in the toolbar can be traced to a missing flag that is misleadingly named “ALWAYSGRIPPER.” When it IS set, it fixes the margin, while keeping it unset, makes the margin not adapt to the absence of a gripper.

COM(ATL) Classes and the Taskbar

Windows has two functions related to filling the addressbar combobox: one of them resets the contents and fills them from scratch, and the other only adds/changes/removes items related to the navigation.

Huw had been working on ATL Aggregation support, and he managed to get the taskbar to work with his changes!

The Win32K module is involved in taskbar operation.

Reusing the Same Explorer Instance

A DDE handler is needed for the explorer so that opening a new folder can use the existing explorer instance. Of course, a workaround is to disable the version of the desktop window from RShell and use the existing one from shell32 instead, the shell takes a shortcut and runs the window directly on its own instance. A proper explorer implementation can detect resolution changes and reposition itself afterwards.


Accelerators are the feature that makes hotkeys work) to work. They might still differ slightly from how Windows does it.


One things examined was the menubar, and how the CDefView and ShellBrowser interact. The ReactOS version was initially recreating the whole menu every time the user clicked on the window. Now there is temporary code that empties the menu, and refills it. This fixes multiple issues with the menubar in the shell browser window, which is now working more reliably.

Shutdown Dialog

You would think that you could call the shutdown dialog from the shell32 function. However, the code in msgina isn't designed to be called externally. So shell32 might require its own copy of that code.


USER handles are handles to objects created by user32, such as windows, menus and dialogs. There are two other types of handles: Kernel handles, which represent files, registry keys, and other kernel32-provided features, and GDI handles, which represent objects such as bitmaps, fonts, brushes, pens, and everything else provided by gdi32. The leaks that were in the explorer appeared to be mostly related to HMENU handles (USER). You may wish to use Deleaker (commercial) if you need to troubleshoot handle leaks or memory allocations. However, Gigaherz discovered at the time he used it, that it might not install correctly in ReactOS. But since he was testing the code in Windows, Deleaker was still able to help. The leak problems highlighted the importance of cleaning up after any windows are closed. You don't want the whole filebrowser class left hanging there for eternity, with all the child components also hanging from it.

When destroying objects, you have to track how many other references were keeping them alive, and the shell classes have to handle the cleanup. A lot of the explorer-related classes had (and many still do) circular references, where one object keeps another alive, and this other object also keeps the first alive.

Help and Support

There is the start menu “Help and Support” button. In Windows, this button opens the support program, which has help topics, troubleshooting, and support links. Since we don’t have such a program in ReactOS, Gigaherz decided to implement the button as a ShellExecute action. That code was written so that it should be extremely easy to change the behaviour in the future, or even make it configurable from the registry.

Special Paths with GUID Identifiers

The ShellExecute function needs to know how to handle “special paths” using GUID identifiers instead of standard filenames. Those identifiers are needed to open special folders such as My Computer, or Control Panel.

File Browser Toolbar

The buttons in the file browser toolbar can be subdivided into 4 groups:

  1. The navigation buttons, with history back/forward and “up”,
  2. The sidebar buttons, with “folders” and “search”,
  3. The file operations, with “copy to”, “move to”, “delete” and “undo”, and lastly
  4. The view mode, which shows a dropdown to select the listview style.

The third group contains buttons that relate to the selected items, and have equivalent actions to menu items. These buttons are usually implemented by using the same “command IDs” as the menus, allowing the code to handle them indifferently. Tracing the creation of the toolbar buttons confirmed that this was the case, so any items not working has to be a problem with the shell view object, which is the one that handles the menu commands related to the folder and its contents. This is handled by the shell32 classes.

Merged Folders and Undocumented Interfaces

The CMergedFolder class structure is almost completely undocumented, and it likely has multiple features that don’t really fit together. Among them, the primary purpose of the merged folder is the ability to aggregate the contents of multiple shell folders into one virtual folder that includes items from all of the sources, and also merges the child items of the same name into one virtual item. This allows the start menu to show the items from both the common start menu, and the user start menu.

But the start menu also appears to use a different feature of these merged folders, where it “removes” the Programs item from the list, and takes the contents of the merged Programs to use as the contents of the Programs static menu item (the one that appears below the separator).

Ideally, the implementation should replicate all the interfaces used by Windows, but we don’t really need such a complex implementation right now. Our implementation of the CMergedFolder class exposes the essential interfaces of a shell folder (that is, IShellFolder and IShellFolder2), and allows the start menu to show the combined view as expected by the users. If (or when) some program decides to use these undocumented interfaces, then we’ll know it’s necessary to implement them, and we’ll have the justification of working on it. For now, having the placeholder for the better implementation feels like the better choice.

Our implementation takes the items from the two sources, sorted by the default sorting algorithm, and adds them to a common list. If two items are equal, then the info comes from one of them, and the item is marked as shared, so that later we can tell when to use a merged folder for the contents, if the item happens to be a folder.

While debugging the merged folders, Gigaherz also tried to implement the “dragging” navigation on menus. That is, where you can press the button down over the menubar, then keep it pressed while you move to a menu item in a submenu, and finally release over the menu item to activate it. This required changing some of the logic of how to treat clicks, and disconnecting the click code from the WM_COMMAND message from the toolbar, since we don’t want to handle the clicking internally anymore.

In Windows, it’s the start menu objects that do the filtering: the CMenuSFToolbar object sends a callback to the CStartMenuCallback object, which returns either S_OK or S_FALSE depending on if it wants the item shown, or hidden. Implementing this callback was quick and painless, since we already had most of the supporting code written for other callbacks, and all that was needed was to add a condition to the item enumeration in the menu toolbar. With this working, it meant the start menu now properly shows the Programs folder in the Programs menu item, and both the top items and the programs folder (and any common folder such as Startup), now shows items from both the user start menu, and the common one.

Now that it was working, all that was left was make it more cross-compatible with Windows, which meant implementing the undocumented IAugmentedShellFolder interface. Luckily, the interface was simpler than expected, and the information of the methods in the virtual function table, enhanced by the info in the PDB files, allowed for giving those methods a name, and their parameters a type.

Shell Service Objects

Shell service objects are just objects that implement IOleCommandTarget, and receive a specific command used to initialize and shutdown the objects. This was verified by testing the implementation in Windows.


Achieve some rudimentary tooltip support meant changing the way mouse input was handled, so that the code could track when the mouse is over a tray icon. Windows shows the tooltips centered on the icon and above it, while our implementation so far shows them at the cursor.

Notification Icons

There is a big misconception that the icons next to the clock, in the taskbar, are called “tray icons” or similar. This is a mistake that originated in the young times of Windows 95 development, where the taskbar was instead a tray. That is, a folder docked to the edge of the screen, which the user could open and close as needed. Because of that, they came to be known as “tray icons”. Their proper name, for future reference, is “notification icon”. But many still prefer to call them tray icons instead. Nearly 20 years of habit are hard to change. Anyway, as you can see, the class name still follows the ancient naming, which it has inherited across all versions of windows, since in those times it was still a separate program that would run at startup.

As a Shell Service Object, the CSysTray class is only expected to have one interface implemented: IOleCommandTarget. Of this interface, only one method is used by SSOs. When it is called in this context, the Exec method can have two values in the command ID: 2 and 3, which correspond to “new”, and “save” in the ole terminology. These two commands are repurposed to mean “init” and “shutdown” (respectively) by the SSO manager.

Inside CSysTray, there is an “icon manager”, which would run a thread with an hidden window, used as a target for the messages sent by the icons. It would also take care of a list of icon handlers, and send them notifications about the state of the icons, while the handlers could call the methods to add, modify, or remove icons.

Volume Icon

The Volume icon handler would receive those calls.

The CSysTray handler would notify the handlers periodically, letting them refresh the state of the icons. For now, there's a hardcoded interval of 2000 milliseconds (2 seconds).

Looking at the imports and pdb function lists from the windows DLL, one could guess that it was using the winmm API to obtain the mixer controls. Tracing the usage of these functions, the primary usage was to obtain the MUTE control, and use it to choose which icon to display. The popup with the volume is handled by a different set of functions. So then it is a matter of replicating the windows behavior by obtaining the default waveOut device, obtaining the mixer id from the device id, then obtaining the line control id corresponding to the MUTE control type. This control id would then later be queried to obtain the mute status, through the update function called by the timer.

Sadly, the equivalent winmm code in ReactOS has some missing features, which means that obtaining the default waveOut device, isn’t working. Given that we have no waveOut device to obtain the mixer id from, there's a fallback case where the first available mixer is used, even if it could be the wrong one.

File Browser Menus

Most of the actual work in managing the file browser menus is done in the WM_INITMENUPOPUP message handler. This message is initially received by the shell browser object, which does its own processing first, and then forwards it to the shell view. The message would normally contain the index of the menu item that opened the submenu that is being initialized. Because the shell uses custom menus based on toolbars, the submenus don’t know the index of the item that opened them, so the shell browser has to check which menu is opening, and then set the appropriate index in the right parameter, before forwarding.

Both the edit menu and the view menu are initialized only once, when the view activates for the first time.

The next step was to implement the File menu. Unlike the edit and view menus, which are only initialized once, the file menu is cleaned up and the items of the current selection are added to it every time the menu is shown. For this, Windows has a special function that gathers the available actions for the current selection, and builds a special menu for those items. If we don’t have this function yet, then the code just obtains the items of the context menu, and adds them to the File menu as-is. This may need future improvements, but serves its purpose for now.

The favorites menu now uses the CMergedFolder.

Since Windows keeps the selection shown on windows (but not the desktop), there is a flag that is only set when not in desktop mode.