This is a detailed summary of the work I've performed over the course of Google Summer of Code 2017. This was a continuation of my GSoC project from last year.
Code Submitted
-
You can check out my branch with SVN at https://svn.reactos.org/reactos/branches/GSoC_2016/NTFS
You can find a list of commits made to my branch during GSoC 2017 at this link.
I also worked on this project after GSoC last year and before the coding period officially began this year. You can find that list of commits here.
For completeness-sake, here's a link to all the commits made to my branch ever.
- I've also written several stand-alone programs to assist with my work. The source to them is available here.
Highlights
-
- Continued last years' project to implement write-support on NTFS volumes within ReactOS.
- Disabled write support by default to prevent data loss. Write-support can now be enabled via the registry.
- Added support for file creation.
- Added support for folder creation.
- Confirmed that Windows sees all files and folders created in ReactOS. Tested with up to 700 files (present limit of tester).
- Confirmed that all files created in ReactOS have proper contents when read in Windows.
- Confirmed that chkdsk won't report any problems after creating files and folders in ReactOS.
- Expanded stand-alone utilities to assist with this years' project.
- Identified and repaired numerous bugs.
How to Observe Work Performed
In order to see the results of my labor, you must first build my branch with the usual method and setup a virtual machine with the output. You can use a bootcd or a livecd. If you have any trouble building ReactOS, you can ask for help on IRC.
Before you build ReactOS, you must enable NTFS write-support if you want to test it. This is because write-support is still experimental; there's still a lot of things that could go wrong and using this driver could result in catostrophic data loss.
To enable write support, open the folder you checked out my branch to, and navigate to boot\bootdata\ and open the hivesys.inf file. Find the section that starts with ; NTFS filesystem driver
Underneath that, find the line that says:
;HKLM,"SYSTEM\CurrentControlSet\Services\Ntfs","MyDataDoesNotMatterSoEnableExperimentalWriteSupportForEveryNTFSVolume",0x00010001,0x00000001
Simply remove the semicolon at the start of that line and save the file and build ReactOS as usual. This will add the registry key needed to enable write support.
Next, in a virtual machine (strongly recommended) boot into Windows and format a second virtual drive as an NTFS volume. Then, use the volume you formatted as a second hard drive in a virtual machine running ReactOS.
What's Working & What Can Be Used Now
Creating a file or folder and overwriting a file have been debugged pretty thoroughly in my branch and should work as you'd expect in all but the rarest of cases.
-
You can create files from the command line, by redirecting standard outpout.
-
Or, you can create a file by right-clicking. Renaming a file isn't supported yet, so if you create a file or folder via right-clicking, you'll have to accept the default name.
-
Also, you can use my test program to create files.
-
Creating a folder works as well. This was a feature I was able to implement at the last minute.
Reboot into the Windows vm and you should notice that all the files you created in ReactOS are available to you. If you run "chdsk x: /f" from a command line, where x is the letter of the ntfs drive you wrote files to, chkdsk should not report that it found any errors.
There's also some features which I have implemented or fixed which are less-obvious to an end-user. These include:
-
- Support for enlarging the master file table when necessary.
- Support for updating the $MFTMirr file.
- Fixed support for POSIX vs Windows naming conventions. See CORE-13700.
Bonus Code
During the course of this activity, I also expanded the stand-alone programs and scripts I wrote last year. Only the minimal amount of time was spent writing these programs, so I wasn't distracted from writing driver code. As such, this bonus code has numerous errors: memory is often leaked; file handles are leaked; comments are missing, deceptive, or just plain wrong; things are done in an inconsistent manner; etc. For the most part, these programs can't be interacted with except at the level of their source code.
-
Tester (AKA "Disk_Test_1")
This automated-tester was absolutely crucial to my progress. Not only did it consistently find errors that would have been impossible to reproduce otherwise, it was the only way I could test many of my incremental improvements to write support. This year I added tests specifically for creating a large number of files and verifying their creation.
-
"NTFSbrowse"
I developed this program as a learning tool for myself, but it has become a powerful debugging tool as well as a learning aid. The program will give you a lot of information about a particular file record, identifying most of the individual fields in its HTML output. This year I added an interface and a detailed breakdown of the $INDEX_ROOT attribute, as well as dumping the $I30 index allocation to another file which is linked to, and a detailed breakdown of the index records within the index allocation.
-
"DumpDrive"
This program is designed to create images of an NTFS partition, then compare these images. If it's able, the tool will keep track of every file that's changed, and display information related to the change. I gave it an interactive interface this summer and it was invaluable in helping me to track down some things that were causing chkdsk to complain.
-
"BuildRosFast.ahk"
I wrote this AutoHotKey script last year and improved it a little this summer. I have it assigned it to a custom toolbar button in Visual Studio. When I click this button, it starts building a livecd of ReactOS, kills any sessions of WindBg, reboots a VM, Launches WindBg, Launches ReactOS with debugging enabled, and finally adds whatever breakpoints I'm interested in before clicking the two buttons prompted by a livecd before you enter the desktop. This script is pretty crude (it would be better if it paid attention to the build results) but at the same time, it's an extremely powerful time saver!
What's left to be done
-
There's still considerable work to be done before the driver is ready for the day-to-day use. Immediate goals include:
- Deleting files and directories
- Renaming files and directories
- Caching
- Page-file support
- Support for async IO
- Support for memory mapped files
-
Once these features are working (maybe sooner), I'd like to see ReactOS booting from NTFS! Wouldn't we all? That will require:
- Adding support for formatting a volume
- Updating setup to use NTFS
- Creating a bootloader for NTFS and using / fixing freeldr's NTFS support
- There might be something else I've forgotten / am not aware of
-
Furthermore, there is some support for these NTFS-specific features, but they aren't fully working right now:
- Sparse files
- Alternate Data Streams (Named Streams)
- File records that have or require an $ATTRIBUTE_LIST attribute
-
There's other features of NTFS that are yet to be supported:
- Support for drives and files with greater-than 2^32 clusters
- Compressed files
- Encrypted files
- Logging
-
It should also go without saying that there will be lots of bugs to fix along the way.
Looking Forward
Thank you to my mentor Pierre, ReactOS, and Google for this wonderful opportunity!!
I definitely plan to keep contributing to ReactOS! I will make one more blog post after this one to summarize my experience from a personal standpoint.
