ReactOS and fragmentation

[PurpleGurl]

Also, if you don't want the file system to fragment, then part of that responsibility is on the OS. The less the OS relies on temporary files, the less it moves files around, the less fragmentation there will be.

The OS can help by providing an easy way to disable ALL logging behavior for those who don't need that functionality. The same for performance counters. While that capability needs to exist for full compatibility with Windows, Reactos could be written with a way to disable it if not needed, and written to not rely upon it. I would even put the event viewer in that category. Give the option to completely disable it when someone has things finely tuned, but still use it for compatibility reasons and as a troubleshooting resource. Just give options to disable non-essential disk writes. And for essential ones, make the writes smarter as to when the files are written, and where.

The system registry can benefit from memory caching. When you have poorly written programs hammering the registry, then a write cache would be good. Sure, that could increase corruption risk if you lose power before updating a key, but there could be less writes. Magic Jack's software, for instance, constantly reads and writes to the registry (like all the same keys every 3 seconds). Same key writes can be eliminated by the OS if the values are the same every time. It could remember it put that value there before recently and sign off on the access as complete without writing anything.

And there are other strategies an OS can use, like reserving space and disk areas at the OS level. Disk calls can refuse to write to certain parts of the drive unless conditions are met. Like requiring the first so many megabytes to be for system critical files (assuming traditional hard drives, SSD devices make the fragmentation problem irrelevant since there is no seeking and all reads are the same) and making it impossible to write other files there. The OS could put files destined to be big near the end of the partition. The OS can use a private set of reservation lists to determine what types and sizes of files go where, even if the FS doesn't have such native capabilities. It would be more "informal" and placement decisions made at the time of space allocation based on decisions made earlier (like during installation or the running of a special tuning utility).

If anticipated sizes can be given to calls, the OS can put the files in parts of the drive which have at least that much available size. Under MS-DOS and early Windows versions, sectors were always given to new files starting from the first empty space. But if a number of small files were deleted, then a new huge file would fragment into all their old locations. But DOS and Windows got wiser and added abilities to tell it the projected size and the file creation calls would choose areas less likely to cause fragmentation.

And we could expand on the Process Idle Tasks thread and include some defragmentation during idle system time (on non-SSD volumes). We could make it to where the OS is aware when it fragments a file, so it can add it to the list of files to spot defragment.

There are lots of ways an OS can be write files more intelligently and responsibly to reduce fragmentation, regardless of the file system used. And for SSD devices, while they don't fragment, they would benefit from less writes, since writing to them is slower, and there is a limited number of writes per cell. So less unnecessary writes would extend their lives. Speaking of SSD drives, in time, we could set things up to where files that seldom change can be read from an SSD drive, but files frequently changed get put on a standard hard drive. The SSD and the regular drive could be read at the same time, helping with performance, and the SSD could help during multiple file seeks since seek time is irrelevant (and no heads to move around, so switching between 2 or more files does not incur seek penalties). Windows 7 is somewhat like this.

At any rate, all proposed strategies above can be implemented without breaking compatibility.

[sh4ring4n]

... Or you can auto-defrag every weed

[Haos]

Write cache for registry is a potential disaster, as you duly noted. Decreasing the number of writes is just decreasing the chance of bad thing (tm) to happen, but wont prevent it. Hive corruption is at the present moment one of the few ways to brick the windows install beyond repair, should be then treated with extra caution.

[PurpleGurl]

Oh, and it seems like a lot of fragmentation under XP is caused by System Restore and any web browsers.

A thought on registry fragmentation is why not initialize the hives to their maximum projected size and never allow any operation to truncate that size? So if you set the registry to like 50 MB and it never falls below that, then it couldn't get fragmented. Then again, that could increase internal fragmentation, since any database can be fragmented, not just a file system. Since the registry is basically a b-tree type structure, I believe data can be allocated anywhere within it and filled with holes as keys are added and deleted. So preventing external fragmentation of the registry within the file system doesn't keep it from fragmenting within itself. There is a way to defrag the keys inside the registry, and that is to basically create new hive files, copy them all from the original hives to the new hives, then reboot immediately and rename upon reboot. Most programs that do this call it "registry compaction." So when new hives are created, any gaps or holes inside the hives and any corrupt or nonindexed data are not copied.

Fixed size swap files are similar, and you can already set those up under Windows to where they won't fragment.

One fragmentation bug that Windows NT (all versions) seem to have is a bug in the API for defragmenting FAT32 folders. Most defraggers for NT won't touch such folders if they are fragmented. Fragmented directories can slow access to the files in them, but a bug in NT won't let you properly defragment them. Defraggers are all aware of this bug and won't touch them. One defragger will partially defrag them, bringing them to 2 fragments. That is certainly better than 6-8 fragments, but a less than ideal solution. Fixing that API and implementing it correctly would not break compatibility, since no 3rd party programs nor even the MS defragger use it. So whether that API is fixed, broken, or missing, nothing uses it, so there would be no harm implementing it correctly. Then if we include a defragger, it could fully defragment FAT32 volumes, not partially defrag it like the NT defragger does.

[PurpleGurl]

Write cache for registry is a potential disaster, as you duly noted. Decreasing the number of writes is just decreasing the chance of bad thing (tm) to happen, but wont prevent it. Hive corruption is at the present moment one of the few ways to brick the windows install beyond repair, should be then treated with extra caution.

True, but this is more of a pass-through cache, not a write back. My idea is merely to track the last so many writes (not cache as in hold before writing) so that identical writes won't be done over and over. If something sets a given registry key to a value like 1 every second when nothing has changed it, then wouldn't it be better to check it against memory to see if it was just written than to call in the drive hardware? I mean, there really is poorly written software and drivers out there which spams the same keys and values with the same information over and over to the point of harming performance and providing no benefit (if it posts it once, then posting the exact same thing a dozen more times changes nothing).

So I don't meant reducing the frequency of unique writes, just preventing redundant ones. Of course, very minor write-back cacheing could help, like writing every 10 reg writes or 10 seconds, whichever comes first, and flushing before allowing any additional program to start. And maybe something like if different values in the same key are being written, then allow a second between writes to see if another value in that same key is accessed, then write them all at once. But let me conclude this since FS fragmentation is the topic, and I apologize for getting OT.

[PurpleGurl]

... Or you can auto-defrag every weed

I think you mean every week. (Though if you were on weed, you wouldn't care if it was fragmented. ) It is true you can use software to do it weekly. But why not build it into the OS as an idle task? And if the OS knows it fragrmented a particular file (and it should, since it knows when there are breaks in the sectors as it writes them), it could do a spot defrag on it as soon as the PC is free. If there are any breaks in the sectors allocated to it, then it should be aware of that already. Sure, it could be possible to defrag on the fly in that case to defragment the file that is in the way of the one being written, but it would be a better use of time to let it fragment for the time being and then fix it during idle time. I mean, if it takes a second to write a fragmented file, but 35 seconds to make sure it is not fragmented, then let it fragment and flag it on a list that it needs defragmented (and it goes for writes to the list too).

People leave their PCs on for hours, and even days at at time, without actually using them, so why not use that time for system maintenance? And like I said, don't touch SSD volumes. They may appear fragmented logically since the file system tables are linear, but they are actual virtual devices and don't suffer the effects of fragmentation. Plus since they are solid state, their memory cells have limited life, and defragging them actually shortens their life while providing pretty much no discernible benefit.

[Z98]

Ahem. Do not double/triple post.

[PurpleGurl]

Ahem. Do not double/triple post.

Huh? I replied to 2 different posts, and added other stuff I forgot to include in a previous post. Those are 3 completely different posts with different points, not duplicates.

[zed260]

also @z98 you cant edit post on here no option to so you'd have to delete then make a new one to add to it

[Z98]

Edit: http://www.reactos.org/forum/viewtopic.php?f=2&t=8814

Our definition of double/triple posting includes posting of multiple different messages one after the other.

[PurpleGurl]

Edit: http://www.reactos.org/forum/viewtopic.php?f=2&t=8814

Our definition of double/triple posting includes posting of multiple different messages one after the other.

And what is wrong with 3 or more helpful posts if they are sincere and not spam? And so I am supposed to reply to separate aspects and different people in the same post?

And yes, editing works fine.

[zefklop]

Calm down, just before this get too much heated there.

Just to make things clear : Z98 wasn't rude with you, he just asked you to not double post.

Question #1

answer #1

Question #2

answer #2

This way to write posts seems very reasonable to me. This wasn't asked axplicitly, but that's what I'd expect in a forum where the rule is "one post at a time". And yes, there are other forums around there where this rule applies. You have never seen it before, now that you have, please do as you're told, there's no reason you should not be treated as everyone here.

Also, please consider that I don't enter in that "moderation" thing very often. You just got me pissed off because you insulted the whole community here because a moderator reminded you a simple rule here. He didn't ban you, nor threatened you, nor was even rude with you. He just didn't say please. Just consider him a smart ass if you want, nobody can stop you from thinking that.

By the way, now I consider you're a smart ass. Try stopping me from thinking that.

[Murmur]

Z98, you want to know what is more annoying than your "triple posting" is that you derailed the conversation. Now if they messaged a bunch of crap you can PM them and situation is fixed. Don't spam in a actual conversation please.

Now, back on topic.

[EmuandCo]

There was no triple posting. And you should watch your voice, before you get more than just a warning.

[nicamarvin2005]

Ahem. Do not double/triple post.

There was no triple posting.

Why so Rude...

where can I report rude people...