Would it be possible to benchmark the Win32 API?

Here you can discuss ReactOS related topics.

Moderator: Moderator Team

BrentNewland
Posts: 175
Joined: Wed Oct 05, 2011 7:32 am

Would it be possible to benchmark the Win32 API?

Post by BrentNewland » Thu Mar 08, 2012 6:05 am

This is just brainstorming, I think it will be a while before the majority of ReactOS development moves from implementation to optimization, so I don't want anyone to think I'm asking for anything besides ideas

I was browsing the Wine wiki and came across WineTest, which runs all the Wine API compliance tests at once and uploads them to the Wine servers.

This led me to wonder: Since the tests can check the functionality of every API component, couldn't they also test the execution time for every component? In effect, benchmark the Win32 API?

It is an intriguing idea. Would it be useful to compare the speed of the Windows API to ReactOS? Or ReactOS to Wine? Would it be able to catch any bugs that might not normally be noticed (such as a piece of code suddenly taking 10 times longer to execute)?

And, I think this is what I would like to find out the most, are there many open source projects which benchmark themselves like that? And are there projects besides Wine and ReactOS that perform tests like the Wine/ReactOS tests do?
Last edited by BrentNewland on Thu Mar 08, 2012 7:56 am, edited 1 time in total.

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: Would it be possible to benchmark the Win32 API?

Post by vicmarcal » Thu Mar 08, 2012 7:39 am

Hi BrentNewLand!
For now we are pleased just passing the WinAPI tests(and there is a lot of work to be done, really).
Iirc, we did some benchmarking from GDI tools and we run them in Arwinss, ReactOS and Windows. Iirc ReactOS was performing better than Windows. But this was some years ago and now ReactOS is bigger and the MM has suffered big changes(which will impact in these benchmarks for sure).
Most of the projects which implements APIs (as Python or QT) have their own API tests/regression tests in order to catch potential regressions or bugs.

:)
Image

mrugiero
Posts: 482
Joined: Sun Feb 14, 2010 9:12 am

Re: Would it be possible to benchmark the Win32 API?

Post by mrugiero » Thu Mar 08, 2012 11:08 am

vicmarcal wrote:Hi BrentNewLand!
For now we are pleased just passing the WinAPI tests(and there is a lot of work to be done, really).
Iirc, we did some benchmarking from GDI tools and we run them in Arwinss, ReactOS and Windows. Iirc ReactOS was performing better than Windows. But this was some years ago and now ReactOS is bigger and the MM has suffered big changes(which will impact in these benchmarks for sure).
Most of the projects which implements APIs (as Python or QT) have their own API tests/regression tests in order to catch potential regressions or bugs.

:)
Vic, what Brent is asking is if a testing bot could automatically benchmark an API function by function. Also, the question about how many projects work that way, is about the API benchmarking.
Also, he seems to be aware that right now ReactOS is focused in implementing the API and conform to the specs, and he stated it's (by now) hypothetical.

BrentNewland
Posts: 175
Joined: Wed Oct 05, 2011 7:32 am

Re: Would it be possible to benchmark the Win32 API?

Post by BrentNewland » Thu Mar 08, 2012 5:51 pm

Yes, unfortunately when he posted I had forgotten to make that bit at the beginning of my post larger.

I was thinking of getting in touch with the people who work on WineTests, to see what their opinion is on the feasibility or desirability of doing this. Not sure if I should post to the mailing list, or check their commit logs and email the authors, or file a bug report with them... any suggestions?

mrugiero
Posts: 482
Joined: Sun Feb 14, 2010 9:12 am

Re: Would it be possible to benchmark the Win32 API?

Post by mrugiero » Thu Mar 08, 2012 6:46 pm

BrentNewland wrote:Yes, unfortunately when he posted I had forgotten to make that bit at the beginning of my post larger.

I was thinking of getting in touch with the people who work on WineTests, to see what their opinion is on the feasibility or desirability of doing this. Not sure if I should post to the mailing list, or check their commit logs and email the authors, or file a bug report with them... any suggestions?
It depends. If there is a feature request section in the bug report system, I think you should post it that way. Make sure of making your idea clear enough. Otherwise, the mailing list is, IMO, the best idea. Also, if you want feedback (as if you want to make sure that if they don't want to take your idea, you'll know why), the latter is clearly the best idea.

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: Would it be possible to benchmark the Win32 API?

Post by vicmarcal » Thu Mar 08, 2012 7:18 pm

BrentNewland wrote:Yes, unfortunately when he posted I had forgotten to make that bit at the beginning of my post larger.

I was thinking of getting in touch with the people who work on WineTests, to see what their opinion is on the feasibility or desirability of doing this. Not sure if I should post to the mailing list, or check their commit logs and email the authors, or file a bug report with them... any suggestions?
Well, about "Time benchmarking" there are some problems:
1)In order to compare ReactOS vs Windows or Wine+Linux vs Windows, you have to run a Release version (Debug lasts more as it has to output some code).
2)Wine is not an emulator but it is an extra layer so most of the times it will perform worse than Windows. So I doubt they will be adding a feature just to show they are slow in a lot of areas. Do you imagine ReactOS time-benchmarking better than Wine? ;)
3)PCs are incredible fast right now, so even if benchmarking is a nice informative tool it just matters at high-plumby-stressing apps. Apps that Wine neither ReactOS is able to run nowadays.

Anyway, as always: Try!Who knows?? :) Just a Tip: Don't tell them you were here asking about it...
Image

BrentNewland
Posts: 175
Joined: Wed Oct 05, 2011 7:32 am

Re: Would it be possible to benchmark the Win32 API?

Post by BrentNewland » Thu Mar 08, 2012 9:34 pm

vicmarcal wrote: Well, about "Time benchmarking" there are some problems:
1)In order to compare ReactOS vs Windows or Wine+Linux vs Windows, you have to run a Release version (Debug lasts more as it has to output some code
2)Wine is not an emulator but it is an extra layer so most of the times it will perform worse than Windows. So I doubt they will be adding a feature just to show they are slow in a lot of areas. Do you imagine ReactOS time-benchmarking better than Wine? ;)
3)PCs are incredible fast right now, so even if benchmarking is a nice informative tool it just matters at high-plumby-stressing apps. Apps that Wine neither ReactOS is able to run nowadays.

Anyway, as always: Try!Who knows?? :) Just a Tip: Don't tell them you were here asking about it...
It would probably be most useful as a historical stat and optimization tool inside one project. You would have to run the tests on an automated test machine (because testing a bunch of random computers woud horribly skew the results). Probably several machines, then you could establish a performance baseline difference for each machine. As long as you're not comparing debug and general releases, it should be fine.

As for the computers being fast, Windows seems to often have really bad response times. I think everyone should run Process Monitor at least once - just to see how much Windows does when it's sitting there (tens of thousands of operations per second). Optimizing all the functions as much as possible would go a long ways towards general speed and responsiveness.

Who knows, perhaps someday Wine and ReactOS will be faster than Windows.

Z98
Release Engineer
Posts: 3379
Joined: Tue May 02, 2006 8:16 pm
Contact:

Re: Would it be possible to benchmark the Win32 API?

Post by Z98 » Thu Mar 08, 2012 9:47 pm

This has been done in the past. The results ended up showing ROS drew faster than the vanilla Windows drawing functions but was significantly slower if compared with hardware accelerated drawing (not surprising).

FlyingIsFun1217
Posts: 475
Joined: Sun Jun 10, 2007 3:56 am

Re: Would it be possible to benchmark the Win32 API?

Post by FlyingIsFun1217 » Fri Mar 09, 2012 12:04 am

Z98 wrote:This has been done in the past. The results ended up showing ROS drew faster than the vanilla Windows drawing functions but was significantly slower if compared with hardware accelerated drawing (not surprising).
Which is all a bit irrelevant when people, thinking they'll use the faster operating system, find out that there isn't 100% compatibility (i.e., users who think their apps will run faster, but end up not even working).

FlyingIsFun1217

mrugiero
Posts: 482
Joined: Sun Feb 14, 2010 9:12 am

Re: Would it be possible to benchmark the Win32 API?

Post by mrugiero » Fri Mar 09, 2012 8:11 am

vicmarcal wrote:
BrentNewland wrote:2)Wine is not an emulator but it is an extra layer so most of the times it will perform worse than Windows. So I doubt they will be adding a feature just to show they are slow in a lot of areas. Do you imagine ReactOS time-benchmarking better than Wine? ;)
3)PCs are incredible fast right now, so even if benchmarking is a nice informative tool it just matters at high-plumby-stressing apps. Apps that Wine neither ReactOS is able to run nowadays.
About the point 2, a little correction. It would perform even a little worse if it were emulated. But yes, it's likely slower than Windows, since it adds a few layers in between which Windows uses with direct contact with the kernel.

About the last point, I disagree. A benchmark is always useful (with a near complete product, of course). Even if it's an app which does its work in twenty seconds, if you get that to a half it's useful.
And you're also forgetting most games, lots of which runs on WINE (but I think they don't run in ReactOS yet, because they require access to the graphics card and the WINE drivers can't be ported, as they act as wrappers to *NIX drivers), there, performance matters. A good experience and a lame experience with a good game, depends just on the performance.
Of course, right now the priority should not be in performance, but in compliance.

fred02
Posts: 551
Joined: Thu Nov 22, 2007 5:54 pm

Re: Would it be possible to benchmark the Win32 API?

Post by fred02 » Fri Mar 09, 2012 3:50 pm

vicmarcal wrote:3)PCs are incredible fast right now, so even if benchmarking is a nice informative tool it just matters at high-plumby-stressing apps. Apps that Wine neither ReactOS is able to run nowadays.
Well, let not forget about all these netbooks and tablets. They are no faster than 10 years old PCs, so performance will matter.
Another point is battery life. The less cycles CPU does, the less battery it uses, regardless its speed. Granted, CPU is no longueur the most power hungry component, but it still helps.

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: Would it be possible to benchmark the Win32 API?

Post by vicmarcal » Fri Mar 09, 2012 5:26 pm

mrugiero wrote:
vicmarcal wrote:
BrentNewland wrote:2)Wine is not an emulator but it is an extra layer so most of the times it will perform worse than Windows. So I doubt they will be adding a feature just to show they are slow in a lot of areas. Do you imagine ReactOS time-benchmarking better than Wine? ;)
3)PCs are incredible fast right now, so even if benchmarking is a nice informative tool it just matters at high-plumby-stressing apps. Apps that Wine neither ReactOS is able to run nowadays.
About the point 2, a little correction. It would perform even a little worse if it were emulated. But yes, it's likely slower than Windows, since it adds a few layers in between which Windows uses with direct contact with the kernel.
Yes,sure. I really meant "Even if Wine is not an emulator its extra layers will force it to perform worse". Implying that being an emulator is even worse.
Bad wording maybe :)
mrugiero wrote:About the last point, I disagree. A benchmark is always useful (with a near complete product, of course). Even if it's an app which does its work in twenty seconds, if you get that to a half it's useful.
A "work" which takes about 20 seconds to be finished is a "stressing" task really.Performance is of course something important, but I doubt that nowadays you will feel the differences in a single call thanks to the actual fast processors.
ReactOS is able to run in just 64MB RAM so it shows the big performance in this area. ReactOS behaves really well inside a Virtualizer(and as you may know VM slows down guest OSes performance).
I don't feel any problems in performance really(even some GDI test apps have shown better performance than Windows, as Z said) but in the medium term..who knows if we will need a "performance measure"!
Image

mrugiero
Posts: 482
Joined: Sun Feb 14, 2010 9:12 am

Re: Would it be possible to benchmark the Win32 API?

Post by mrugiero » Fri Mar 09, 2012 10:11 pm

vicmarcal wrote: A "work" which takes about 20 seconds to be finished is a "stressing" task really.Performance is of course something important, but I doubt that nowadays you will feel the differences in a single call thanks to the actual fast processors.
ReactOS is able to run in just 64MB RAM so it shows the big performance in this area. ReactOS behaves really well inside a Virtualizer(and as you may know VM slows down guest OSes performance).
I don't feel any problems in performance really(even some GDI test apps have shown better performance than Windows, as Z said) but in the medium term..who knows if we will need a "performance measure"!
I agree with you, there doesn't seem to be performance problems currently.
Anyway, the idea got me curious :P
Also, the proposal to include it in WINE tests makes it a lot better, since I do notice WINE + Linux a lot slower than running on Windows in several cases.

PascalDragon
Posts: 123
Joined: Wed Aug 04, 2010 7:34 pm

Re: Would it be possible to benchmark the Win32 API?

Post by PascalDragon » Sat Mar 10, 2012 10:44 am

vicmarcal wrote:A "work" which takes about 20 seconds to be finished is a "stressing" task really.Performance is of course something important, but I doubt that nowadays you will feel the differences in a single call thanks to the actual fast processors.
It's not a single call that makes a difference but their accumulation.

Recent example from my work: We wanted to display a grid with one column that contained around 130000 records, but filling the grid and displaying it took around 5 minutes. In the end I was able to cut it down to 6 seconds (without loss of functionality). While a few factors were involved in this one particular one was a chain of filtering event handlers that basically did nothing besides calling other methods that called other methods that just returned True. So I had a stack of around five or six methods that were called for each record when inserting it into the underlying dataset. After I disabled the filtering the time needed went down from 2-3 minutes to 45 seconds (this was one intermediate observation, so I had already improved performance).

So calls still DO matter today in case of performance, but not a single call, but the sum of many of them.

Regards,
Sven
Free Pascal compiler developer

Smiley
Developer
Posts: 156
Joined: Fri Nov 10, 2006 3:36 pm

Re: Would it be possible to benchmark the Win32 API?

Post by Smiley » Sun Mar 11, 2012 8:07 pm

BrentNewland wrote: And, I think this is what I would like to find out the most, are there many open source projects which benchmark themselves like that? And are there projects besides Wine and ReactOS that perform tests like the Wine/ReactOS tests do?
Generally speaking every programmer should also write test for his code.
A great example for the testing process is firefox. For every few revisions, firefox is built and several tests are run in it including standards compiance tests, crash tests and performance tests on different platforms: https://tbpl.mozilla.org/

Post Reply

Who is online

Users browsing this forum: DotBot [Crawler] and 3 guests