Can sb tell something about testman ?

Here you can discuss ReactOS related topics.

Moderator: Moderator Team

Post Reply
wildschwein
Posts: 415
Joined: Tue Sep 16, 2008 1:13 pm

Can sb tell something about testman ?

Post by wildschwein »

http://www.reactos.org/testman/

I heard about it, but can please sb explain for community in a few words what it is, and how can one use it, and what is the benefit of it in testing.

is testman new or old ?

User avatar
EmuandCo
Developer
Posts: 4402
Joined: Sun Nov 28, 2004 7:52 pm
Location: Germany, Bavaria, Steinfeld
Contact:

Re: Can sb tell something about testman ?

Post by EmuandCo »

Its a automatic test suite comparing winetests of every build to let us scan for regressions. I see no point for a normal user to work with it in any way.
ReactOS is still in alpha stage, meaning it is not feature-complete and is recommended only for evaluation and testing purposes.

wildschwein
Posts: 415
Joined: Tue Sep 16, 2008 1:13 pm

Re: Can sb tell something about testman ?

Post by wildschwein »

It is right, that when all fields are green, that means, ROS 99% kompatibel to Windows (1% undocumented funkctions )

But why is the "red total fail counter" in later revisions higher ?

in about 53xxx in was about 5000, and in the 55xxx from now it is about 10 000 !

Should't this amount got to zero, there higher the revision is ?

mattsharp
Posts: 9
Joined: Sun Jan 15, 2012 3:12 pm

Re: Can sb tell something about testman ?

Post by mattsharp »

I guess this is because some tests are dependent on other tests. now imagine 100 tests are dependent on one test that failed all the time and with the revision 53xxx you can make the one test work but 60 of the 100 dependent tests fail. the result is, that you are one big step further, but the log shows you 59 more errors.

milon
Posts: 969
Joined: Sat Sep 05, 2009 9:26 pm

Re: Can sb tell something about testman ?

Post by milon »

So, any tests that can't be performed get marked as a pass? Or they're just excluded? It would make more sense to me to mark any that can't be tested as an automatic fail.

Z98
Release Engineer
Posts: 3379
Joined: Tue May 02, 2006 8:16 pm
Contact:

Re: Can sb tell something about testman ?

Post by Z98 »

A test that is unrunnable and gets marked as an automatic fail is a false negative. If an API is outright missing or unimplemented, it is not a regression and is just noise when developers are trying to determine if their changes broke previously passed tests.

Smiley
Developer
Posts: 156
Joined: Fri Nov 10, 2006 3:36 pm

Re: Can sb tell something about testman ?

Post by Smiley »

wildschwein wrote:It is right, that when all fields are green, that means, ROS 99% kompatibel to Windows (1% undocumented funkctions )
I think that it would be much more correct to say that greem fields mean that no bugs have been found so far in this area... That means that if new tests are added they may reveal new bugs and more importantly, these tests do not test even 50% of the features of windows so it is very likely that there are sooooo many things that are not covered by these tests

What these tests are really useful is that they let us know if a commit has caused a very bad result. For example is someone breaks a very important part of the os, the problem will be reflected in the results in testbot.

dark
Posts: 278
Joined: Wed Apr 06, 2005 9:40 pm

Re: Can sb tell something about testman ?

Post by dark »

wildschwein wrote:It is right, that when all fields are green, that means, ROS 99% kompatibel to Windows (1% undocumented funkctions )

But why is the "red total fail counter" in later revisions higher ?

in about 53xxx in was about 5000, and in the 55xxx from now it is about 10 000 !

Should't this amount got to zero, there higher the revision is ?
Not really, and I don't think anyone has quite answered this question yet. The reason it never seems to get closer to zero and there are more total failed tests is that new tests are added to testman quite often and this makes such one to one comparisons of revisions so far apart difficult. If you look at r53100 and r55842 you see that r55842 has 278,131 more tests run on it than r53100 did, new test categories, and some categories with less tests that have probably been moved to a different category.

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: Can sb tell something about testman ?

Post by vicmarcal »

As dark says, from time to time we update the winetest suites. So each time new tests are being added and reveals new bugs :)

milon
Posts: 969
Joined: Sat Sep 05, 2009 9:26 pm

Re: Can sb tell something about testman ?

Post by milon »

Z98 wrote:A test that is unrunnable and gets marked as an automatic fail is a false negative. If an API is outright missing or unimplemented, it is not a regression and is just noise when developers are trying to determine if their changes broke previously passed tests.
Oh, oops. I missed the part about it being regression testing. Yeah, that makes sense.

Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot] and 1 guest