Testman Blacklist

Here you can discuss ReactOS related topics.

Moderator: Moderator Team

Post Reply
dark
Posts: 275
Joined: Wed Apr 06, 2005 9:40 pm

Testman Blacklist

Post by dark » Sat Feb 18, 2012 8:01 pm

Is there anything testers can do to help resolve the erratic behavior of the blacklisted tests in testman?

wildschwein
Posts: 413
Joined: Tue Sep 16, 2008 1:13 pm

Re: Testman Blacklist

Post by wildschwein » Sat Feb 18, 2012 10:23 pm

please explain, and provide some links, that testers can understand the problem. thanks !

vicmarcal
Test Team
Posts: 2732
Joined: Mon Jul 07, 2008 12:35 pm

Re: Testman Blacklist

Post by vicmarcal » Sat Feb 18, 2012 10:43 pm

dark wrote:Is there anything testers can do to help resolve the erratic behavior of the blacklisted tests in testman?
Erratic behaviors are related to potential race conditions,memory bugs,etc...
They are difficult to catch, as they are inherent to the operating system so there isnt a clear way about how testers can help in this case.
You can help with CRASHED tests, reporting why they are crashing(run that particular winetest and obtaining a debuglog).Please check they haven't been reported yet :)
Image

dark
Posts: 275
Joined: Wed Apr 06, 2005 9:40 pm

Re: Testman Blacklist

Post by dark » Sat Feb 18, 2012 11:15 pm

wildschwein wrote:please explain, and provide some links, that testers can understand the problem. thanks !
There's a handful of tests on reactos.org/testman that give random results and crashes so much that they are no longer included in the totals. They are marked by a black outline now.

If we can only help on crashes, then it appears we would have to monitor revisions in testman for crashes, download dbgwin build from /getbuilds, run it to try to cause crash, then get debug log and create or add to bug report.

edit: and here's a wiki page with some relevant information: http://www.reactos.org/wiki/Winetest_crashes

Tried this: can get random results, no crashes. thinking crash is merely a function of memory leaking and would need to know how testman runs scripts (as batch file?) and then do the same when testing.

preston
Developer
Posts: 272
Joined: Wed Sep 27, 2006 8:19 pm
Location: Prague, Czech Republic

Re: Testman Blacklist

Post by preston » Sun Feb 19, 2012 3:54 am

dark wrote:Is there anything testers can do to help resolve the erratic behavior of the blacklisted tests in testman?
We will eventually make bug reports for all flaky tests and testers can then check if the erratic behavior is caused by a particular revision and report back. But I'm afraid some of those tests are random "by design". Wine has a similar blacklist of misbehaving tests too by the way.

Random crashes are usually caused by underlaying problems which may be very hard to find. "Random" is the keyword here. When assembling the initial black list I tried to replicate the issues myself without any success (using the same test cd iso as the test bot).
dark wrote:Tried this: can get random results, no crashes. thinking crash is merely a function of memory leaking and would need to know how testman runs scripts (as batch file?) and then do the same when testing.
The tests are ran by an app called rosautotest, you can find it in rostests folder in the main repo.

Post Reply

Who is online

Users browsing this forum: DotBot [Crawler] and 40 guests