Regression in crypt32-cert winetest!

[Lone_Rifle]

Using a self-written script to parse the buildbot logs...
Revision 38738:
http://reactos.agilityhoster.com/get.php?revision=38738
crypt32-cert: executed 366, 0 failures
http://reactos.agilityhoster.com/builds ... 2-cert.txt

Revision 38743:
http://reactos.agilityhoster.com/get.php?revision=38743
crypt32-cert: executed 356, 3 failures
http://reactos.agilityhoster.com/builds ... 2-cert.txt

Somebody get in touch with the devs please. Thanks.

[Christoph_vW]

looks pretty random:

r38745: cert: 361 tests executed (0 marked as todo, 2 failures), 0 skipped.
r38744: cert: 361 tests executed (0 marked as todo, 1 failure), 0 skipped.
r38743: cert: 356 tests executed (0 marked as todo, 3 failures), 0 skipped.
r38742: cert: 366 tests executed (0 marked as todo, 0 failures), 0 skipped.
r38741: cert: 361 tests executed (0 marked as todo, 2 failures), 0 skipped.

[vicmarcal]

Ok Lone-Rifle...They have been notified.Thanks a lot.

BTW i love ur script...a nice way to compare results,finding regressions
Could you improve it making also a compare Diff?I mean you choose two revisions(or just comparing with previous revision) and show just the Tests that have changed,and how they have changed:if Reactos made a regression, if Reactos is passing/skipping more tests...So Devs can find them faster.

[Lone_Rifle]

I'm working on it in my head. =)
Just bear in mind that this is just a stopgap measure cum public information tool so that people can have a clearer idea of what to look for in our source tree.

Another thing i missed: the scripts seem to pick up that the advapi32-service test now has 3 more failures from 38738 to 38743.
http://reactos.agilityhoster.com/builds ... ervice.txt
http://reactos.agilityhoster.com/builds ... ervice.txt

[vicmarcal]

uhmm...maybe cert test have discovered a random bug.

Lone Rifle, a question and a enhancement:

1)How long does the full test take?

Enhancement:
Could you make the test TWICE each revision?Then script compare both Logs (from same revision) and create a Final Log where is checked as "Random Failures" failures that seems to be Random(maybe You can add a new COL). So then Devs can look Test code(maybe it´s wrong) or search why ReactOS fails randomly.

[Black_Fox]

Lone_Rifle parses what is already available at BuildBot (eg. here), no additional testing.