Trunk breakage

Here you can discuss ReactOS related topics.

Moderator: Moderator Team

phy_lbc
Posts: 15
Joined: Sat Nov 21, 2009 4:32 pm

Trunk breakage

Post by phy_lbc »

I'm not much of a developer, but after years of lurking I've noticed that people keep on committing code with the comment "fixes build". I know that they aren't intentionally trying to break trunk, but is there a way that testman could check if a particular commit would break the trunk, or make a particular revision unbootable? So if this was the case, it would prevent that particular piece of code from being committed, then the devs could find out what aspect of that commit was causing the problem, and fix it before its included in trunk. From what I understand, if the trunk becomes unbootable and a lot of commits occur afterwards, its very hard to find out if any of the subsequent commits break anything, or which one did.

I know that Haos recently activated a new builder which can download patches, I'm not sure if it works the same as what I'm outlining though.

Do devs think this be a useful feature, and how hard would it be to implement?


PS - If anything I said is is incorrect, please don't hesitate to correct me.
vicmarcal
Test Team
Posts: 2733
Joined: Mon Jul 07, 2008 12:35 pm

Re: Trunk breakage

Post by vicmarcal »

Hi phy_lbc
but is there a way that testman could check if a particular commit would break the trunk
ReactOS testman tests after the build is done, so our testman can not check the patch before commiting.
From what I understand, if the trunk becomes unbootable and a lot of commits occur afterwards, its very hard to find out if any of the subsequent commits break anything, or which one did.
Yes,you are right any regression created in that big/small gap can lead to not find where the problem was introduced easily. Regtesting becomes an impossible task, ask the testers :)
I know that Haos recently activated a new builder which can download patches, I'm not sure if it works the same as what I'm outlining though.
Do devs think this be a useful feature, and how hard would it be to implement?
Right now Haos-new-buildbot is testing patches on demand to check for regressions, so it is not testing all the changes sent to the repository. Maybe in a short future it could be used to test the commit before sending to the repository or automatically revert any commit that breaks the building process and freezing trunk automatically. Haos is doing a quite nice work with buildbots, who knows his next surprise(if any)?
sh4ring4n
Posts: 120
Joined: Thu Oct 30, 2008 2:05 am
Location: Canada
Contact:

Re: Trunk breakage

Post by sh4ring4n »

vicmarcal wrote:Yes,you are right any regression created in that big/small gap can lead to not find where the problem was introduced easily. Regtesting becomes an impossible task, ask the testers :)
What's regtesting?
The cake is a lie!
vicmarcal
Test Team
Posts: 2733
Joined: Mon Jul 07, 2008 12:35 pm

Re: Trunk breakage

Post by vicmarcal »

sh4ring4n wrote:
vicmarcal wrote:Yes,you are right any regression created in that big/small gap can lead to not find where the problem was introduced easily. Regtesting becomes an impossible task, ask the testers :)
What's regtesting?
Regtesting is the short form of "Regression Testing". Regression testing is the process of binary search trying to find which commit has introduced a bug. If there is a blank-impossible-to-test gap and the gap is big enough, then you can just test the limits of that gap but you can't find which is the exactly guilty commit.
Haos
Test Team
Posts: 2954
Joined: Thu Mar 22, 2007 5:42 am
Contact:

Re: Trunk breakage

Post by Haos »

For such things a new builder has been devised. Named "Patch_x86_GCCWin Debug", it can be used by ROS team to test patches sent to bugzilla, up to running winetest suite on them.
vicmarcal
Test Team
Posts: 2733
Joined: Mon Jul 07, 2008 12:35 pm

Re: Trunk breakage

Post by vicmarcal »

Haos wrote:For such things a new builder has been devised. Named "Patch_x86_GCCWin Debug", it can be used by ROS team to test patches sent to bugzilla, up to running winetest suite on them.
Yup, he mentionated it in his first post:
phy_lbc wrote:I know that Haos recently activated a new builder which can download patches, I'm not sure if it works the same as what I'm outlining though.
But seems he is asking about a permanent builder which checks any daily dev commit, rejecting the ones that breaks building. In other words, extending current Patch_x86 functionality, which now works mainly on demand and just with Bugzilla reports.
Haos
Test Team
Posts: 2954
Joined: Thu Mar 22, 2007 5:42 am
Contact:

Re: Trunk breakage

Post by Haos »

I dont see any problem for devs to use Patch buildbot on their own commits, not only if they are unsure of correctness, but perhaps also willing a speedy check on them. It is only up to them to use it. Several build breakages can arise not from just coding errors, but often with simply missing a single file in a large commit.
phy_lbc
Posts: 15
Joined: Sat Nov 21, 2009 4:32 pm

Re: Trunk breakage

Post by phy_lbc »

vicmarcal wrote:Maybe in a short future it could be used to test the commit before sending to the repository or automatically revert any commit that breaks the building process and freezing trunk automatically. Haos is doing a quite nice work with buildbots, who knows his next surprise(if any)?
I also wasn't sure how feasible it would be to implement that feature, it might increase the time to commit code.
Haos wrote:I dont see any problem for devs to use Patch buildbot on their own commits, not only if they are unsure of correctness, but perhaps also willing a speedy check on them. It is only up to them to use it.
I'm not sure how willing devs would be to manually test all of their commits before they commit them. It could get tedious after a few days of doing it, thats why I thought an automatic tool might be helpful here.
Haos wrote:Several build breakages can arise not from just coding errors, but often with simply missing a single file in a large commit.
Even still if it was just missing files and not coding errors, it might be hours before somebody notices, by which time more commits could have happened which might make regtesting harder. I was only thinking about the poor testers!
vicmarcal
Test Team
Posts: 2733
Joined: Mon Jul 07, 2008 12:35 pm

Re: Trunk breakage

Post by vicmarcal »

phy_lbc wrote:I also wasn't sure how feasible it would be to implement that feature, it might increase the time to commit code.
It shouldn't be quite difficult, but Haos knows much better:

1)The commits reaches to one of our BuildBots (call it Approval BuildBot) as if the commit is a patch
2)The "Approval BuildBot" compiles it , if it doesnt break..then..
3)The commit is sent to the repository and recompiled by any of our current buildbots.
4)If the commit breaks our "Approval Buildbot", then the patch is rejected and never reaches the repository.

It can even been improved, so the "Approval Buildbot" doesnt just check if it builds, but also if it regresses anything.In that case the patch will be rejected too.
Z98
Release Engineer
Posts: 3379
Joined: Tue May 02, 2006 8:16 pm
Contact:

Re: Trunk breakage

Post by Z98 »

All devs are supposed to test their changes before committing. That's been a rule since probably the project's inception. Things still break due either to minor mistakes in the actual committing or the dev didn't do a run of the testsuite themselves. An automated system carries its own complications, especially if there is a lot of commit activity. Building and running the testsuite takes time. Queuing up a lot of commits runs the risk of conflicts that the system cannot resolve, which may fail all the commits afterward. A dev running their commit through Haos' tool by hand lessens that risk but requires more hands on time from the dev. It's all about tradeoffs and what which we're prepared to live with.
vicmarcal
Test Team
Posts: 2733
Joined: Mon Jul 07, 2008 12:35 pm

Re: Trunk breakage

Post by vicmarcal »

Z98 wrote:All devs are supposed to test their changes before committing. That's been a rule since probably the project's inception. Things still break due either to minor mistakes in the actual committing or the dev didn't do a run of the testsuite themselves. An automated system carries its own complications, especially if there is a lot of commit activity. Building and running the testsuite takes time. Queuing up a lot of commits runs the risk of conflicts that the system cannot resolve, which may fail all the commits afterward. A dev running their commit through Haos' tool by hand lessens that risk but requires more hands on time from the dev. It's all about tradeoffs and what which we're prepared to live with.
But some other projects use a "regressions checker bot" to avoid regressions before sending to the repository. That will be quite useful as they will be catched quite easy
Z98
Release Engineer
Posts: 3379
Joined: Tue May 02, 2006 8:16 pm
Contact:

Re: Trunk breakage

Post by Z98 »

Vic, please reread what I posted, cause I have no idea what your response is relating to with respect to my post.
vicmarcal
Test Team
Posts: 2733
Joined: Mon Jul 07, 2008 12:35 pm

Re: Trunk breakage

Post by vicmarcal »

Z98 wrote:Vic, please reread what I posted, cause I have no idea what your response is relating to with respect to my post.
It was related to:
Z98 wrote:An automated system carries its own complications, especially if there is a lot of commit activity. Building and running the testsuite takes time. Queuing up a lot of commits runs the risk of conflicts that the system cannot resolve, which may fail all the commits afterward.
I'm agree that an automated system has its own complications but some of the opensource projects are using "regressions checker bots" and seems to work pretty well for them.
You have described all the problems that creating an "automated system" exposes, and I wanted to pinpoint that there are good reasons to use them too, reasons which seems to be higher than those problems,i.e:"Regression catching".
Z98
Release Engineer
Posts: 3379
Joined: Tue May 02, 2006 8:16 pm
Contact:

Re: Trunk breakage

Post by Z98 »

Highlighting their "good" points doesn't really address their issues. We are aware of their benefits, but putting them into practice requires overcoming the complications.
greenie
Posts: 145
Joined: Mon Jan 19, 2009 12:10 am

Re: Trunk breakage

Post by greenie »

Wine has a regression test suite doesn't it? on the wine mailing list i regularly get messages from a bot telling everyone that revision x may have caused a regression. Though that it is not good at pattern matching because they are a bot.
Last edited by greenie on Fri Mar 25, 2011 12:20 am, edited 1 time in total.
Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot], Baidu [Spider], DotBot [Crawler], Semrush [Bot] and 2 guests