Trunk breakage

[phy_lbc]

I'm not much of a developer, but after years of lurking I've noticed that people keep on committing code with the comment "fixes build". I know that they aren't intentionally trying to break trunk, but is there a way that testman could check if a particular commit would break the trunk, or make a particular revision unbootable? So if this was the case, it would prevent that particular piece of code from being committed, then the devs could find out what aspect of that commit was causing the problem, and fix it before its included in trunk. From what I understand, if the trunk becomes unbootable and a lot of commits occur afterwards, its very hard to find out if any of the subsequent commits break anything, or which one did.

I know that Haos recently activated a new builder which can download patches, I'm not sure if it works the same as what I'm outlining though.

Do devs think this be a useful feature, and how hard would it be to implement?


PS - If anything I said is is incorrect, please don't hesitate to correct me.

[vicmarcal]

Hi phy_lbc

but is there a way that testman could check if a particular commit would break the trunk

ReactOS testman tests after the build is done, so our testman can not check the patch before commiting.

From what I understand, if the trunk becomes unbootable and a lot of commits occur afterwards, its very hard to find out if any of the subsequent commits break anything, or which one did.

Yes,you are right any regression created in that big/small gap can lead to not find where the problem was introduced easily. Regtesting becomes an impossible task, ask the testers

I know that Haos recently activated a new builder which can download patches, I'm not sure if it works the same as what I'm outlining though.
Do devs think this be a useful feature, and how hard would it be to implement?

Right now Haos-new-buildbot is testing patches on demand to check for regressions, so it is not testing all the changes sent to the repository. Maybe in a short future it could be used to test the commit before sending to the repository or automatically revert any commit that breaks the building process and freezing trunk automatically. Haos is doing a quite nice work with buildbots, who knows his next surprise(if any)?

[sh4ring4n]

Yes,you are right any regression created in that big/small gap can lead to not find where the problem was introduced easily. Regtesting becomes an impossible task, ask the testers

What's regtesting?

[vicmarcal]

Yes,you are right any regression created in that big/small gap can lead to not find where the problem was introduced easily. Regtesting becomes an impossible task, ask the testers

What's regtesting?

Regtesting is the short form of "Regression Testing". Regression testing is the process of binary search trying to find which commit has introduced a bug. If there is a blank-impossible-to-test gap and the gap is big enough, then you can just test the limits of that gap but you can't find which is the exactly guilty commit.

[Haos]

For such things a new builder has been devised. Named "Patch_x86_GCCWin Debug", it can be used by ROS team to test patches sent to bugzilla, up to running winetest suite on them.

[vicmarcal]

For such things a new builder has been devised. Named "Patch_x86_GCCWin Debug", it can be used by ROS team to test patches sent to bugzilla, up to running winetest suite on them.

Yup, he mentionated it in his first post:

I know that Haos recently activated a new builder which can download patches, I'm not sure if it works the same as what I'm outlining though.

But seems he is asking about a permanent builder which checks any daily dev commit, rejecting the ones that breaks building. In other words, extending current Patch_x86 functionality, which now works mainly on demand and just with Bugzilla reports.

[Haos]

I dont see any problem for devs to use Patch buildbot on their own commits, not only if they are unsure of correctness, but perhaps also willing a speedy check on them. It is only up to them to use it. Several build breakages can arise not from just coding errors, but often with simply missing a single file in a large commit.

[phy_lbc]

Maybe in a short future it could be used to test the commit before sending to the repository or automatically revert any commit that breaks the building process and freezing trunk automatically. Haos is doing a quite nice work with buildbots, who knows his next surprise(if any)?

I also wasn't sure how feasible it would be to implement that feature, it might increase the time to commit code.

I dont see any problem for devs to use Patch buildbot on their own commits, not only if they are unsure of correctness, but perhaps also willing a speedy check on them. It is only up to them to use it.

I'm not sure how willing devs would be to manually test all of their commits before they commit them. It could get tedious after a few days of doing it, thats why I thought an automatic tool might be helpful here.

Several build breakages can arise not from just coding errors, but often with simply missing a single file in a large commit.

Even still if it was just missing files and not coding errors, it might be hours before somebody notices, by which time more commits could have happened which might make regtesting harder. I was only thinking about the poor testers!

[vicmarcal]

I also wasn't sure how feasible it would be to implement that feature, it might increase the time to commit code.

It shouldn't be quite difficult, but Haos knows much better:

1)The commits reaches to one of our BuildBots (call it Approval BuildBot) as if the commit is a patch
2)The "Approval BuildBot" compiles it , if it doesnt break..then..
3)The commit is sent to the repository and recompiled by any of our current buildbots.
4)If the commit breaks our "Approval Buildbot", then the patch is rejected and never reaches the repository.

It can even been improved, so the "Approval Buildbot" doesnt just check if it builds, but also if it regresses anything.In that case the patch will be rejected too.

[Z98]

All devs are supposed to test their changes before committing. That's been a rule since probably the project's inception. Things still break due either to minor mistakes in the actual committing or the dev didn't do a run of the testsuite themselves. An automated system carries its own complications, especially if there is a lot of commit activity. Building and running the testsuite takes time. Queuing up a lot of commits runs the risk of conflicts that the system cannot resolve, which may fail all the commits afterward. A dev running their commit through Haos' tool by hand lessens that risk but requires more hands on time from the dev. It's all about tradeoffs and what which we're prepared to live with.

[vicmarcal]

All devs are supposed to test their changes before committing. That's been a rule since probably the project's inception. Things still break due either to minor mistakes in the actual committing or the dev didn't do a run of the testsuite themselves. An automated system carries its own complications, especially if there is a lot of commit activity. Building and running the testsuite takes time. Queuing up a lot of commits runs the risk of conflicts that the system cannot resolve, which may fail all the commits afterward. A dev running their commit through Haos' tool by hand lessens that risk but requires more hands on time from the dev. It's all about tradeoffs and what which we're prepared to live with.

But some other projects use a "regressions checker bot" to avoid regressions before sending to the repository. That will be quite useful as they will be catched quite easy

[Z98]

Vic, please reread what I posted, cause I have no idea what your response is relating to with respect to my post.

[vicmarcal]

Vic, please reread what I posted, cause I have no idea what your response is relating to with respect to my post.

It was related to:

An automated system carries its own complications, especially if there is a lot of commit activity. Building and running the testsuite takes time. Queuing up a lot of commits runs the risk of conflicts that the system cannot resolve, which may fail all the commits afterward.

I'm agree that an automated system has its own complications but some of the opensource projects are using "regressions checker bots" and seems to work pretty well for them.
You have described all the problems that creating an "automated system" exposes, and I wanted to pinpoint that there are good reasons to use them too, reasons which seems to be higher than those problems,i.e:"Regression catching".

[Z98]

Highlighting their "good" points doesn't really address their issues. We are aware of their benefits, but putting them into practice requires overcoming the complications.

[greenie]

Wine has a regression test suite doesn't it? on the wine mailing list i regularly get messages from a bot telling everyone that revision x may have caused a regression. Though that it is not good at pattern matching because they are a bot.