I am following the pre-release regression test pages for years and have been doing my share of testing certain OS/non-OS software for every release since a while. I noted that the test page for ROS 0.4.14 has not seen much updates over the last months. The corresponding release branch was branched in April 2020.
Since a few weeks there's this notice on the page:
I have long pondered whether this approach is doing the project any good or not, and I have come to the conclusion that it might do more harm to the project than intended. Let me outline why:The Testing will not begin until we are < 21 regressions, and the Release will be blocked until we are < 16 regressions!
Pro strict regression limits:
- Each release will be better than the previous one if the number of regressions does not increase. This is the obvious benefit
- People using ROS productively can rely that nothing serious will break (Well, I hope nobody uses alpha software in production.)
- It enforces a penalty to the team for not caring enough for regressions or avoiding them in the first place, so it might increase the quality awareness.
- My guess is we will not see any release this year this way.
- Many people (outside the group of developers) use releases to test "the latest and greatest" features. If people realize that their release version actually reflects the state of the project (feature-wise) from a year ago, they are less likely to use the release, more likely to skip to the daily builds. Which – in my impression – makes the release pointless for many users
- No releases also means less media coverage, less people caring about the project. Sometime the home page doesn't have news for months, which makes it look as if the project is inactive. Not the impression you want to give, do you?
- The longer the time between two software regression test cycles, the more unnoticed regressions will pile up. If the complete software stack has not been tested for a year now, I fear the next branch to contain a multiple of regressions of the current one, making future releases even less likely. (I hope I am wrong.)
- The project is in alpha stage. Nobody can expect a "release" to pass a certain quality gate at this stage. Were there a 1.x release I would fully agree with such strict rules.