| Summary: | QA and release process not reproducable | ||
|---|---|---|---|
| Product: | [QA/Testing] Test Plans/Suite | Reporter: | Wolfgang Denk <wd> |
| Component: | test-plans/suite | Assignee: | Beth Flanagan <elizabeth.flanagan> |
| Status: | VERIFIED FIXED | QA Contact: | |
| Severity: | major | ||
| Priority: | High | CC: | alexandru.c.georgescu, corneliux.stoicescu, dvhart, elizabeth.flanagan, ionutx.chisanovici, laurentiu.serban, richard.purdie, sgw, srifenbark |
| Version: | 1.4 | ||
| Target Milestone: | 1.6 M1 | ||
| Hardware: | All | ||
| OS: | Multiple | ||
| Whiteboard: | 24 March 2014: Set Documentation flag to "Done". | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Done (doc changes complete) | |
Wolfgang, have you looked at Testopia part of bugzilla? QA Team: do we have a wiki documenation of our test process Beth: What about the release process? Trying to get this re-started Here is the Yocto 1.4 Overall Test Plan: https://wiki.yoctoproject.org/wiki/Yocto_1.4_Overall_Test_Plan#What_will_not_Be_Tested_in_Yocto_v1.4 If you would like, I can make a wiki page that contains all our test cases sorted by product. Also, the testing process can be followed by accessing the Testopia extension of Bugzilla (click on "Product Dashboard" - you need an active Bugzilla account). Here are 2 pages with documentation on Testopia for the Yocto Project: https://wiki.yoctoproject.org/wiki/Testopia https://wiki.yoctoproject.org/wiki/Testopia_HOWTO Last but certainly not least here is the QA wiki page that contains information on the QA process used from Yocto 0.9 until 1.4(including the pages above): https://wiki.yoctoproject.org/wiki/QA I hope this helps :) The release process is also documented at: https://wiki.yoctoproject.org/wiki/Yocto_Release_Engineering#Technical_Release_Procedures I'll be reviewing the document to make sure it's accurate (it should be mostly accurate). (In reply to comment #1) > Wolfgang, have you looked at Testopia part of bugzilla? I have to admit that I was not even aware that this exists. But then, my question was not so much about testing - it was about tracking the development process leading to a release, and the release itself. For example, I still don't see how I can learn (early in the release cycle) what the branch name for the upcoming release is / will be? (In reply to comment #3) > The release process is also documented at: > https://wiki.yoctoproject.org/wiki/ > Yocto_Release_Engineering#Technical_Release_Procedures Thanks - but this document also does not answer the questions I raised. For example, what is the official, documented way to find out which branch I should use to track the development process for the upcoming 1.4 release? Beth, Please add some notes for Scott to incorporate into the wiki pages When we get to updating or creating content for any wiki pages, let me know and I can help out. Scott This is a larger issue that starts hitting into workflow (specifically tagging.) I'm going to slip this out to 1.5. I have a few ideas about how to fix this so that the tagging issue is automatic via the AB but it requires a slight change in our release workflow. Beth, could you set a target milestone please since this is a medium+ Working on updating the wiki. ETA: Friday Aug 30th 2013 This got delayed. Updated ETA: Sept 4th. Kicking this to 1.6 We should also document here branch creation process for a release: When the branch for a release is created, what happens to it until release and between the point releases (1.5.1, 1.5.2, 1.5.3, etc)) I've added some more documentation on the wiki regarding the release process. Specifically: https://wiki.yoctoproject.org/wiki/Yocto_Release_Engineering https://wiki.yoctoproject.org/wiki/Frequently_Asked_Release_Engineering_Questions and I've updated the main release process document to remove some legacy technical release steps: https://wiki.yoctoproject.org/wiki/Yocto_Project_Release_Process as well as our tagging conventions: https://wiki.yoctoproject.org/wiki/Yocto_Project_Tagging_Conventions This does answer (I think, Wolfgang?) all of your questions. (In reply to comment #14) > I've added some more documentation on the wiki regarding the release > process. Specifically: > > https://wiki.yoctoproject.org/wiki/Yocto_Release_Engineering > > https://wiki.yoctoproject.org/wiki/ > Frequently_Asked_Release_Engineering_Questions > > and I've updated the main release process document to remove some legacy > technical release steps: > > https://wiki.yoctoproject.org/wiki/Yocto_Project_Release_Process > > as well as our tagging conventions: > > https://wiki.yoctoproject.org/wiki/Yocto_Project_Tagging_Conventions Thanks. > This does answer (I think, Wolfgang?) all of your questions. Yes, it does - mostly. Is there a proces sin place to keep these pages updated? For example, https://wiki.yoctoproject.org/wiki/Frequently_Asked_Release_Engineering_Questions reads: "The next major release is due out Oct 18th 2013." That was a couple of weeks ago... Wolfgang, I've added a section in the FAQ to address this. I'm thinking a full review 4 times a year should keep us mostly up to date. Wolfgang, can you please set to VERIFIED if you think this bug is resolved correctly? Thanks! Beth, I think you can flip the Doc flag to "Done" on this one. Once you do that it will not show up on my radar. Thanks, Scott Resetting the documentation flag to "Done". Beth says it is good to go. Scott Hi Wolfgang, Can you please let us know if this is solved so we can close this ? Thanks, Ionut C (In reply to comment #20) > > Can you please let us know if this is solved so we can close this ? Dear Ionut, the situation is indeed much better now, especially the documentation of the releases is really helpful. However, I cannot really confirm that all aspects of the issue are actually solved. One of my questions was: | - I don't know of any document which explains or documents the git | branch names being used for development. For example, how can I find | out ("officially") what the branch name for Yocto 1.4 will be / is ? I maybe just missing it, but I still cannot find much documentation of the develoment resources. Even the dev-man states only: Note You can view the Yocto Project Source Repositories at http://git.yoctoproject.org/cgit.cgi This page just lists a ton of repositories, but it does not explain at all where I can find any develoment resources related to the release process. I've been working with Yocto for some time, but I still have problems to find answers to questions like these: - There is for example yocto-1.4.3 and yocto-1.5.1; there should be some branches where the development of this code has taken place. Where can I find these? - Where can I find information about ongoing maintenance work - for example, is there any work in progress for some yocto-1.4.4 or yocto-1.5.2 release? If so, where would I find the respective branches? And how could I find out if any such work is in progress at all? I would really appreciate to have this documented somewhere. hi, I will try to answer your questions inline. (In reply to comment #21) > (In reply to comment #20) > > > > Can you please let us know if this is solved so we can close this ? > > Dear Ionut, > > the situation is indeed much better now, especially the documentation > of the releases is really helpful. > > However, I cannot really confirm that all aspects of the issue are > actually solved. One of my questions was: > > | - I don't know of any document which explains or documents the git > | branch names being used for development. For example, how can I find > | out ("officially") what the branch name for Yocto 1.4 will be / is ? > > I maybe just missing it, but I still cannot find much documentation > of the develoment resources. Even the dev-man states only: > > Note > You can view the Yocto Project Source Repositories at > http://git.yoctoproject.org/cgit.cgi > > This page just lists a ton of repositories, but it does not explain at > all where I can find any develoment resources related to the release > process. > > I've been working with Yocto for some time, but I still have problems > to find answers to questions like these: > > - There is for example yocto-1.4.3 and yocto-1.5.1; there should be > some branches where the development of this code has taken place. > Where can I find these? > > - Where can I find information about ongoing maintenance work - for > example, is there any work in progress for some yocto-1.4.4 or > yocto-1.5.2 release? If so, where would I find the respective > branches? > > And how could I find out if any such work is in progress at all? > > I would really appreciate to have this documented somewhere. (In reply to comment #21) > (In reply to comment #20) > > > > Can you please let us know if this is solved so we can close this ? > > Dear Ionut, > > the situation is indeed much better now, especially the documentation > of the releases is really helpful. > > However, I cannot really confirm that all aspects of the issue are > actually solved. One of my questions was: > > | - I don't know of any document which explains or documents the git > | branch names being used for development. For example, how can I find > | out ("officially") what the branch name for Yocto 1.4 will be / is ? > > I maybe just missing it, but I still cannot find much documentation > of the develoment resources. Even the dev-man states only: > > Note > You can view the Yocto Project Source Repositories at > http://git.yoctoproject.org/cgit.cgi > > This page just lists a ton of repositories, but it does not explain at > all where I can find any develoment resources related to the release > process. > > I've been working with Yocto for some time, but I still have problems > to find answers to questions like these: > > - There is for example yocto-1.4.3 and yocto-1.5.1; there should be > some branches where the development of this code has taken place. > Where can I find these? They are always announced before the major release on the public mailing list(e.g. before releasing 1.4, 1.5, 1.6, etc. For ex, for 1.6 it is called "daisy" and all the work that is done after 1.6 was released and will be on future point releases, like 1.6.x you will find it in there. > > - Where can I find information about ongoing maintenance work - for > example, is there any work in progress for some yocto-1.4.4 or > yocto-1.5.2 release? If so, where would I find the respective > branches? I think the answer is the same as above. For reference, there is a "releases wiki": https://wiki.yoctoproject.org/wiki/Releases which contains links to the release status, features, qa runs (for every release + release names). > > And how could I find out if any such work is in progress at all? > > I would really appreciate to have this documented somewhere. |
For parties outside the Yocto project it is impossible to follow, or to reproduce, the Yocto release process. At least the puclicly visible documentation of the development, test and release processes is insufficient. The major drawback is that there is virtually no communication about which branches or commit IDs represent the respective states of the development process. This is an ongoing problem, that has been raised on the mailing list anumber of times before, but I don't see much improvments. - I don't know of any document which explains or documents the git branch names beingused for development. For example, how can I find out ("officially") what the branch name for Yocto 1.4 will be / is ? - There appears to be no formal procedure in place to assign consistent git tags to releases or milestone releases. For example, for Yocto 1.2 and before, we had release tags like denzil-7.0, edison-6.0, bernard-5.0, laverne-4.0, etc. Some parts of the current 1.3 Yocto release refer to it as "Poky 8.0" or Version "8.0 Danny", see for example here: https://www.yoctoproject.org/download/yocto-project-13-poky-80 However, the release tag "danny-8.0" had been "forgotten" until I asked about it, see [1] [1] http://thread.gmane.org/gmane.linux.embedded.yocto.general/10929 - It appears milestone releases are merely documented by a release date, which is not much help to identify the specific git commit ID in the large number of branches in the Yocto git repository. For example, all publicly available information about the M2 milestone release appears to be this little bit of information, see [2]: M2 release: 1/11/13 [2] https://wiki.yoctoproject.org/wiki/Yocto_1.4_Schedule#M2_Stabilize_schedule There appears to be no public information which exact git commit ID this may represent, nor is there any git tag similar to 1.4_M2 or similar in the git repository. It would be very nice if the whole developpment and release provess would be a bit more documented so it can be traced and reproduced. As is, this is impossible.