| Summary: | Toaster: bitbake variables image_fstypes not being passed to build process | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] Toaster | Reporter: | Alexandru Roman <alexandru.costinx.roman> | ||||||||||||
| Component: | toaster | Assignee: | Elliot Smith <elliot.smith> | ||||||||||||
| Status: | RESOLVED FIXED | QA Contact: | Libertad <libertad.gonzalez.de.la.cruz> | ||||||||||||
| Severity: | normal | ||||||||||||||
| Priority: | Medium+ | CC: | belen.barros.pena, brian.avery, jessica.zhang, jose.perez.carranza, libertad.gonzalez.de.la.cruz, stanciux.mihail | ||||||||||||
| Version: | 2.1 | ||||||||||||||
| Target Milestone: | 2.2 M2 | ||||||||||||||
| Hardware: | x86 | ||||||||||||||
| OS: | Multiple | ||||||||||||||
| Whiteboard: | |||||||||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||||||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||||||||||
| Bug Depends on: | |||||||||||||||
| Bug Blocks: | 8440, 9784 | ||||||||||||||
| Attachments: |
|
||||||||||||||
Created attachment 2811 [details]
bitbake variables
Created attachment 2812 [details]
vmdk
Please verify that this is still a bug, as the fix for bug 8910 should have fixed this. I think this is fixed, but I cannot tell for sure because of bug 8956. However, builds no longer fail, and looking at the IMAGE_FSTYPES variable value it shows that vmdk has been successfully appended. I guess the fix for bug 8781 had also something to do with it. I'll leave it to Alex Roman to decide if we can close it or if he wants to wait for bug 8956 to verify. Created attachment 2993 [details]
core-image-minimal
I tested again on master: 152914f2983c5d69001de1d46ce99547fa1e75fe and I found the same behavior.
I selected vmdk for IMAGES_FSTYPES and built a core-image-minimal. On build summary page I didn't found .vdmk artifact. (core-image-minimal attachment).
Then with vmdk selected for IMAGES_FSTYPES I built a core-image-base. On build summary page I found "core-image-minimal.vmdk" artifact. (core-image-base attachment).
I deselected vmdk, built a core-image-minimal and found core-image-base.vmdk artifact.
Created attachment 2994 [details]
core-image-base
I can reproduce this with toaster-next. I wonder if it's somehow connected to bug 9023 I can reproduce it again on master: b58e5b1fb3bcacbcfce17a748bca5f60c3528b4f 1. I selected from project configuration -> BitBake variables for Image_fstypes: ext3, jffs2, tar.bz2, hddimg, ext4 2. Build a core-image-minimal 3. Click on the build. On image files I can see ext4 but I can't see hddimg. 4. Go again on project page -> Bitbake Variables and deselect hddimg for image_fstypes. 5. Build a core-image-minimal 6. Go to build page. On other artifacts I have: core-image-minimal-qemux86-20160325074958.hddimg (40.6 MB) *** Bug 9358 has been marked as a duplicate of this bug. *** This bug prevents iso and hddimg rootfs files from being displayed in Toaster. This is quite a problem, since those files are probably the ones users are looking for. If such users don't have access to the tmp directory, they are stuck. It should probably be bumped to medium+ and the fix backported to 2.1. I can show vmdk files ok; but I'm not 100% sure how we can reliably show artifacts related to a build. The problem is that some files are produced for one build, but we want to show them for other builds; but the build doesn't tell us which files or when We can figure it out for image files, because we know what the prefix for those is; but for a file like the bzImage, there's not really a reliable way to tell whether we should include it in the build artifacts section or not; and there could be other files which are even more obscure, potentially. I have discussed this with RP, who mentioned that bitbake doesn't really know which files it is producing, as that's down to tasks. So bitbake doesn't emit any events which relate a file to the target which produced it. NB fstypes is being passed correctly, it's just that we don't detect vmdk files correctly. The same fix should also work for iso and hddimg files, too. But there is a fundamental problem in that we don't know which files go with which build targets. I'll look into this. One further complication: we attach our handlers which pick up build artifacts to the do_image_complete task as postfuncs. However, if this task is skipped because it is in the sstate-cache, the postfuncs for do_image_complete don't run at all. This means we can't rely on our build artifact collection functions being run consistently for every build. One way round this might be to scan the image deploy directory after the build is completed, and remove the postfunc which does this job. Having sorted out most of the mess around how we scan for build artifacts, I noticed that we still have an issue where the IMAGE_FSTYPES variable supposedly passed to a build appears to be amended *during* the build. I followed these steps to isolate the issue: 1. Remove database and build-toaster* directories (NB my sstate-cache is still primed) 2. Start Toaster from the <toaster source>/build/ directory (as per usual) 3. Create a new local project 4. Look at Project > Bitbake Variables: IMAGE_FSTYPES is ext3 jffs2 tar.bz2 (Toaster default, as expected) 5. Start a build of core-image-minimal for this project The sequence of commands which Toaster runs is: 2016-06-21 10:51:52,411 DEBUG lbc_shellcmmd: (/home/ell/dev/toaster/poky) bash -c 'source /home/ell/dev/toaster/poky/oe-init-build-env /home/ell/dev/toaster/poky/build-toaster-2' 2016-06-21 10:51:52,513 DEBUG localhostbecontroller: shellcmd success 2016-06-21 10:51:52,518 DEBUG lbc_shellcmmd: (/home/ell/dev/toaster/poky) bash -c "source /home/ell/dev/toaster/poky/oe-init-build-env /home/ell/dev/toaster/poky/build-toaster-2; BITBAKE_UI="knotty" /home/ell/dev/toaster/poky/bitbake/bin/bitbake --read /home/ell/dev/toaster/poky/build-toaster-2/conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0" 2016-06-21 10:51:53,575 DEBUG localhostbecontroller: shellcmd success 2016-06-21 10:51:53,575 DEBUG localhostbecontroller: bitbake port 43041 2016-06-21 10:51:53,577 DEBUG lbc_shellcmmd: (/home/ell/dev/toaster/poky/build-toaster-2) ['bash -c "(TOASTER_BRBE="1:1" BBSERVER="0.0.0.0:-1" ../bitbake/bin/bitbake core-image-minimal -u toasterui --token="" >>/home/ell/dev/toaster/poky/build-toaster-2/toaster_ui.log 2>&1;BITBAKE_UI="knotty" BBSERVER=0.0.0.0:-1 /home/ell/dev/toaster/poky/bitbake/bin/bitbake -m)&"'] 6. While building, look at ../build-toaster-2/conf/toaster.conf; IMAGE_FSTYPES is ext3 jffs2 tar.bz2 (same as the project's setting for this variable, as expected) Note that ../build-toaster-2/conf/toaster.conf is read by the Bitbake server with --read. The server is started just before we run the actual bitbake command line; the build uses this server to run its targets. 7. When the build finishes, navigate to <project> > Builds > <finished build> > Configuration tab > "BitBake Variables"; search for IMAGE_FSTYPES The IMAGE_FSTYPES variable for this particular build has the value ext3 jffs2 tar.bz2 tar.bz2 ext4 ******* THIS IS NOT THE SAME AS THE VALUE IN THE PROJECT OR IN TOASTER.CONF ******* (also note spurious duplication of tar.bz2 in the string) The IMAGE_FSTYPES variable has changed its value somewhere during the build. In the build summary page, we have corresponding ext3, ext4, jffs2 and tar.bz2 image files listed, and image files for all four extensions are in the ../build-toaster-2/tmp/deploy/images/qemux86 directory. But *the ext4 file should not be there*, as we never asked for it to be built. Since some BSP's will require a certain image type to function, perhaps we should phrase the IMAGE_FSTYPES as an append so that the users realize that their settings will append an fstype but will not necessarily override the type the bsp sets. For example, if the bootloader on a particular board only understands a jffs image and your target that board but set IMAGE_FSTYPES to tar.gz (because you want to run a tool over the binaries), it makes sense that you would get both jffs and tar.gz. Perhaps we can put in an enhancement for a override checkbox and a help note that says this variable may be modified by recipes unless the checkbox is active. I wouldn't do the force override unless it was explicitly selected and I would add warning language that a force override may very well break the image build. I am somehow wary of giving some variables special treatment: people might think those variables are special in the build system too, when they really are not. It will also add more delay to fixing this issue, which I think we should solve sooner rather than later. If what makes sense most of the times is to append to IMAGE_FSTYPES, let's just do that in the toaster.conf file, and without user intervention for the time being. It was always our intention to expose the BitBake operators in the "bitbake variables" page at some point, so that you can choose from them when setting variables. We can do so in a way that explains what each operator means, which should help Toaster users learn about them, and hopefully select the right one (whichever that might be in each case). |
Created attachment 2810 [details] error Appears on master: 505a82673ac2487df5ea343a6422c2fc47018831 Step to reproduce: 1. open toaster 2. create a new project 3. build a core-image-minimal 4. wait until build finish 5. go to project configuration page -> BitBake variables 6. for IMAGE_FSTYPES select vmdk and click save 7. build another core-image-minimal (wait until build finish - on build summary page we should have an artifact with .vmdk extension - actually nothing like this appears) 8. Try to build another core-image-minimal with the same project configuration (vmdk selected) - build fails with 1 error: Function failed: build_boot_dd (log file is located at /home/romanalx/yocto/poky/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_bootdirectdisk.28373) 9. Then if you deselect vmdk from project configuration page - bitbake variables and try to build a core-image-minimal the build complete successfully and you find on other artifacts a image with .vmdk extension