9414 2016-04-07 09:04:21 +0000 add support for bmap-tools 2016-05-19 12:59:33 +0000 1 1 7 Build System, Metadata & Runtime OE-Core deployment 2.2 x86 Multiple RESOLVED FIXED Medium+ enhancement 2.2 M1 9413 1 eduard.bartosh eduard.bartosh alexander.kanevskiy bluelightning jose.perez.carranza joshuagloe kergoth francisco.j.pedraza.gonzalez --- --- Yes (doc changes required) oldest_to_newest 60892 0 eduard.bartosh 2016-04-07 09:04:21 +0000 bmap-tools(https://github.com/01org/bmap-tools/tree/master/docs) is a set of generic tools for creating the block map (bmap) for a file, and copying files using the block map. The idea is that large file containing unused blocks, like raw system image files, can be copied or flashed a lot faster with bmaptool than with traditional tools like "dd" or "cp". There is ongoing work in this area in Ostro project: https://github.com/ostroproject/meta-ostro/pull/70/commits that can be reused and should make the task of bringing bmap-tools support to yocto/oe-core much simpler. 61687 1 eduard.bartosh 2016-04-26 12:49:00 +0000 Patchset has been sent for review: http://lists.openembedded.org/pipermail/openembedded-core/2016-April/thread.html 61715 2 eduard.bartosh 2016-04-27 11:32:01 +0000 The recipe is failing when MACHINE is changed: $ bitbake bmap-tools-native WARNING: Host distribution "openSUSE-13.2" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 100% |########################################################################################################| Time: 00:00:11 Parsing of 882 .bb files complete (0 cached, 882 parsed). 1314 targets, 44 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.30.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "x86_64-poky-linux" MACHINE = "genericx86-64" DISTRO = "poky" DISTRO_VERSION = "2.1" TUNE_FEATURES = "m64 core2" TARGET_FPU = "" meta meta-poky meta-yocto-bsp meta-selftest = "ed/oe-core/bmap-tools-9414:7416c3ba19f26fe6eea485fefad4db4881f829ad" NOTE: Preparing RunQueue NOTE: Executing SetScene Tasks ERROR: bmap-tools-native-3.2-r0 do_deploy_setscene: The recipe bmap-tools-native is trying to install files into a shared area when those files already exist. Those files and their manifest location are: /home/ed/git/yocto/poky/build/tmp/deploy/tools/bmaptool-3.2 Matched in manifest-qemux86-64-bmap-tools-native.deploy /home/ed/git/yocto/poky/build/tmp/deploy/tools/bmaptool Matched in manifest-qemux86-64-bmap-tools-native.deploy Please verify which recipe should provide the above files. I'll remove preparing standalone binary and installing it into tmp/deploy/tools from the recipe. the tool still can be run as any native oe tool, so removing standalone version doesn't influence anything. 61718 3 eduard.bartosh 2016-04-27 12:35:14 +0000 sent v2 for review: http://lists.openembedded.org/pipermail/openembedded-core/2016-April/120687.html 61816 4 alexander.kanevskiy 2016-04-28 22:06:05 +0000 better to add "inherit allarch" than removing standalone version of tool. Otherwise, user end-up in situation where bmap created, but there is no easily usable tool to use generated bmap for the image. 61817 5 bluelightning 2016-04-28 23:05:16 +0000 For reference, the ostro recipe (with do_deploy) is here: https://raw.githubusercontent.com/ostroproject/meta-ostro/master/meta-ostro/recipes-devtools/bmap-tools/bmap-tools_3.2.bb 61821 6 bluelightning 2016-04-29 04:21:03 +0000 I figured out the issue with the do_deploy error above - since the bmap-tools recipe is doing this in a non-machine-specific manner it should set the following: do_deploy[stamp-extra-info] = "" This overrides the default value of "${MACHINE}" set in deploy.bbclass which causes the task to re-execute when MACHINE changes (since what's in stamp-extra-info gets added to the name of the stamp file, so if it changes it doesn't see that it's stamped as already having been completed). 61822 7 bluelightning 2016-04-29 04:22:01 +0000 I should mention also, inherit allarch would not be appropriate for this recipe - when building for the target it *is* arch-dependent. 61823 8 bluelightning 2016-04-29 04:24:47 +0000 For clarification, the reason it's arch-dependent (even though it's written in an arch-independent scripting language) is that the signature depends on python which will rebuild for the target architecture and thus influence the signature of tasks within this recipe; this is something we don't currently optimise for. 61835 9 eduard.bartosh 2016-05-02 07:15:49 +0000 (In reply to comment #4) > better to add "inherit allarch" than removing standalone version of tool. > Otherwise, user end-up in situation where bmap created, but there is no > easily usable tool to use generated bmap for the image. I can run the tool this way: PATH=tmp/sysroots/x86_64-linux/usr/bin/ bmaptool. It works just fine. 61841 10 alexander.kanevskiy 2016-05-02 10:46:58 +0000 I wouldn't call this method of tool invocation as "convenient" :) Better to fix recipe, especially, because one line fix available from Paul. 61885 11 eduard.bartosh 2016-05-03 10:20:14 +0000 Sent v3 for review: http://lists.openembedded.org/pipermail/openembedded-core/2016-May/120834.html 61888 12 alexander.kanevskiy 2016-05-03 11:31:10 +0000 there is no much sense to separate bmap-tools recipe into two commits, but end result is ok. 62286 13 eduard.bartosh 2016-05-18 12:48:54 +0000 There are at least 3 concerns about standalone script mentioned on the mailing list: http://lists.openembedded.org/pipermail/openembedded-core/2016-May/120921.html http://lists.openembedded.org/pipermail/openembedded-core/2016-May/120922.html http://lists.openembedded.org/pipermail/openembedded-core/2016-May/121472.html I'd propose to fix standalone script generation and send it for review separately. 62288 14 eduard.bartosh 2016-05-18 12:49:58 +0000 The core functionality has been accepted to poky master. I think it's enough to resolve this issue. 62314 15 alexander.kanevskiy 2016-05-19 10:52:27 +0000 For issue mentioned in http://lists.openembedded.org/pipermail/openembedded-core/2016-May/121472.html I had chance to talk directly with Richard. Common understanding that it should be changed to "addtask deploy before do_build after do_install" As for other comments from Chris, we need to revisit how tool is used in wic code. My understanding so far was that it uses python library from sysroot, not the standalone tool from deploy dir. Is it so Ed ? 62315 16 eduard.bartosh 2016-05-19 12:17:56 +0000 wic is calling native bmaptool to generate .bmap file: exec_native_cmd("bmaptool create %s -o %s.bmap" % (full_path, full_path), self.native_sysroot) However, I'm not sure Chris is talking about wic in his comment: You're right that this is very handy, but we hit a problem when building from shared state. This uses do_deploy to deploy bmaptool into DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends on do_populate_sysroot, not do_deploy, so an image build from sstate won't write this script out. I'm also concerned about Richard saying in his comment: I'm not sure if we want to do this in the first place. I think it's better to discuss this with Richard and Chris before sending the patch again. 62316 17 alexander.kanevskiy 2016-05-19 12:46:22 +0000 That's very interesting. Ed, from your code it seems that you're calling native version of the tool and it is executed from native sysroot. thus, it should be correct usage, and it shouldn't be dependant on any content in deploy directory... Thus, dependency on do_populate_sysroot is expected one... Are you able to reproduce same error as Chris having with building image from sstate ? 62317 18 eduard.bartosh 2016-05-19 12:59:33 +0000 No, I didn't try to reproduce it. Chris, can you show us how to reproduce the issue with standalone bmaptool script you're talking about here http://lists.openembedded.org/pipermail/openembedded-core/2016-May/120921.html ?