Bug 12264 - IMAGE_FSTYPES += "... wic.bmap ..." bmaptool freaking out
Summary: IMAGE_FSTYPES += "... wic.bmap ..." bmaptool freaking out
Status: RESOLVED NOTABUG
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: deployment (show other bugs)
Version: 2.4
Hardware: x86 Multiple
: Medium normal
Target Milestone: 2.5 M1
Assignee: Robert Berger
QA Contact:
URL:
Whiteboard: Backport 2.3.3 and 2.4.1?
Depends on:
Blocks:
 
Reported: 2017-10-21 17:42 UTC by Robert Berger
Modified: 2017-11-06 09:44 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Berger 2017-10-21 17:42:01 UTC
I am using something like this here [1]

The problem seems to be:

IMAGE_FSTYPES += "tar.bz2 wic wic.bmap"
                              ^^^^^^^^
wic alone (without bmap) seems to work, so I'll remove wic.bmap for now.

A failed build looks something like this:

https://pastebin.com/XXURHxKY

[1] https://github.com/RobertBerger/meta-mainline/blob/rocko-training-v4.9.x/u-boot-wic-bsp/conf/machine/common.inc#L16

I tried to 
1) manually install bmap-tools
2) do_image_wic[depends] += "...bmap-tools-native:do_populate_sysroot"
3) bitbake bmap-tools-native:do_addto_recipe_sysroot

none of which worked
Comment 1 Joshua Lock 2017-10-26 14:54:04 UTC
Have you observed this outside of a yocto-autobuilder build? Are you running the latest yocto-autobuilder code? Specifically, do you have this commit?

http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder/commit/?id=7c77c01d4818f0cdb9a325a865379bb673b26bcc

(and a few follow on patches to fix that)

bmap-tools was ported to Python 3 in the rocko timeline which resulted in some incompatibilities with the yocto-autobuilder, more detail in: 

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12076
Comment 2 Robert Berger 2017-10-26 15:18:08 UTC
Yes I have observed this outside of autobuilder as well as with autobuilder.

I mean even if I run bitbake from the commandline myself (no autobuilder involved) it still happens.

I am running it from within my docker build container, where I build everything.

the build output from above is named ...yocto-autobuilder because it's my yocto-autobuilder container.
Comment 3 Joshua Lock 2017-10-26 15:59:11 UTC
Thanks for the quick reply.

The IMAGE_FSTYPES += notation is common throughout the YP core machines, each generates a wic.bmap without issue.

There were some changes to u-boot conversion type chaining in Rocko, I wonder if something was missed. 

Can you share what commitish of what branch you're using, your build host distro and ideally some reproduction steps to replicate your setup (I tried cloning the branch of the layer you link to but there were dangling bbappends when I added meta-mainline and meta-mainline/u-boot-wic-bsp layers).
Comment 4 Robert Berger 2017-10-26 21:50:29 UTC
You can try with this one:

https://github.com/RobertBerger/meta-mainline/blob/rocko-training-v4.9.x/u-boot-wic-bsp/yocto-autobuilder/res-custom-rocko-multi-v7-core-image-minimal-uboot-vic.conf

There is no need to build for 3 machines as this autobuilder config would try, but 'machine': 'imx6q-phytec-mira-rdk-nand' is sufficient.

This should build out of the box (it does for me with and without autobuilder).

To make it break just uncomment line 17 and comment line 18 in order to put also wic.bmap in the mix.

https://github.com/RobertBerger/meta-mainline/blob/rocko-training-v4.9.x/u-boot-wic-bsp/conf/machine/common.inc#L17

... I use a bastardized poky pre-release from 10-02 with patches on top ...

https://github.com/RobertBerger/poky/tree/2017-10-02-rocko-2.4-pre-release

I had to use u-boot 2017.09 since with the gcc version which comes with rocko version before don't work;)

Here are my patches to u-boot 2017.09

https://github.com/RobertBerger/meta-mainline/tree/rocko-training-v4.9.x/u-boot-wic-bsp/recipes-bsp/u-boot

If you know what changes were made to u-boot please point me there, sine I might have missed this.

Assuming all this works on your end and we can't figure out the problem I can also give you my docker container which is my default build environment for everything and some oldish autobuilder - commit f9b4e02730a5e712ee9085e2480f9a3b8b00f56d plus some patches.
Comment 5 Robert Berger 2017-10-26 22:00:13 UTC
I think I'm on this commit plus some patches:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=rocko&id=fcdfe09d9cc4e2977bd602dfd18f729cd9f6d73d
Comment 6 Stephen K Jolley 2017-10-27 18:29:24 UTC
Joshua,

It appears the question has been answered.  If not please restate the question.

Thanks,
Comment 7 Joshua Lock 2017-10-30 14:14:34 UTC
(In reply to comment #4)
> You can try with this one:
> 
> https://github.com/RobertBerger/meta-mainline/blob/rocko-training-v4.9.x/u-
> boot-wic-bsp/yocto-autobuilder/res-custom-rocko-multi-v7-core-image-minimal-
> uboot-vic.conf
> 
> There is no need to build for 3 machines as this autobuilder config would
> try, but 'machine': 'imx6q-phytec-mira-rdk-nand' is sufficient.
> 
> This should build out of the box (it does for me with and without
> autobuilder).
> 
> To make it break just uncomment line 17 and comment line 18 in order to put
> also wic.bmap in the mix.
> 
> https://github.com/RobertBerger/meta-mainline/blob/rocko-training-v4.9.x/u-
> boot-wic-bsp/conf/machine/common.inc#L17
> 
> ... I use a bastardized poky pre-release from 10-02 with patches on top ...
> 
> https://github.com/RobertBerger/poky/tree/2017-10-02-rocko-2.4-pre-release

Could you please try and reproduce with master?

> I had to use u-boot 2017.09 since with the gcc version which comes with
> rocko version before don't work;)
> 
> Here are my patches to u-boot 2017.09
> 
> https://github.com/RobertBerger/meta-mainline/tree/rocko-training-v4.9.x/u-
> boot-wic-bsp/recipes-bsp/u-boot

Yikes! Is that just u-boot for certain machines? Or is the u-boot we ship with the Rocko release generally broken? If so has this been reported as a separate bug?
Comment 8 Robert Berger 2017-10-30 14:59:17 UTC
Hi,

(In reply to comment #7)
> (In reply to comment #4)
> 
> Could you please try and reproduce with master?

I will eventually update to rocko master, but at the moment I am pretty busy with other things and the fact that it does not work with wic.bmap, but that it works with wic is not a show stopper for me.

I need to prepare/deliver a couple of trainings and my rocko pre-release is fine  for that.

I'll upgrade to the rocko release as soon as I'll find some time but I'll deliver Yocto trainings in weeks 47, 50 and 51 so I am not sure I'll have time to upgrade and test everything sometime in between.

Anyhow, I will update this bug accordingly.

> 
> > I had to use u-boot 2017.09 since with the gcc version which comes with
> > rocko version before don't work;)
> > 
> > Here are my patches to u-boot 2017.09
> > 
> > https://github.com/RobertBerger/meta-mainline/tree/rocko-training-v4.9.x/u-
> > boot-wic-bsp/recipes-bsp/u-boot

Those patches are unrelated to the gcc/u-boot problem. It's just some extra stuff so I can create an SD card which has u-boot + uEnv.txt on it plus some rootfs (which is not used out of the box).

> 
> Yikes! Is that just u-boot for certain machines? Or is the u-boot we ship
> with the Rocko release generally broken? If so has this been reported as a
> separate bug?

The u-boot shipped with rocko is fine since it is 2017.09. 

Only if someone (crazy like me) gets the idea to use an older version of u-boot it will not work with the gcc which comes with rocko. I tried on 3 different boards with 2017.05, 2017.07 and 2017.09 and only the 2017.09 works on all of them.
Comment 9 Joshua Lock 2017-10-30 16:37:10 UTC
(In reply to comment #8)
> Hi,
> 
> (In reply to comment #7)
> > (In reply to comment #4)
> > 
> > Could you please try and reproduce with master?
> 
> I will eventually update to rocko master, but at the moment I am pretty busy
> with other things and the fact that it does not work with wic.bmap, but that
> it works with wic is not a show stopper for me.
> 
> I need to prepare/deliver a couple of trainings and my rocko pre-release is
> fine  for that.
> 
> I'll upgrade to the rocko release as soon as I'll find some time but I'll
> deliver Yocto trainings in weeks 47, 50 and 51 so I am not sure I'll have
> time to upgrade and test everything sometime in between.
> 
> Anyhow, I will update this bug accordingly.

Understood. 

We are using the same IMAGE_FSTYPES += "wic.bmap" semantics in meta-yocto-bsp without issue, so I'm curious to understand where the issue lies.

OOI which host OS are you using in your build container?

> > Yikes! Is that just u-boot for certain machines? Or is the u-boot we ship
> > with the Rocko release generally broken? If so has this been reported as a
> > separate bug?
> 
> The u-boot shipped with rocko is fine since it is 2017.09. 
> 
> Only if someone (crazy like me) gets the idea to use an older version of
> u-boot it will not work with the gcc which comes with rocko. I tried on 3
> different boards with 2017.05, 2017.07 and 2017.09 and only the 2017.09
> works on all of them.

ah, thanks for clarifying. I am not very familiar with u-boot.
Comment 10 Robert Berger 2017-10-30 17:20:50 UTC
Hi,

> 
> We are using the same IMAGE_FSTYPES += "wic.bmap" semantics in
> meta-yocto-bsp without issue, so I'm curious to understand where the issue
> lies.
> 
> OOI which host OS are you using in your build container?

the short answer:

I use Ubuntu 16 as a host distro and the container is also something like Ubuntu 16

the long answer:

reliableembeddedsystems/yocto-autobuilder:pyro-2017-05-29

the Dockerfile for it:

https://github.com/RobertBerger/yocto-autobuilder/blob/master/dockerfile/Dockerfile

depends on

https://github.com/RobertBerger/yocto/blob/master/dockerfile/Dockerfile

depends on 

https://github.com/RobertBerger/ubuntu-base/blob/master/dockerfile/Dockerfile

depends on

phusion/baseimage:0.9.22

depends on

ubuntu:16.04[1]

[1] https://github.com/phusion/baseimage-docker/blob/master/image/Dockerfile

hehe ;)

you asked for it;)
Comment 11 Robert Berger 2017-11-03 12:50:15 UTC
I am trying today's rocko branch:

commit f7b90ab3eaf832bd81f3efc1dab4dcf6863ac284
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Sat Oct 21 12:50:07 2017 +0100

On top of it I have (at the moment):

git diff --stat official-upstream/rocko 2017-11-03-rocko-2.4+
 meta/recipes-core/packagegroups/nativesdk-packagegroup-sdk-host.bb                      |   1 -
 scripts/lib/bsp/substrate/target/arch/arm/conf/machine/machine.conf                     |  43 ++++++--
 scripts/lib/bsp/substrate/target/arch/common/recipes-kernel/linux/kernel-list.noinstall |   6 ++
 scripts/lib/bsp/substrate/target/arch/common/recipes-kernel/linux/linux-yocto-custom.bb |  10 ++
 scripts/pkgdataui.glade                                                                 | 335 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 scripts/pkgdataui.py                                                                    | 241 +++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 627 insertions(+), 9 deletions(-)

which does not seem too bad and I don't think is in any way related to this issue.

=====

commit 298820ee45781a5326216692274bf4a79cef1050
Author: Robert Berger <robert.berger@ReliableEmbeddedSystems.com>
Date:   Thu Oct 26 18:29:25 2017 +0300

    poky: LICENSE in packagegroup makes no sense - removed
    
    Signed-off-by: Robert Berger <robert.berger@ReliableEmbeddedSystems.com>

 meta/recipes-core/packagegroups/nativesdk-packagegroup-sdk-host.bb | 1 -
 1 file changed, 1 deletion(-)

commit fa4a2f97803b64eb551c2496db37afdc913df7c7
Author: Robert Berger <robert.berger@ReliableEmbeddedSystems.com>
Date:   Tue Oct 3 23:58:47 2017 +0300

    rocko: linux-yocto-custom.bb no more double inclusions
    
    Signed-off-by: Robert Berger <robert.berger@ReliableEmbeddedSystems.com>

 scripts/lib/bsp/substrate/target/arch/common/recipes-kernel/linux/linux-yocto-custom.bb | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

commit bc077c1e4db13504cc6ee30af4696d177cffa3ad
Author: Robert Berger <robert.berger@ReliableEmbeddedSystems.com>
Date:   Mon Oct 2 08:54:44 2017 +0300

    [PATCH] hacked bsp-tool, fixed also user-features
    
    Signed-off-by: Robert Berger <robert.berger@ReliableEmbeddedSystems.com>

 scripts/lib/bsp/substrate/target/arch/arm/conf/machine/machine.conf                     | 43 +++++++++++++++++++++++++++++++++++--------
 scripts/lib/bsp/substrate/target/arch/common/recipes-kernel/linux/kernel-list.noinstall |  6 ++++++
 scripts/lib/bsp/substrate/target/arch/common/recipes-kernel/linux/linux-yocto-custom.bb | 17 +++++++++++++++--
 3 files changed, 56 insertions(+), 10 deletions(-)

commit 2d18f25e86dcd12d65b1ed17cad6e3dfff8bc0ef
Author: Robert Berger <robert.berger@ReliableEmbeddedSystems.com>
Date:   Fri Nov 3 14:12:15 2017 +0200

    Subject: [PATCH] pkgdataui is a Python 3/GObject Introspection/GTK+ 3 tool to  browse the pkgdata database at your leisure. By being graphical it is easier  to explore and can follow links between packages.
    
    This is very much a work in progress, so be gentle and patches are welcome.
    
    Signed-off-by: Ross Burton <ross.burton-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
    Signed-off-by: Robert Berger <robert.berger@ReliableEmbeddedSystems.com>

=========
Comment 12 Robert Berger 2017-11-04 21:23:36 UTC
I just gave it a quick try with the latest and greatest rocko and it looks promising:

ls tmp/deploy/images/imx6q-phytec-mira-rdk-nand/*wic*
tmp/deploy/images/imx6q-phytec-mira-rdk-nand/core-image-minimal-imx6q-phytec-mira-rdk-nand-20171104210109.rootfs.wic.md5sum
tmp/deploy/images/imx6q-phytec-mira-rdk-nand/core-image-minimal-imx6q-phytec-mira-rdk-nand-20171104211820.rootfs.wic
tmp/deploy/images/imx6q-phytec-mira-rdk-nand/core-image-minimal-imx6q-phytec-mira-rdk-nand-20171104211820.rootfs.wic.bmap
tmp/deploy/images/imx6q-phytec-mira-rdk-nand/core-image-minimal-imx6q-phytec-mira-rdk-nand.wic
tmp/deploy/images/imx6q-phytec-mira-rdk-nand/core-image-minimal-imx6q-phytec-mira-rdk-nand.wic.bmap

close to midnight and I need to get up at 5 in the morning to catch my plane, so I'll make some more tests during next week or in a week or so
Comment 13 Robert Berger 2017-11-05 17:27:11 UTC
Now in the hotel I had some time to check a bit more.

find | grep wic.bmap
./am335x-phytec-wega/core-image-minimal-am335x-phytec-wega-20171105111011.rootfs.wic.bmap
./am335x-phytec-wega/core-image-minimal-am335x-phytec-wega.wic.bmap
./am335x-phytec-wega/core-image-minimal-am335x-phytec-wega-20171105111011.rootfs.wic.bmap.md5sum
./imx6q-phytec-mira-rdk-nand/core-image-minimal-imx6q-phytec-mira-rdk-nand-20171105110755.rootfs.wic.bmap.md5sum
./imx6q-phytec-mira-rdk-nand/core-image-minimal-imx6q-phytec-mira-rdk-nand-20171105110755.rootfs.wic.bmap
./imx6q-phytec-mira-rdk-nand/core-image-minimal-imx6q-phytec-mira-rdk-nand.wic.bmap
./beagle-bone-black/core-image-minimal-beagle-bone-black-20171105111106.rootfs.wic.bmap
./beagle-bone-black/core-image-minimal-beagle-bone-black-20171105111106.rootfs.wic.bmap.md5sum
./beagle-bone-black/core-image-minimal-beagle-bone-black.wic.bmap

Also autobuilder can build .wic.bmap now

Did not actually create an SD card with it, but I think we can close the bug.
Comment 14 Joshua Lock 2017-11-06 09:44:21 UTC
Thanks.