Bug 5890

Summary: package arch selection logic in rootfs creation is broken
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Nitin Kamble <nitin.a.kamble>
Component: configurationAssignee: Laurentiu Palcu <laurentiu.palcu>
Status: VERIFIED FIXED QA Contact: Lucian Musat <georgex.l.musat>
Severity: major    
Priority: Medium+ CC: alexandru.c.georgescu, sgw
Version: unspecified   
Target Milestone: 1.6 M4   
Hardware: All   
OS: Multiple   
Whiteboard: TC needed
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Nitin Kamble 2014-02-27 00:13:59 UTC
When building multiple BSPs in a single build directory, the package arch selection is going wrong. For example in the log seen below, the alsa-state package is available in 3 flavors viz @core2_64, @corei7_64 and @intel_corei7_64. But for the intel_corei7_64 machine, the package alsa-state@corei7-64 is installed instead of the desired package: alsa-state-0.2.0-r3@intel_corei7_64.

Currently the recipes reproducing this issue are not up-streamed yet because of this issue. the commits for meta-intel to reproduce this issue are available here: http://git.yoctoproject.org/cgit/cgit.cgi/meta-intel-contrib/log/?h=nitin/bsp-ng2

To reproduce build multiple BSPs in a single build directory in this sequence: sugarbay crownbay intel-core2-32 intel-corei7-64


log:

nitin@yocto-hm1:~/build-test-bsps/build-common/tmp/work/intel_corei7_64-poky-linux/core-image-sato/1.0-r0> grep -r alsa-state *
Binary file core-image-sato-1.0/iso/rootfs.img matches
Binary file core-image-sato-1.0/hddimg/rootfs.img matches
installed_pkgs.txt:alsa-state corei7-64
installed_pkgs.txt:alsa-states corei7-64
Binary file pseudo/files.db matches
rootfs/etc/init.d/alsa-state:# Filename: alsa-state
rootfs/etc/init.d/alsa-state:    echo "Usage: /etc/init.d/alsa-state {start|stop|status}"
Binary file rootfs/var/lib/rpm/Nvra matches
Binary file rootfs/var/lib/rpm/Requirename matches
Binary file rootfs/var/lib/rpm/Name matches
Binary file rootfs/var/lib/rpm/log/log.0000000003 matches
Binary file rootfs/var/lib/rpm/Providename matches
Binary file rootfs/var/lib/rpm/Filepaths matches
Binary file rootfs/var/lib/rpm/Packages matches
Binary file rootfs/var/lib/rpm/Basenames matches
temp/log.do_rootfs.70297:Output from alsa-state-0.2.0-r3@corei7_64:
temp/log.do_rootfs.70297: Removing any system startup links for alsa-state ...
temp/log.do_rootfs.70297: Adding system startup for /srv/home/nitin/build-test-bsps/build-common/tmp/work/intel_corei7_64-poky-linux/core-image-sato/1.0-r0/rootfs/etc/init.d/alsa-state.
temp/log.do_rootfs.70297:  alsa-state                                                                    
temp/log.do_rootfs.70297:  alsa-states                                                                   
temp/log.do_rootfs.70297:  25:Installing alsa-states     ######################################## [  3%]
temp/log.do_rootfs.70297: 178:Installing alsa-state      ######################################## [ 23%]
temp/log.do_rootfs.44565:Output from alsa-state-0.2.0-r3@corei7_64:
temp/log.do_rootfs.44565: Removing any system startup links for alsa-state ...
temp/log.do_rootfs.44565: Adding system startup for /srv/home/nitin/build-test-bsps/build-common/tmp/work/intel_corei7_64-poky-linux/core-image-sato/1.0-r0/rootfs/etc/init.d/alsa-state.
temp/log.do_rootfs.44565:  alsa-state                                                                    
temp/log.do_rootfs.44565:  alsa-states                                                                   
temp/log.do_rootfs.44565:  25:Installing alsa-states     ######################################## [  3%]
temp/log.do_rootfs.44565: 168:Installing alsa-state      ######################################## [ 22%]
temp/saved/available_pkgs.txt:alsa-state-0.2.0-r3@core2_64
temp/saved/available_pkgs.txt:alsa-state-0.2.0-r3@corei7_64
temp/saved/available_pkgs.txt:alsa-state-0.2.0-r3@intel_corei7_64
temp/saved/available_pkgs.txt:alsa-state-dbg-0.2.0-r3@core2_64
temp/saved/available_pkgs.txt:alsa-state-dbg-0.2.0-r3@corei7_64
temp/saved/available_pkgs.txt:alsa-state-dbg-0.2.0-r3@intel_corei7_64
temp/saved/available_pkgs.txt:alsa-state-dev-0.2.0-r3@core2_64
temp/saved/available_pkgs.txt:alsa-state-dev-0.2.0-r3@corei7_64
temp/saved/available_pkgs.txt:alsa-state-dev-0.2.0-r3@intel_corei7_64
temp/saved/available_pkgs.txt:alsa-states-0.2.0-r3@core2_64
temp/saved/available_pkgs.txt:alsa-states-0.2.0-r3@corei7_64
temp/saved/available_pkgs.txt:alsa-states-0.2.0-r3@intel_corei7_64
Comment 1 Laurentiu Palcu 2014-02-27 09:24:38 UTC
Do you think it would be possible, in the future, that you provide: poky commit used to replicate the problem, package manager backend used, local.conf and/or bblayers.conf settings? So that I don't have to deduce them all?

thanks
Comment 2 Laurentiu Palcu 2014-02-27 13:24:02 UTC
With the latest master and the meta-intel branch you specified, the 'MACHINE=intel-core2-32 bb -e core-image-sato' fails to fetch the kernel, as below. It built just fine for sugarbay & crownbay though...

Build Configuration:
BB_VERSION        = "1.21.1"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Ubuntu-12.04"
TARGET_SYS        = "i586-poky-linux"
MACHINE           = "intel-core2-32"
DISTRO            = "poky"
DISTRO_VERSION    = "1.5+snapshot-20140227"
TUNE_FEATURES     = "m32 core2"
TARGET_FPU        = ""
meta
meta-yocto
meta-yocto-bsp    = "master:5ab9d7e28f987fba8bab03bb7851c263f746c208"
meta-intel
meta-sugarbay
meta-crownbay     = "test:00008bc26002354f89c377c3ec9732e4ff79aa56"

NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
WARNING: Failed to fetch URL git://git.pokylinux.org/linux-yocto-dev.git;nocheckout=1;branch=standard/base,meta;name=machine,meta, attempting MIRRORS if available
ERROR: Fetcher failure: Unable to find revision 29594404d7fe73cd80eaa4ee8c43dcc53970c60e in branch meta even from upstream
ERROR: Function failed: Fetcher failure for URL: 'git://git.pokylinux.org/linux-yocto-dev.git;nocheckout=1;branch=standard/base,meta;name=machine,meta'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /ssd/work/yp2/build/tmp/work/core2-32-intel-common-poky-linux/linux-yocto-dev/3.13++gitAUTOINC+29594404d7_29594404d7-r0/temp/log.do_fetch.22378
ERROR: Task 73 (/ssd/work/yp2/poky/meta/recipes-kernel/linux/linux-yocto-dev.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 4845 tasks of which 4844 didn't need to be rerun and 1 failed.
No currently running tasks (237 of 4886)

Summary: 1 task failed:
  /ssd/work/yp2/poky/meta/recipes-kernel/linux/linux-yocto-dev.bb, do_fetch
Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
Comment 3 Nitin Kamble 2014-02-27 17:15:09 UTC
My git repository commits:
* poky.git:
** master: commit e47afff7d812c74f8091d85facfdf34d6a1b698e

* meta-intel.git
** nitin/bsp-ng2 commit 00008bc26002354f89c377c3ec9732e4ff79aa56


Additional lines in local.conf

EXTRA_IMAGE_FEATURES += "tools-testapps"
LICENSE_FLAGS_WHITELIST += "commercial"
SRCREV_machine = "${AUTOREV}"
SRCREV_meta = "${AUTOREV}"

I will try adding this kind of information in the future bugs.
Comment 4 Nitin Kamble 2014-02-27 17:19:38 UTC
BTW when the BSP layers need to be setup properly. I see that in the comment #3 meta-sugarbay and meta-crownbay layers are included while building intel-core2-32 BSP, which is not right. These layer requirements are specified in the README files in the meta-intel layer.
Comment 5 Laurentiu Palcu 2014-02-28 11:21:55 UTC
Reproduced it. Working on finding a proper fix.
Comment 6 Laurentiu Palcu 2014-02-28 18:29:58 UTC
I think the problem is related to this commit:

commit 12f47c23df8c109676f66d580d666a3147f3b046
Author: Laurentiu Palcu <laurentiu.palcu@intel.com>
Date:   Mon Feb 10 09:32:04 2014 +0200

    lib/oe/package_manager.py: RpmPM: fix issue with multilib builds
    
    Use python sets instead of lists, to avoid duplicates. When doing a
    multilib build, "smart channel --add" fails because it tries to add
    'all' channel twice.
    
    (From OE-Core rev: 730d675090eec5c03e444f0448d96a52035d0bef)

Apparently, converting a list to a set does not keep the order of the items, as they were in the list... So, reverting this commit should solve the ordering issue. The problem, however, is that I cannot reproduce the original problem that fix was meant for so I can find another fix for it...

So, please try reverting the commit for now and continue with your work.
Comment 7 Nitin Kamble 2014-02-28 19:23:41 UTC
Thanks for the update Laurentiu. I will try reverting the commit you mentioned to proceed with my work.

Nitin
Comment 8 Laurentiu Palcu 2014-03-03 13:49:20 UTC
Patches sent on the ML.
Comment 10 Alexandru Georgescu 2014-07-31 10:58:59 UTC
Assinging this to Lucian for bug verification and TC creation. 
We need to add more this type of scenarios to our testing, e.g. building more bsp's using the same build directory.
As I see from Nitin's original problem, he used different arches to perform the build. Maybe a TC that builds machines with all the flavors @core2_64, @corei7_64, and the others should be sufficent.
Comment 11 Lucian Musat 2014-08-14 10:43:48 UTC
Tested this on sugarbay, crownbay, core2-32 and corei7-64. This is the result:

lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build$ cd tmp/work/intel_corei7_64-poky-linux/core-image-sato/1.0-r0/
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-sato/1.0-r0$ grep -r alsa-state *
Binary file core-image-sato-1.0/hddimg/rootfs.img matches
Binary file core-image-sato-1.0/iso/rootfs.img matches
installed_pkgs.txt:alsa-state corei7-64
installed_pkgs.txt:alsa-states corei7-64
Binary file pseudo/files.db matches
Binary file rootfs/var/lib/rpm/Requirename matches
Binary file rootfs/var/lib/rpm/Providename matches
Binary file rootfs/var/lib/rpm/Filedigests matches
Binary file rootfs/var/lib/rpm/Filepaths matches
Binary file rootfs/var/lib/rpm/Basenames matches
Binary file rootfs/var/lib/rpm/Nvra matches
Binary file rootfs/var/lib/rpm/log/log.0000000004 matches
Binary file rootfs/var/lib/rpm/Name matches
Binary file rootfs/var/lib/rpm/Packages matches
rootfs/etc/init.d/alsa-state:# Filename: alsa-state
rootfs/etc/init.d/alsa-state:    echo "Usage: /etc/init.d/alsa-state {start|stop|status}"
temp/log.do_rootfs.12350:  alsa-state                                                                    
temp/log.do_rootfs.12350:  alsa-states                                                                   
temp/log.do_rootfs.12350:  34:Installing alsa-states     ######################################## [  2%]
temp/log.do_rootfs.12350:Output from alsa-state-0.2.0-r5@corei7_64:
temp/log.do_rootfs.12350: Removing any system startup links for alsa-state ...
temp/log.do_rootfs.12350: 541:Installing alsa-state      ######################################## [ 47%]
temp/log.do_rootfs.12350: Adding system startup for /media/lucian/SSD/work/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-sato/1.0-r0/rootfs/etc/init.d/alsa-state.
temp/saved/available_pkgs.txt:alsa-state-0.2.0-r5@corei7_64
temp/saved/available_pkgs.txt:alsa-state-dbg-0.2.0-r5@corei7_64
temp/saved/available_pkgs.txt:alsa-state-dev-0.2.0-r5@corei7_64
temp/saved/available_pkgs.txt:alsa-states-0.2.0-r5@corei7_64
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-sato/1.0-r0$ cd ../../../../

lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp$ cd work/intel_core2_32-poky-linux/core-image-sato/1.0-r0/
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work/intel_core2_32-poky-linux/core-image-sato/1.0-r0$ grep -r alsa-state *
Binary file core-image-sato-1.0/hddimg/rootfs.img matches
Binary file core-image-sato-1.0/iso/rootfs.img matches
installed_pkgs.txt:alsa-state core2-32
installed_pkgs.txt:alsa-states core2-32
Binary file pseudo/files.db matches
Binary file rootfs/var/lib/rpm/Requirename matches
Binary file rootfs/var/lib/rpm/Providename matches
Binary file rootfs/var/lib/rpm/Filepaths matches
Binary file rootfs/var/lib/rpm/Basenames matches
Binary file rootfs/var/lib/rpm/Nvra matches
Binary file rootfs/var/lib/rpm/log/log.0000000004 matches
Binary file rootfs/var/lib/rpm/Name matches
Binary file rootfs/var/lib/rpm/Packages matches
rootfs/etc/init.d/alsa-state:# Filename: alsa-state
rootfs/etc/init.d/alsa-state:    echo "Usage: /etc/init.d/alsa-state {start|stop|status}"
temp/saved/available_pkgs.txt:alsa-state-0.2.0-r5@core2_32
temp/saved/available_pkgs.txt:alsa-state-dbg-0.2.0-r5@core2_32
temp/saved/available_pkgs.txt:alsa-state-dev-0.2.0-r5@core2_32
temp/saved/available_pkgs.txt:alsa-states-0.2.0-r5@core2_32
temp/log.do_rootfs.20094:  alsa-state                                                                    
temp/log.do_rootfs.20094:  alsa-states                                                                   
temp/log.do_rootfs.20094:  33:Installing alsa-states     ######################################## [  2%]
temp/log.do_rootfs.20094:Output from alsa-state-0.2.0-r5@core2_32:
temp/log.do_rootfs.20094: Removing any system startup links for alsa-state ...
temp/log.do_rootfs.20094: 516:Installing alsa-state      ######################################## [ 39%]
temp/log.do_rootfs.20094: Adding system startup for /media/lucian/SSD/work/poky/build/tmp/work/intel_core2_32-poky-linux/core-image-sato/1.0-r0/rootfs/etc/init.d/alsa-state.
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work/intel_core2_32-poky-linux/core-image-sato/1.0-r0$ cd ../../../

lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work$ cd 
all-poky-linux/                    corei7-64-intel-common-poky-linux/ intel_corei7_64-poky-linux/
core2-32-emgd-poky-linux/          corei7-64-poky-linux/              sugarbay-poky-linux/
core2-32-intel-common-poky-linux/  crownbay-poky-linux/               x86_64-linux/
core2-32-poky-linux/               intel_core2_32-poky-linux/         x86_64-poky-linux/
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work$ cd sugarbay-poky-linux/core-image-sato/1.0-r0/
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work/sugarbay-poky-linux/core-image-sato/1.0-r0$ 
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work/sugarbay-poky-linux/core-image-sato/1.0-r0$ 
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work/sugarbay-poky-linux/core-image-sato/1.0-r0$ 
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work/sugarbay-poky-linux/core-image-sato/1.0-r0$ grep -r alsa-state *
Binary file core-image-sato-1.0/hddimg/rootfs.img matches
Binary file core-image-sato-1.0/iso/rootfs.img matches
installed_pkgs.txt:alsa-state corei7-64
installed_pkgs.txt:alsa-states corei7-64
Binary file pseudo/files.db matches
Binary file rootfs/var/lib/rpm/Requirename matches
Binary file rootfs/var/lib/rpm/Providename matches
Binary file rootfs/var/lib/rpm/Filedigests matches
Binary file rootfs/var/lib/rpm/Filepaths matches
Binary file rootfs/var/lib/rpm/Basenames matches
Binary file rootfs/var/lib/rpm/Nvra matches
Binary file rootfs/var/lib/rpm/log/log.0000000004 matches
Binary file rootfs/var/lib/rpm/Name matches
Binary file rootfs/var/lib/rpm/Packages matches
rootfs/etc/init.d/alsa-state:# Filename: alsa-state
rootfs/etc/init.d/alsa-state:    echo "Usage: /etc/init.d/alsa-state {start|stop|status}"
temp/saved/available_pkgs.txt:alsa-state-0.2.0-r5@corei7_64
temp/saved/available_pkgs.txt:alsa-state-dbg-0.2.0-r5@corei7_64
temp/saved/available_pkgs.txt:alsa-state-dev-0.2.0-r5@corei7_64
temp/saved/available_pkgs.txt:alsa-states-0.2.0-r5@corei7_64
temp/log.do_rootfs.11307:  alsa-state                                                                    
temp/log.do_rootfs.11307:  alsa-states                                                                   
temp/log.do_rootfs.11307:  25:Installing alsa-states     ######################################## [  2%]
temp/log.do_rootfs.11307:Output from alsa-state-0.2.0-r5@corei7_64:
temp/log.do_rootfs.11307: Removing any system startup links for alsa-state ...
temp/log.do_rootfs.11307: 716:Installing alsa-state      ######################################## [ 64%]
temp/log.do_rootfs.11307: Adding system startup for /media/lucian/SSD/work/poky/build/tmp/work/sugarbay-poky-linux/core-image-sato/1.0-r0/rootfs/etc/init.d/alsa-state.
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work/sugarbay-poky-linux/core-image-sato/1.0-r0$ cd ../../../

lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work$ cd crownbay-poky-linux/core-image-sato/1.0-r0/
lucian@lucian-OptiPlex-7010:/media/lucian/SSD/work/poky/build/tmp/work/crownbay-poky-linux/core-image-sato/1.0-r0$ grep -r alsa-state *
Binary file core-image-sato-1.0/hddimg/rootfs.img matches
Binary file core-image-sato-1.0/iso/rootfs.img matches
installed_pkgs.txt:alsa-state core2-32
installed_pkgs.txt:alsa-states core2-32
Binary file pseudo/files.db matches
Binary file rootfs/var/lib/rpm/Requirename matches
Binary file rootfs/var/lib/rpm/Providename matches
Binary file rootfs/var/lib/rpm/Filepaths matches
Binary file rootfs/var/lib/rpm/Basenames matches
Binary file rootfs/var/lib/rpm/Nvra matches
Binary file rootfs/var/lib/rpm/log/log.0000000003 matches
Binary file rootfs/var/lib/rpm/Name matches
Binary file rootfs/var/lib/rpm/Packages matches
rootfs/etc/init.d/alsa-state:# Filename: alsa-state
rootfs/etc/init.d/alsa-state:    echo "Usage: /etc/init.d/alsa-state {start|stop|status}"
temp/saved/available_pkgs.txt:alsa-state-0.2.0-r5@core2_32
temp/saved/available_pkgs.txt:alsa-state-dbg-0.2.0-r5@core2_32
temp/saved/available_pkgs.txt:alsa-state-dev-0.2.0-r5@core2_32
temp/saved/available_pkgs.txt:alsa-states-0.2.0-r5@core2_32
temp/log.do_rootfs.12621:  alsa-state                                                                    
temp/log.do_rootfs.12621:  alsa-states                                                                   
temp/log.do_rootfs.12621:  24:Installing alsa-states     ######################################## [  3%]
temp/log.do_rootfs.12621:Output from alsa-state-0.2.0-r5@core2_32:
temp/log.do_rootfs.12621: Removing any system startup links for alsa-state ...
temp/log.do_rootfs.12621: 289:Installing alsa-state      ######################################## [ 42%]
temp/log.do_rootfs.12621: Adding system startup for /media/lucian/SSD/work/poky/build/tmp/work/crownbay-poky-linux/core-image-sato/1.0-r0/rootfs/etc/init.d/alsa-state.

It seems fine now so I am marking this as verified.