Bug 4071

Summary: Building an image fails at quilt-native using BA for 1.4M5.rc1
Product: [Build System, Metadata & Runtime] Runtime Reporter: Stefan Stanacar <sstncr>
Component: build-applianceAssignee: Cristian Iorga <cristian.iorga>
Status: VERIFIED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: Martin.Jansa, sgw
Version: 1.4   
Target Milestone: 1.4 M6   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---
Attachments:
Description Flags
run.sstate_create_package
none
log.do_populate_lic none

Description Stefan Stanacar 2013-03-21 14:44:47 UTC
Created attachment 1131 [details]
run.sstate_create_package

Building core-image-minimal, MACHINE=qemux86 using BA (1.4M5.rc1 http://autobuilder.yoctoproject.org/pub/nightly/20130321-2/machines/build-appliance/ ) fails at quilt-native:

DEBUG: Executing python function sstate_task_postfunc
DEBUG: Staging files from /home/builder/poky/build/tmp/work/x86_64-linux/quilt-native/0.60-r0/license-destdir to /home/builder/poky/build/tmp/deploy/licenses
DEBUG: Preparing tree /home/builder/poky/build/tmp/work/x86_64-linux/quilt-native/0.60-r0/license-destdir for packaging at /home/builder/poky/build/tmp/work/x86_64-linux/quilt-native/0.60-r0/sstate-build-populate-lic/license-destdir
Removing hardcoded paths from sstate package: 'grep -Irl /home/builder/poky/build/tmp/sysroots /home/builder/poky/build/tmp/work/x86_64-linux/quilt-native/0.60-r0/sstate-build-populate-lic/ | xargs grep -l -e '/home/builder/poky/build/tmp/sysroots' | tee /home/builder/poky/build/tmp/work/x86_64-linux/quilt-native/0.60-r0/sstate-build-populate-lic/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/builder/poky/build/tmp/sysroots:FIXMESTAGINGDIR:g''
DEBUG: Executing shell function sstate_create_package
mktemp: too few X's in template `/home/builder/poky/build/sstate-cache/poky'
DEBUG: Python function sstate_task_postfunc finished
ERROR: Function failed: sstate_create_package (see /home/builder/poky/build/tmp/work/x86_64-linux/quilt-native/0.60-r0/temp/log.do_populate_lic.1425 for further information)
Comment 1 Stefan Stanacar 2013-03-21 14:45:14 UTC
Created attachment 1132 [details]
log.do_populate_lic
Comment 2 Stefan Stanacar 2013-03-21 14:58:16 UTC
I think I know where the issue lies - the recent changes made to the distro idenfier used for creating the sstate dir and nativelibstring from lsb_release -a. Related commits: e5f3b992d7edc7b5e2467865a9fad738fbeaf40f b892099902402c1179d7183b6ac97ca517faae7a
 
On BA:

qemux86-64:~/poky/build$ pwd
/home/builder/poky/build
qemux86-64:~/poky/build$ ls -l sstate-cache/
total 1
drwxr-xr-x 4 builder builder 1024 Mar 21 14:48 poky?-1.3+snapshot-20130321?

Notice the extra ? 

qemux86-64:~/poky/build$ lsb_release -a
LSB Version:	core-4.1-noarch:core-4.1-x86_64
Distributor ID:	poky
Description:	Poky 8.0 (Yocto Project 1.3 Reference Distro) 1.3+snapshot-20130321
Release:	1.3+snapshot-20130321
Codename:	n/a
qemux86-64:~/poky/build$ 

run.sstate_create_package has:

TFILE=`mktemp /home/builder/poky/build/sstate-cache/poky
-1.3+snapshot-20130321
/49/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-3-498a8ea397c67282c6b57422b09cc3a9_populate-lic.tgz.XXXXXXXX`

that's why mktemp fails. I think someone owes me a couple of hours :)
Comment 3 Martin Jansa 2013-03-21 15:15:13 UTC
What does bitbake -e show as NATIVELSBSTRING?
Comment 4 Stefan Stanacar 2013-03-21 16:13:51 UTC
NATIVELSBSTRING="poky \
-1.3+snapshot-20130321"
Comment 5 Cristian Iorga 2013-03-26 17:21:52 UTC
Investigation started.
Comment 6 Stefan Stanacar 2013-03-27 15:40:52 UTC
It's important to know that the Build Appliance image built by Autobuilder uses the latest poky rev but if you build your own BA the poky clone inside it's older (the BA recipes hardcodes a certain poky rev)

The BA image used by me before (comment #1) was built by AutoBuilder. Head for 1.4M5.rc1 was 9ffee9add2d0380f4025e7a678a166f25a7d659e

Same thing happens (quilt-native fails with the same error as before) with latest BA image from AB: http://autobuilder.yoctoproject.org/pub/nightly/20130321-2/machines/build-appliance/ 
poky rev: bdca49274173153b1b0a72887de6321b4756b684


The root cause is as I suspected the wrong NATIVELSBSTRING, and that's looks wrong because of the code in meta/lib/oe/lsb.py (and release_dict and release_dict_file functions)

The code tries to use lsb_release -a and if that fails (and in our case it fails because even if lsb_release exists it return 3 becasue we don't have a compliant /etc/lsb-release file, specifically we are missing DISTRIB_CODENAME, see /bin/lsb_release script) it falls back to reading from /etc/lsb_release. However it's the way it reads it:

with open('/etc/lsb-release') as f:
                for line in f:
                    key, value = line.split("=", 1)

Try this in BA: 

qemux86-64:~/poky/build$ python -c 'with open("/etc/lsb-release") as f:
                for line in f:
                    key, value = line.split("=", 1)
                    print key, value

'
Output:

LSB_VERSION "core-4.1-noarch:core-4.1-x86_64"

DISTRIB_ID poky

DISTRIB_RELEASE 1.3+snapshot-20130327

DISTRIB_DESCRIPTION "Poky 8.0 (Yocto Project 1.3 Reference Distro) 1.3+snapshot-20130312312327"

There are new lines there and that leads to:
# bitbake -e |grep -A 2 -E "^NATIVELSB"
NATIVELSBSTRING="poky \
-1.3+snapshot-20130327"
#

qemux86-64:~/poky/build$ cat /etc/lsb-release
LSB_VERSION="core-4.1-noarch:core-4.1-x86_64"
DISTRIB_ID=poky
DISTRIB_RELEASE=1.3+snapshot-20130327
DISTRIB_DESCRIPTION="Poky 8.0 (Yocto Project 1.3 Reference Distro) 1.3+snapshot-20130312312327"
qemux86-64:~/poky/build$ lsb_release -a
LSB Version:	core-4.1-noarch:core-4.1-x86_64
Distributor ID:	poky
Description:	Poky 8.0 (Yocto Project 1.3 Reference Distro) 1.3+snapshot-20130312312327
Release:	1.3+snapshot-20130327
Codename:	n/a
qemux86-64:~/poky/build$ echo $?
3

Now let's add Codename


root@qemux86-64:~# echo "DISTRIB_CODENAME=mycode" >> /etc/lsb-release
root@qemux86-64:~# lsb_release -a
LSB Version:	core-4.1-noarch:core-4.1-x86_64
Distributor ID:	poky
Description:	Poky 8.0 (Yocto Project 1.3 Reference Distro) 1.3+snapshot-20130312312327
Release:	1.3+snapshot-20130327
Codename:	mycode
root@qemux86-64:~# echo $?
0
root@qemux86-64:~# su - builder
mesg: /dev/pts/0: Operation not permitted
qemux86-64:~$ cd poky/   
qemux86-64:~/poky$ source oe-init-build-env 
qemux86-64:~/poky/build$ bitbake -e |grep -A 2 -E "^NATIVELSB"
NATIVELSBSTRING="poky-1.3+snapshot-20130327"



Solutions:
 1 - instead of lsb_release -a in lsb.py use lsb_release -ri (return code is always 0 in this case, because it looks at ID and Release only and that's what we use for NATIVELSBSTRING)
 2 - add DISTRIB_CODENAME in /etc/lsb-release
 3 - fix the alternative way (release_dict_file) when reading file for constructing NATIVELSBSTRING
Comment 7 Cristian Iorga 2013-04-03 06:41:44 UTC
Fixed with commits:
3ee9d3631920e68d2970aa9f1f851d48191892cb,
e9866ee0ff17f749d824fa40944f220a2a7f05b6,
bacdb99a424fd96e5e60b740e1910163daa74aef.
Comment 8 Stefan Stanacar 2013-04-12 09:33:45 UTC
Verified in  1.4_M6.rc1 / dylan:dab237bc6a7679dfeaa981c14736aedc7b04ecb9