Bug 5132

Summary: pseudo: Different file attributes detected by buildhistory
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Martin Jansa <Martin.Jansa>
Component: coreAssignee: Seebs <seebs>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: meta.mr.watcher, meta.watcher, richard.purdie, sgw
Version: 1.4   
Target Milestone: 1.5.1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
procps-pseudo-4.tgz
none
procps-pseudo-5.tgz
none
proposed patch for the atexit() approach none

Description Martin Jansa 2013-09-06 07:37:36 UTC
Steps to reproduce:
distro-less config (internal toolchain)

echo 'CORE_IMAGE_EXTRA_INSTALL += "procps"' >> conf/local.conf

bitbake -k core-image-minimal
bitbake -c cleansstate procps; bitbake -k core-image-minimal

Compare files-in-image.txt in buildhistory

--- from-scratch3/buildhistory/images/qemux86_64/eglibc/core-image-minimal/files-in-image.txt   2013-09-05 20:34:42.326637279 +0200
+++ from-scratch4/buildhistory/images/qemux86_64/eglibc/core-image-minimal/files-in-image.txt   2013-09-06 09:32:28.336762345 +0200
@@ -731,7 +731,7 @@
--r-xr-xr-x root       root            12360 ./usr/bin/slabtop
+-rwxr-xr-x root       root            12360 ./usr/bin/slabtop

--rwxr-xr-x root       root             5040 ./usr/bin/uptime.procps
+-r-xr-xr-x root       root             5040 ./usr/bin/uptime.procps

-crw-rw---- root       plugdev           10, 20:34 ./dev/apm_bios
-crw-rw--w- root       tty                5, 20:34 ./dev/console
....
+crw-rw---- root       plugdev           10, 01:39 ./dev/apm_bios
+crw-rw--w- root       tty                5, 01:39 ./dev/console
....

Lets try once more to be sure:
--- from-scratch4/buildhistory/images/qemux86_64/eglibc/core-image-minimal/files-in-image.txt   2013-09-06 09:32:28.336762345 +0200
+++ from-scratch5/buildhistory/images/qemux86_64/eglibc/core-image-minimal/files-in-image.txt   2013-09-06 01:39:38.151078371 +0200
@@ -681,7 +681,7 @@
--rwxr-xr-x root       root            12360 ./usr/bin/slabtop
+-r-xr-xr-x root       root            12360 ./usr/bin/slabtop

--r-xr-xr-x root       root             8512 ./usr/bin/free.procps
+-rwxr-xr-x root       root             8512 ./usr/bin/free.procps


all /dev files have different creation date, but we can ignore them. Interesting part is write permission flipping on and off and it happens sometimes on different files (not just slabtop and free.procps).

I'll attach pseudo directory from procps WORKDIR, this time built with PSEUDO_OPTS=-l added to FAKEROOTBASEENV from both builds.
Comment 1 Martin Jansa 2013-09-06 07:38:07 UTC
Created attachment 1466 [details]
procps-pseudo-4.tgz
Comment 2 Martin Jansa 2013-09-06 07:38:26 UTC
Created attachment 1467 [details]
procps-pseudo-5.tgz
Comment 3 Martin Jansa 2013-09-06 08:31:12 UTC
It looks like -rwxr-xr-x is used everywhere except files-in-image.txt:

OE @ ~/oe-core/tmp-eglibc $ ll from-scratch*/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop
-rwxr-xr-x 2 bitbake bitbake 12360 Sep  5 19:50 from-scratch1/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop*
-rwxr-xr-x 2 bitbake bitbake 12360 Sep  5 20:16 from-scratch2/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop*
-rwxr-xr-x 2 bitbake bitbake 12360 Sep  5 20:33 from-scratch3/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop*
-rwxr-xr-x 2 bitbake bitbake 12360 Sep  6 09:30 from-scratch4/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop*
-rwxr-xr-x 2 bitbake bitbake 12360 Sep  6 01:34 from-scratch5/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop*

OE @ ~/oe-core/tmp-eglibc $ ll from-scratch*/work/x86_64-oe-linux/procps/3.2.8-r11/packages-split/procps/usr/bin/slabtop
-rwxr-xr-x 2 bitbake bitbake 12360 Sep  5 19:50 from-scratch1/work/x86_64-oe-linux/procps/3.2.8-r11/packages-split/procps/usr/bin/slabtop*
-rwxr-xr-x 2 bitbake bitbake 12360 Sep  5 20:16 from-scratch2/work/x86_64-oe-linux/procps/3.2.8-r11/packages-split/procps/usr/bin/slabtop*
-rwxr-xr-x 2 bitbake bitbake 12360 Sep  5 20:33 from-scratch3/work/x86_64-oe-linux/procps/3.2.8-r11/packages-split/procps/usr/bin/slabtop*
-rwxr-xr-x 2 bitbake bitbake 12360 Sep  6 09:30 from-scratch4/work/x86_64-oe-linux/procps/3.2.8-r11/packages-split/procps/usr/bin/slabtop*
-rwxr-xr-x 2 bitbake bitbake 12360 Sep  6 01:34 from-scratch5/work/x86_64-oe-linux/procps/3.2.8-r11/packages-split/procps/usr/bin/slabtop*

OE @ ~/oe-core/tmp-eglibc $ ll from-scratch*/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop
-rwxr-xr-x 1 bitbake bitbake 12360 Sep  5 19:50 from-scratch1/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop*
-rwxr-xr-x 1 bitbake bitbake 12360 Sep  5 20:16 from-scratch2/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop*
-rwxr-xr-x 1 bitbake bitbake 12360 Sep  5 20:33 from-scratch3/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop*
-rwxr-xr-x 1 bitbake bitbake 12360 Sep  6 09:30 from-scratch4/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop*
-rwxr-xr-x 1 bitbake bitbake 12360 Sep  6 01:34 from-scratch5/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop*


OE @ ~/oe-core/tmp-eglibc $ sqlite3 core-image-minimal-pseudo-4/files.db
sqlite> select * from files where path like '%slabtop%';
206|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|2081|19244656|0|0|33261|0|0

OE @ ~/oe-core/tmp-eglibc $ sqlite3 core-image-minimal-pseudo-5/files.db
sqlite> select * from files where path like '%slabtop%';
206|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|2081|18927165|0|0|33133|0|0

OE @ ~/oe-core/tmp-eglibc $ sqlite3 core-image-minimal-pseudo-4/logs.db
sqlite> select * from logs where path like '%slabtop%';
16093|1378452666|6|5415||2081|19244656|33188|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|opkg-cl|6
16094|1378452666|14|5415||2081|19244656|33188|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||1026|1026|2|opkg-cl|6
16095|1378452666|3|5415||2081|19244656|33188|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|opkg-cl|6
16096|1378452666|2|5415||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|opkg-cl|6
54059|1378452693|16|11512||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|3
72852|1378452729|16|12518||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|du|3
80923|1378452730|16|12520||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|tar|3
80925|1378452730|14|12520||2081|19244656|32813|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||1026|1026|4|tar|6
80927|1378452730|10|12520||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|tar|3
80930|1378452730|10|12520||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|tar|3
83111|1378452734|16|12547||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|du|3
115893|1378452746|16|14083||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|stat|3
115895|1378452746|14|12554||2081|19244656|32813|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||1026|1026|4|debugfs|6
115897|1378452746|10|12554||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|debugfs|3
120567|1378452747|16|14279||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|find|3
122446|1378452748|16|14337||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|find|3
124174|1378452748|16|14350||2081|19244656|33261|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|du|3

OE @ ~/oe-core/tmp-eglibc $ sqlite3 core-image-minimal-pseudo-5/logs.db
sqlite> select * from logs where path like '%slabtop%';
16097|1378424324|6|10619||2081|18927165|33188|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|opkg-cl|6
16098|1378424324|14|10619||2081|18927165|33188|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||1026|1026|2|opkg-cl|6
16099|1378424324|3|10619||2081|18927165|33188|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|opkg-cl|6
16100|1378424324|2|10619||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|opkg-cl|6
54015|1378424351|16|19727||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|3
72808|1378424361|16|21005||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|du|3
80613|1378424361|16|21013||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|tar|3
80615|1378424361|14|21013||2081|18927165|32813|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||1026|1026|4|tar|6
80616|1378424361|10|21013||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|tar|3
80618|1378424361|10|21013||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|tar|3
82802|1378424366|16|21224||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|du|3
115366|1378424376|16|23609||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|stat|3
115368|1378424376|14|21303||2081|18927165|32813|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||1026|1026|4|debugfs|6
115369|1378424376|10|21303||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|debugfs|3
119930|1378424377|16|23883||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|find|3
121838|1378424378|16|23962||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|find|3
123566|1378424378|16|23978||2081|18927165|33133|/OE/oe-core/tmp-eglibc/work/qemux86_64-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/bin/slabtop|1|2|||0|0|0|du|3
Comment 4 Martin Jansa 2013-09-06 09:12:43 UTC
OK, lets go back from rootfs WORKDIR to procps pseudo, because it's wrong in .ipk already:

bitbake@jama:/OE$ dpkg --contents procps-4.ipk | grep slab
-rwxr-xr-x root/root     12360 2013-09-06 09:30 ./usr/bin/slabtop
bitbake@jama:/OE$ dpkg --contents procps-5.ipk | grep slab
-r-xr-xr-x root/root     12360 2013-09-06 01:34 ./usr/bin/slabtop

I don't know how to properly read this log yet, but the issue seems to start on id 7801/8031 when scratch4 logs python2.7 before strip is completed.

from-scratch4:
7799|1378452647|15|4650||2081|19244451|33225|WORK/package/usr/bin/slabtop|1|2|||1026|1026|0|x86_64-oe-linux-strip|6
7800|1378452647|2|4650||2081|19244451|33261|WORK/package/usr/bin/slabtop|1|2|||0|0|0|x86_64-oe-linux-strip|6
7801|1378452647|2|4641||2081|19244451|33133|WORK/package/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|6
7802|1378452647|3|4650||2081|19244451|33133|WORK/package/usr/bin/slabtop|1|2|||0|0|0|x86_64-oe-linux-strip|6
7804|1378452647|2|4650||2081|19244451|33261|WORK/package/usr/bin/slabtop|1|2|||0|0|0|x86_64-oe-linux-strip|6
9182|1378452647|16|4300||2081|19244451|33261|WORK/package/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|3
9183|1378452647|16|4300||2081|19244451|33261|WORK/package/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|3
9184|1378452647|16|4300||2081|19244451|33261|WORK/package/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|3
9185|1378452647|11|4300||2081|19244451|33261|WORK/packages-split/procps/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|6
9186|1378452647|2|4300||2081|19244451|33261|WORK/packages-split/procps/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|6
9187|1378452647|3|4300||2081|19244451|33261|WORK/packages-split/procps/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|6
9310|1378452647|16|4300||2081|19244451|33261|WORK/package/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|3

from-scratch5:
8021|1378424090|15|31890||2081|18822904|33225|WORK/package/usr/bin/slabtop|1|2|||1026|1026|0|x86_64-oe-linux-strip|6
8022|1378424090|2|31890||2081|18822904|33261|WORK/package/usr/bin/slabtop|1|2|||0|0|0|x86_64-oe-linux-strip|6
8024|1378424090|3|31890||2081|18822904|33261|WORK/package/usr/bin/slabtop|1|2|||0|0|0|x86_64-oe-linux-strip|6
8026|1378424090|2|31890||2081|18822904|33261|WORK/package/usr/bin/slabtop|1|2|||0|0|0|x86_64-oe-linux-strip|6
8031|1378424090|2|31857||2081|18822904|33133|WORK/package/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|6
9260|1378424090|16|30521||2081|18822904|33133|WORK/package/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|3
9261|1378424090|16|30521||2081|18822904|33133|WORK/package/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|3
9262|1378424090|16|30521||2081|18822904|33133|WORK/package/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|3
9263|1378424090|11|30521||2081|18822904|33133|WORK/packages-split/procps/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|6
9264|1378424090|2|30521||2081|18822904|33133|WORK/packages-split/procps/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|6
9265|1378424090|3|30521||2081|18822904|33133|WORK/packages-split/procps/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|6
9388|1378424090|16|30521||2081|18822904|33133|WORK/package/usr/bin/slabtop|1|2|||0|0|0|/usr/bin/python2.7|3
Comment 5 Martin Jansa 2013-09-06 09:27:23 UTC
It's easier to see in pseudolog output:

--- from-scratch4/pseudo/slabtop.log4   2013-09-06 11:23:58.443923636 +0200
+++ from-scratch5/pseudo/slabtop.log4   2013-09-06 11:24:01.023923698 +0200
@@ -46,89 +46,89 @@
 x86_64-oe-li fastop did-unlink: [mode 0755,  -] WORK/package/usr/bin/slabtop
 x86_64-oe-li fastop  rename: [mode 0711,  -] WORK/package/usr/bin/slabtop
 x86_64-oe-li fastop   chmod: [mode 0755,  -] WORK/package/usr/bin/slabtop
-/usr/bin/pyt fastop   chmod: [mode 0555,  -] WORK/package/usr/bin/slabtop 
-x86_64-oe-li fastop   chown: [mode 0555,  -] WORK/package/usr/bin/slabtop 
+x86_64-oe-li fastop   chown: [mode 0755,  -] WORK/package/usr/bin/slabtop 
 x86_64-oe-li fastop   chmod: [mode 0755,  -] WORK/package/usr/bin/slabtop
-/usr/bin/pyt op      stat: [mode 0755,  -] WORK/package/usr/bin/slabtop 
-/usr/bin/pyt op      stat: [mode 0755,  -] WORK/package/usr/bin/slabtop 
-/usr/bin/pyt op      stat: [mode 0755,  -] WORK/package/usr/bin/slabtop 
-/usr/bin/pyt fastop    link: [mode 0755,  -] WORK/packages-split/procps/usr/bin/slabtop 
-/usr/bin/pyt fastop   chmod: [mode 0755,  -] WORK/packages-split/procps/usr/bin/slabtop 
-/usr/bin/pyt fastop   chown: [mode 0755,  -] WORK/packages-split/procps/usr/bin/slabtop 
-/usr/bin/pyt op      stat: [mode 0755,  -] WORK/package/usr/bin/slabtop 
-/usr/bin/pyt op      stat: [mode 0755,  -] WORK/packages-split/procps/usr/bin/slabtop 
+/usr/bin/pyt fastop   chmod: [mode 0555,  -] WORK/package/usr/bin/slabtop 
+/usr/bin/pyt op      stat: [mode 0555,  -] WORK/package/usr/bin/slabtop 
+/usr/bin/pyt op      stat: [mode 0555,  -] WORK/package/usr/bin/slabtop 
+/usr/bin/pyt op      stat: [mode 0555,  -] WORK/package/usr/bin/slabtop 
+/usr/bin/pyt fastop    link: [mode 0555,  -] WORK/packages-split/procps/usr/bin/slabtop 
+/usr/bin/pyt fastop   chmod: [mode 0555,  -] WORK/packages-split/procps/usr/bin/slabtop 
+/usr/bin/pyt fastop   chown: [mode 0555,  -] WORK/packages-split/procps/usr/bin/slabtop 
+/usr/bin/pyt op      stat: [mode 0555,  -] WORK/package/usr/bin/slabtop 
+/usr/bin/pyt op      stat: [mode 0555,  -] WORK/packages-split/procps/usr/bin/slabtop 
 /usr/bin/pyt fastop    open: [mode 0755,  r] WORK/packages-split/procps/usr/bin/slabtop
...
only difference on following lines is 0555/0755 mode.

So the problem is chmod from python happening a bit sooner in some builds.
Comment 6 Martin Jansa 2013-09-06 16:18:56 UTC
There is also some reuse of the same inode in logs:
sqlite> select distinct path from logs where ino=5530781;
/OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/stctlQEd
/OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop
/OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/stNWC016
/OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/free.procps
/OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/packages-split/procps/usr/bin/free.procps

/OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/sstate-build-package/package/usr/bin/free.procps
/OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/sstate-build-package/packages-split/procps/usr/bin/free.procps

I was suspecting race-condition openembedded-core/meta/lib/oe/package.py:runstrip() 

which has that chmod adding +W+R and is executed in parallel from package.bbclass:
        import multiprocessing
        nproc = multiprocessing.cpu_count()
        pool = bb.utils.multiprocessingpool(nproc)
        processed = list(pool.imap(oe.package.runstrip, sfiles))
        pool.close()
        pool.join()

But with added warnings in runstrip code it looks to be executed in the same order (if we can trust order in log file).

from-scratch7/log.txt:WARNING: runstrip: /OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop
from-scratch7/log.txt:WARNING: runstrip: /OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/free.procps
from-scratch7/log.txt:WARNING: runstrip chmod: /OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/free.procps - 33261
from-scratch7/log.txt:WARNING: runstrip chmod: /OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop - 33261

from-scratch8/log.txt:WARNING: runstrip: /OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop
from-scratch8/log.txt:WARNING: runstrip: /OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/free.procps
from-scratch8/log.txt:WARNING: runstrip chmod: /OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/slabtop - 33261
from-scratch8/log.txt:WARNING: runstrip chmod: /OE/oe-core/tmp-eglibc/work/x86_64-oe-linux/procps/3.2.8-r11/package/usr/bin/free.procps - 33261
Comment 7 Richard Purdie 2013-09-25 14:17:44 UTC
Peter, I hate to say this but I think this is a bug in pseudo, specifically in http://git.yoctoproject.org/cgit.cgi/pseudo/commit/?id=01bc8866a4aa5af17350b36e2347dc1695dbceb0. 

In one of the builds where I could show this breaking, this is the output from pseudolog (btw, please put the default format string in the help text!):

$ PSEUDO_BINDIR=/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin PSEUDO_LIBDIR=/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/lib/pseudo/lib PSEUDO_LOCALSTATEDIR=/media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/pseudo/ PSEUDO_NOSYMLINKEXP=1 PSEUDO_PASSWD=/media/build1/poky/build/tmp/sysroots/qemux86 PSEUDO_PREFIX=/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr ../../../../../sysroots/x86_64-linux/usr/bin/pseudolog -F "%s %R %-4y %7o: [mode %04m, %2a] %p %T" | grep package/usr/bin/slabtop
13:56:08 rm op   may-unlink: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:08 rm fastop did-unlink: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:08 tar fastop   creat: [mode 0500,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:08 tar fastop    open: [mode 0500,  w] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 python op      stat: [mode 0555,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 /media/build1/poky/build/tmp/work/x86_64-linux/file-native/5.14-r0/image//media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/file op      stat: [mode 0555,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 /media/build1/poky/build/tmp/work/x86_64-linux/file-native/5.14-r0/image//media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/file op      stat: [mode 0555,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 python op      stat: [mode 0555,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 python fastop   chmod: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/lib/rpm/bin/debugedit op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/lib/rpm/bin/debugedit fastop   chmod: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/lib/rpm/bin/debugedit fastop    open: [mode 0055, r+] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/lib/rpm/bin/debugedit op     fstat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/lib/rpm/bin/debugedit op     fstat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/lib/rpm/bin/debugedit fastop   chmod: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy fastop    open: [mode 0755,  r] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy fastop    open: [mode 0755,  r] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy op   may-unlink: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy fastop did-unlink: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy fastop  rename: [mode 0711,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy fastop   chmod: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy fastop   chown: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 i586-poky-linux-objcopy fastop   chmod: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:09 python fastop   chmod: [mode 0555,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 python op      stat: [mode 0555,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 python fastop   chmod: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip fastop    open: [mode 0755,  r] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip op   may-unlink: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip fastop did-unlink: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip fastop  rename: [mode 0711,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip fastop   chmod: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 python fastop   chmod: [mode 0555,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip fastop   chown: [mode 0555,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip fastop   chmod: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 python op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 python op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 python op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 python op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:12 find op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 

what scares me in this log specifically is:

13:56:10 i586-poky-linux-strip fastop  rename: [mode 0711,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip fastop   chmod: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 python fastop   chmod: [mode 0555,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip fastop   chown: [mode 0555,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 i586-poky-linux-strip fastop   chmod: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 
13:56:10 python op      stat: [mode 0755,  -] /media/build1/poky/build/tmp/work/i586-poky-linux/procps/3.2.8-r12/package/usr/bin/slabtop 

i.e. python ran its chmod 0555 *before* the strip process finished?! Looking at the code in lib/oe/package.py, that should not happen.

Think about it though, strip does chmod 755, sends the request to the pseudo server but them immediately exits. The system exits back to the parent python process which does a chmod 0555 and again, carries straight on.

How do we guarantee the 755 reaches the pseudo server before the 555?

So I think there is a good reason you have to wait for a reply from the server. Its not to say ok/not ok, its to say "hey this change was now queued".
Comment 8 Richard Purdie 2013-09-25 14:45:57 UTC
We may have a way to keep some of the speedup. If at process exit time, pseudo were to ping the server and wait for a reply, this particular problem won't happen.

You could devise some pathological case where process A does something to the file, sends a message to process B and it tries to do something but those cases should be very rare.
Comment 9 Seebs 2013-09-25 17:22:20 UTC
Created attachment 1532 [details]
proposed patch for the atexit() approach

Here's a proposed patch to send a ping on shutdown (using atexit). I have verified that it compiled and ran, I haven't verified that it, say, actually works. Because I can't easily reproduce the problem.

It might be fairly possible to simulate this, except we'd have to update the pseudo server to have smarter/fancier processing of incoming messages to allow it to delay a specific message for a bit.
Comment 10 Martin Jansa 2013-09-25 20:23:48 UTC
Looks good with that patch, at least procps reproduced doesn't show it anymore, I'll try to build it once more and then let whole image to rebuild from scratch over night.
Comment 11 Richard Purdie 2013-09-25 20:36:28 UTC
I added the patch, I couldn't reproduce the problem so it would appear to solve the race issue we were seeing...
Comment 12 Martin Jansa 2013-09-26 08:15:24 UTC
Complete core-image-minimal also doesn't show any differences in permissions now, so really looks good.

BTW: Unrelated issue from last build from scratch:
diff from-scratch20/buildhistory/images/qemux86_64/eglibc/core-image-minimal/installed-packages.txt from-scratch19/buildhistory/images/qemux86_64/eglibc/core-image-minimal/installed-packages.txt
8,13c8,13
< kernel-3.10.11-yocto-standard_3.10.11+git0+4f689aa30a_e1aa804148-r0.4_qemux86_64.ipk
< kernel-module-binfmt-misc_3.10.11+git0+4f689aa30a_e1aa804148-r0.4_qemux86_64.ipk
< kernel-module-cfbcopyarea_3.10.11+git0+4f689aa30a_e1aa804148-r0.4_qemux86_64.ipk
< kernel-module-cfbfillrect_3.10.11+git0+4f689aa30a_e1aa804148-r0.4_qemux86_64.ipk
< kernel-module-cfbimgblt_3.10.11+git0+4f689aa30a_e1aa804148-r0.4_qemux86_64.ipk
< kernel-module-uvesafb_3.10.11+git0+4f689aa30a_e1aa804148-r0.4_qemux86_64.ipk
---
> kernel-3.10.11-yocto-standard_3.10.11+git0+4f689aa30a_e1aa804148-r0.1_qemux86_64.ipk
> kernel-module-binfmt-misc_3.10.11+git0+4f689aa30a_e1aa804148-r0.1_qemux86_64.ipk
> kernel-module-cfbcopyarea_3.10.11+git0+4f689aa30a_e1aa804148-r0.1_qemux86_64.ipk
> kernel-module-cfbfillrect_3.10.11+git0+4f689aa30a_e1aa804148-r0.1_qemux86_64.ipk
> kernel-module-cfbimgblt_3.10.11+git0+4f689aa30a_e1aa804148-r0.1_qemux86_64.ipk
> kernel-module-uvesafb_3.10.11+git0+4f689aa30a_e1aa804148-r0.1_qemux86_64.ipk

Both builds were using local PR service and build log doesn't show any errors from PR service, only that it was started:
from-scratch19/log.txt:NOTE: Started PRServer with DBfile: /OE/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 47078, PID: 17065
from-scratch20/log.txt:NOTE: Started PRServer with DBfile: /OE/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 52381, PID: 24743

sqlite> select * from PRMAIN_nohist where version like 'linux-yocto-3.10.11%';
linux-yocto-3.10.11+gitAUTOINC+4f689aa30a_e1aa804148-r0|qemux86_64|c2bf1442ea618b900226b1ac7ae84b69|3
linux-yocto-3.10.11+gitAUTOINC+4f689aa30a_e1aa804148-r0|qemux86_64|7035787fa02a96383b10bf0abc60f255|4
linux-yocto-3.10.11+gitAUTOINC+4f689aa30a_e1aa804148-r0|qemux86_64|50c0c802cd8cfbba14e970a31183d5c4|5

-rw-rw-r-- 1 bitbake bitbake 141282 Sep 25 23:54 tmp-eglibc/from-scratch20/stamps/qemux86_64-oe-linux/linux-yocto/3.10.11+gitAUTOINC+4f689aa30a_e1aa804148-r0.do_package.sigdata.7035787fa02a96383b10bf0abc60f255

in build "from-scratch19" it was from sstate, so there is only do_package_setscene stamp
Comment 13 Richard Purdie 2013-09-30 22:01:03 UTC
Fixed merged into OE-Core master. http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a1d9b6df86b0829d591cae57ae351f6d96e63557 Its assumed seebs will catch up in the pseudo repository when he has time.