| Summary: | pseudo: Incorrect UID/GID in packaged files | ||
|---|---|---|---|
| Product: | [Yocto Project Subprojects] Pseudo | Reporter: | Martin Jansa <Martin.Jansa> |
| Component: | pseudo | Assignee: | Randy MacLeod <randy.macleod> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | alex.kanavin, juro.bystricky, otavio, pavel.modilaynen, randy.macleod, richard.purdie, ross.burton, stephano, thomas.axelsson, yp.pseudo.watcher, yp.watcher |
| Version: | unspecified | ||
| Target Milestone: | 2.8 M3 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| See Also: | https://bugzilla.yoctoproject.org/show_bug.cgi?id=11299 | ||
| Whiteboard: | Backport 2.5.1, 2.6.1 | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
| Attachments: | |||
|
Description
Martin Jansa
2017-12-19 15:35:47 UTC
rm -rf tmp-glibc cp sstate:glibc-locale:cortexa7hf-neon-vfpv4-webos-linux-gnueabi:2.24:r0:cortexa7hf-neon-vfpv4:3:c7ed399f9e4d6c519db264c1bf4a9349_package.tgz.bad sstate-cache/c7/sstate:glibc-locale:cortexa7hf-neon-vfpv4-webos-linux-gnueabi:2.24:r0:cortexa7hf-neon-vfpv4:3:c7ed399f9e4d6c519db264c1bf4a9349_package.tgz bitbake glibc-locale -f -c package_qa it should fail here rm -rf tmp-glibc cp sstate:glibc-locale:cortexa7hf-neon-vfpv4-webos-linux-gnueabi:2.24:r0:cortexa7hf-neon-vfpv4:3:c7ed399f9e4d6c519db264c1bf4a9349_package.tgz.ok sstate-cache/c7/sstate:glibc-locale:cortexa7hf-neon-vfpv4-webos-linux-gnueabi:2.24:r0:cortexa7hf-neon-vfpv4:3:c7ed399f9e4d6c519db264c1bf4a9349_package.tgz bitbake glibc-locale -f -c package_qa it should be successful again So the only difference between successful and failed build is the do_package sstate archive. I've rebuilt glibc-locale from the same metadata on some other build host to have another example of successful build, then I've unpacked all 3 archives and I'm comparing "ls -lain --time-style=+%Y" "ls -lan --time-style=+%Y" The file itself is identical: $ md5sum package*/packages-split//glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 113de4719b934d388efa2252c6a11e78 package.bad/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 113de4719b934d388efa2252c6a11e78 package.ok2/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 113de4719b934d388efa2252c6a11e78 package.ok/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT $ du -s package*/packages-split//glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 4 package.bad/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 4 package.ok2/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 4 package.ok/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT $ file package*/packages-split//glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT package.bad/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT: data package.ok2/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT: data package.ok/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT: data $ ls -lanR --time-style=+%Y package*/packages-split//glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT -rw-r--r-- 2 1101 1101 28 2017 package.bad/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT -rw-r--r-- 34 1101 1101 28 2017 package.ok2/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT -rw-r--r-- 34 1101 1101 28 2017 package.ok/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT Just the number of links to this file is different between ok and bad builds. $ ls -lainR --time-style=+%Y package*/packages-split//glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 16661265 -rw-r--r-- 2 1101 1101 28 2017 package.bad/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 package.ok2/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 package.ok/packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT $ grep 16661265 package.bad/ls.lain.txt 16661265 -rw-r--r-- 2 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 16661265 -rw-r--r-- 2 1101 1101 28 2017 ./package/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT $ grep 16647965 package.ok2/ls.lain.txt 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-kw.iso-8859-6/usr/lib/locale/ar_KW.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-sy.iso-8859-6/usr/lib/locale/ar_SY.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-tn.iso-8859-6/usr/lib/locale/ar_TN.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-om.iso-8859-6/usr/lib/locale/ar_OM.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-iq.iso-8859-6/usr/lib/locale/ar_IQ.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-sa.iso-8859-6/usr/lib/locale/ar_SA.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ma.iso-8859-6/usr/lib/locale/ar_MA.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-bh.iso-8859-6/usr/lib/locale/ar_BH.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-dz.iso-8859-6/usr/lib/locale/ar_DZ.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-sd.iso-8859-6/usr/lib/locale/ar_SD.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ly.iso-8859-6/usr/lib/locale/ar_LY.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-jo.iso-8859-6/usr/lib/locale/ar_JO.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ye.iso-8859-6/usr/lib/locale/ar_YE.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-lb.iso-8859-6/usr/lib/locale/ar_LB.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-qa.iso-8859-6/usr/lib/locale/ar_QA.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-eg.iso-8859-6/usr/lib/locale/ar_EG.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_TN.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_BH.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_YE.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_QA.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_OM.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_JO.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_DZ.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_EG.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_SD.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_SY.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_IQ.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_KW.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_MA.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_SA.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_LB.ISO-8859-6/LC_MEASUREMENT 16647965 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_LY.ISO-8859-6/LC_MEASUREMENT $ grep 16545878 package.ok/ls.lain.txt 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-kw.iso-8859-6/usr/lib/locale/ar_KW.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-sy.iso-8859-6/usr/lib/locale/ar_SY.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-tn.iso-8859-6/usr/lib/locale/ar_TN.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-om.iso-8859-6/usr/lib/locale/ar_OM.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-iq.iso-8859-6/usr/lib/locale/ar_IQ.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-sa.iso-8859-6/usr/lib/locale/ar_SA.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ma.iso-8859-6/usr/lib/locale/ar_MA.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-bh.iso-8859-6/usr/lib/locale/ar_BH.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-dz.iso-8859-6/usr/lib/locale/ar_DZ.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-sd.iso-8859-6/usr/lib/locale/ar_SD.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ly.iso-8859-6/usr/lib/locale/ar_LY.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-jo.iso-8859-6/usr/lib/locale/ar_JO.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ye.iso-8859-6/usr/lib/locale/ar_YE.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-lb.iso-8859-6/usr/lib/locale/ar_LB.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-qa.iso-8859-6/usr/lib/locale/ar_QA.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-eg.iso-8859-6/usr/lib/locale/ar_EG.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_TN.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_BH.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_YE.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_QA.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_OM.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_JO.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_DZ.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_EG.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_SD.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_SY.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_IQ.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_KW.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_MA.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_SA.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_LB.ISO-8859-6/LC_MEASUREMENT 16545878 -rw-r--r-- 34 1101 1101 28 2017 ./package/usr/lib/locale/ar_LY.ISO-8859-6/LC_MEASUREMENT So in ok build all these files are pointing to the same inode, but in bad build, the file which causes the QA error is separated from them: $ grep ar_.*.ISO-8859-6/LC_MEASUREMENT package.bad/ls.lain.txt 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-kw.iso-8859-6/usr/lib/locale/ar_KW.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-sy.iso-8859-6/usr/lib/locale/ar_SY.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-tn.iso-8859-6/usr/lib/locale/ar_TN.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-om.iso-8859-6/usr/lib/locale/ar_OM.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-iq.iso-8859-6/usr/lib/locale/ar_IQ.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-sa.iso-8859-6/usr/lib/locale/ar_SA.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ma.iso-8859-6/usr/lib/locale/ar_MA.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-bh.iso-8859-6/usr/lib/locale/ar_BH.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-dz.iso-8859-6/usr/lib/locale/ar_DZ.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-sd.iso-8859-6/usr/lib/locale/ar_SD.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ly.iso-8859-6/usr/lib/locale/ar_LY.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-jo.iso-8859-6/usr/lib/locale/ar_JO.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ye.iso-8859-6/usr/lib/locale/ar_YE.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-lb.iso-8859-6/usr/lib/locale/ar_LB.ISO-8859-6/LC_MEASUREMENT 16661265 -rw-r--r-- 2 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-ae.iso-8859-6/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-qa.iso-8859-6/usr/lib/locale/ar_QA.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./packages-split/glibc-binary-localedata-ar-eg.iso-8859-6/usr/lib/locale/ar_EG.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_TN.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_BH.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_YE.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_QA.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_OM.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_JO.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_DZ.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_EG.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_SD.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_SY.ISO-8859-6/LC_MEASUREMENT 16661265 -rw-r--r-- 2 1101 1101 28 2017 ./package/usr/lib/locale/ar_AE.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_IQ.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_KW.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_MA.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_SA.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_LB.ISO-8859-6/LC_MEASUREMENT 16659063 -rw-r--r-- 32 1101 1101 28 2017 ./package/usr/lib/locale/ar_LY.ISO-8859-6/LC_MEASUREMENT I know this is still far from root cause and small reproducer, I just wanted to document (somewhere) my current findings. It doesn't explain why the bad do_package archive was created in first place or why the inode for this file is different, but at least I know now what archive to remove from sstate-cache when this issue hits us again. I'll again trigger rebuilding glibc-locale in for loop, with detection of this kind of difference in package directory and hopefully I'll be able to catch it together with complete WORKDIR and pseudo.log. Created attachment 4157 [details]
My comments without bugzilla formatting (much easier to read without wrapped lines)
I've tried with newer pseudo, with: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=akanavin/pseudo-1.9.0 SRCREV: b6a015a Handle O_TMPFILE more better and epoll disabled I was still able to reproduce this issue, I'm just re-running do_package in glibc-locale and checking for host-user-contamination QA issue: echo 'ERROR_QA_append = " host-user-contaminated"' >> conf/local.conf for i in `seq -w 1 999`; do echo $i; bitbake -v -v -DDDD -f -c package glibc-locale 2>glibc-locale-logerr.$i >glibc-locale-log.$i; bitbake -v -v -DDDD -f -c package_qa glibc-locale 2>glibc-locale-logerr.qa.$i >glibc-locale-log.qa.$i || { cp -ra BUILD/work/i586-oe-linux/glibc-locale/ glibc-locale-workdir-$i; bitbake -c cleansstate glibc-locale ; } done And I got 13 failures in first 205 iterations (then I've stopped the loop). Now I'm testing with even newer SRCREV (f3f4459 Epoll: use the correct client) and epoll enabled again, with only 12 iterations finished sofar it's to early to say, will add another comment when I see some failures. Yes, it's still reproducible with latest SRCREV. It took just 12 iterations to get the first failed one: glibc-locale-logerr.qa.012:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-an-es.iso-8859-15/usr/lib/locale/an_ES.ISO-8859-15/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.012:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-an-es.iso-8859-15/usr/lib/locale/an_ES.ISO-8859-15/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] In our CI we just see this failure in rocko and master. Earlier releases we never saw it. I'm seeing it since morty with pseudo-1.8.1 (I've actually backported 1.8.2 to morty because it seemed to trigger this issue a bit less often, but it's hard to compare as it doesn't fail very often and in different components). In our internal builds I've already added INSANE_SKIP in 3 components where it failed most often (glibc-locale and 2 internal recipes, where in one I was able to reproduce it myself and in 2nd only some other colleagues were reporting the issue, but I never reproduced it even with quite long loop). With newer qt (I think since 5.9) I see this issue quite often in various qt* recipes (qtbase, qtdeclarative most often I think), but those aren't included in our builds so I don't track how often it actually fails. In rocko I see it daily in our CI, also in Qt components. Otavio: do you share sstate-cache between those daily builds? As described above once the "bad" do_package sstate archive is created it will always fail in follow-up builds which reuse it. The world builds also show: openembedded-core/meta/recipes-core/glibc/glibc-locale_2.26.bb:do_package_qa failure almost every single run, but on the other hand sstate for glibc-locale isn't recreated very often. seebs: It doesn't need to do anything really fancy to trigger the issue (AFAIK) the internal component which triggers this does just:
do_install_append() {
tar xvzf ${S}/files/conf/fonts/fonts.tgz --directory=${D}${datadir}/fonts
}
and all the files in ${D}${datadir}/fonts sometimes get wrong UID. It doesn't do anything fancy even in normal do_install that it would get pseudo into some weird state before this tar call, it just installs other few files with CMake (in the same way as all other components).
The pseudo.log also doesn't show anything more strange in these components than in other components where I never reproduced this issue. But again it's hard to properly compare when every recipe shows many potential errors in pseudo.log.
Martin, yes, we share sstate. I am wondering if we could trigger a removal of those sstate objects when this situation is identified. hah, finally I came with much simpler reproducer.
The common behavior is creating the same file many times in parallel (it doesn't need to be many times - our internal component was installing group of 3 files, 3 times to trigger this).
But to make it fail quickly lets recreate foo.txt 1000 times with make and use PARALLEL_MAKE for parallelism (in my case -j 9).
Here is simple recipe:
docker-shr qemux86-64@ ~/build/oe-core $ cat openembedded-core/meta/recipes-devtools/pseudo/pseudo-test.bb
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
inherit allarch
DEPENDS = "coreutils-native"
CLEANBROKEN = "1"
RULES = "1000"
do_compile() {
echo > Makefile; ALL="all: "; for i in `seq -w 1 ${RULES}`; do echo -e "R$i:\n\techo $i > foo.txt" >> Makefile; ALL="$ALL R$i"; done; echo ${ALL} >> Makefile
}
do_install() {
oe_runmake all
cat foo.txt
mv foo.txt ${D}
}
FILES_${PN} = "/"
and here is the immediate failure:
WARNING: pseudo-test-1.0-r0 do_qa_pseudo: QA Issue: This /work/all-oe-linux/pseudo-test/1.0-r0/pseudo/pseudo.log indicates 19 error
s [qa-pseudo]
see grep -e '^creat ignored for existing file' -e '^creat for.*replaces existing' /OE/build/oe-core/tmp-glibc/work/all-oe-linux/pse
udo-test/1.0-r0/pseudo/pseudo.log or grep -v '^path mismatch' /OE/build/oe-core/tmp-glibc/work/all-oe-linux/pseudo-test/1.0-r0/pseu
do/pseudo.log: debug_logfile: fd 2
pid 13654 [parent 13653], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 13653.
db cleanup for server shutdown, 20:32:36.390
memory-to-file backup complete, 20:32:36.391.
db cleanup finished, 20:32:36.391
debug_logfile: fd 2
pid 1576 [parent 1575], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 1575.
inode mismatch: '/OE/build/oe-core/tmp-glibc/work/all-oe-linux/pseudo-test/1.0-r0/temp/log.do_install' ino 19229345 in db, 19233525
in request.
symlink mismatch: '/OE/build/oe-core/tmp-glibc/work/all-oe-linux/pseudo-test/1.0-r0/temp/log.do_install' [19229345] db mode 0120777
, header mode 0100644 (unlinking db)
db cleanup for server shutdown, 20:37:56.713
memory-to-file backup complete, 20:37:56.714.
db cleanup finished, 20:37:56.714
debug_logfile: fd 2
pid 2140 [parent 2139], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 2139.
creat for '/OE/build/oe-core/tmp-glibc/work/all-oe-linux/pseudo-test/1.0-r0/pseudo-test-1.0/foo.txt' replaces existing 19233995 ['/OE/build/oe-core/tmp-glibc/work/all-oe-linux/pseudo-test/1.0-r0/pseudo-test-1.0/foo.txt'].
creat ignored for existing file '/OE/build/oe-core/tmp-glibc/work/all-oe-linux/pseudo-test/1.0-r0/pseudo-test-1.0/foo.txt'.
[qa-pseudo]
ERROR: pseudo-test-1.0-r0 do_package_qa: QA Issue: pseudo-test: /pseudo-test/foo.txt is owned by uid 1026, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated]
ERROR: pseudo-test-1.0-r0 do_package_qa: QA run found fatal errors. Please consider fixing them.
ERROR: pseudo-test-1.0-r0 do_package_qa: Function failed: do_package_qa
ERROR: Logfile of failure stored in: /OE/build/oe-core/tmp-glibc/work/all-oe-linux/pseudo-test/1.0-r0/temp/log.do_package_qa.3278
ERROR: Task (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/pseudo/pseudo-test.bb:do_package_qa) failed with exit code '1'
Seebs: I guess you prefer to reproduce it without whole OE build, then you can generate the Makefile as in do_compile and just run "make -j 10 all" under pseudo.
Created attachment 4216 [details]
workdir of pseudo-test
So I think I may have found this, but I couldn't reproduce it quite as such on my system. The underlying problem, I think, is that OP_CREAT does two things when there may be existing entries: * If there's an existing entry with the same inode, it's deleted because obviously the new file being created means the old one must have gotten deleted. * If there's an existing entry with the same path, we don't create a new link, because there's already an entry. But what if those were the *same* entry? Then it gets deleted and not recreated. You could in principle hit this with open() because the determination of whether or not to send an OP_CREAT is based on whether a stat() (real filesystem stat, not pseudo stat) succeeded before the open was attempted. But that's racy! So if you have two processes: P1: real_stat (fails) P2: real_stat (fails) P1: open with O_CREAT (succeeds) P2: open with O_CREAT (succeeds) you now have two OP_CREAT which will have the same path and inode. We now distinguish between "we found a single exact match" and "we found a matching path and a matching inode but they were not the same entry". Pushed to master. You may need to test it on faster machines with more cores to find out whether this helps. Thanks for update! Unfortunately with pseudo-native-1.9.0+gitAUTOINC+dfc21be281 I was still able to reproduce this issue. Hmm. How reliable is it? If you just run the make in pseudo, outside of bitbake, does it happen 100% of the time? 1% of the time? Does number of jobs matter? I've been trying with -j1 through -j500, and can't get it to happen if I don't start with something broken in the database. I'll continue looking. It was quite reliable when I've added the comment. I was testing it from oe-core build. I was using '-j 9'. First I've tried with just 10 rules in Makefile and it didn't fail, then with 100 and still didn't fail, then with 1000 and it failed 3 times in row, which I've considered reliable reproducer at that time. Today I wasn't so lucky (with and without your latest fix), on the same host with the same -j 9 and the same 1000 rules I wasn't able to reproduce it for a while, then I've triggered builds with the same latest pseudo SRCREV on multiple builders and got first failure quite soon. Now I'm trying to include this in many more builds (100+ builds across 20+ different builders) to see how often it will fail now. It's possible that you've fixed one cause of the corruption and some other is still there. I had hoped this would fix: https://bugzilla.yoctoproject.org/show_bug.cgi?id=11299 Bug 11299 - glibc-locale warning [host-user-contaminated] but it did not. Just to clarify my previous comment: I rebuilt pseudo with the commit SRCREV = "dfc21be28185956f61bd904022312f781cbe1572" (Don't delete things and not create them.) Then, using the configuration: Build Configuration: BB_VERSION = "1.37.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.4+snapshot-20180329" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:ba54521c6782b5ca16581676dd6c603e5ed983b4" I was still able to reproduce the error: WARNING: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ve-za/usr/lib/locale/ve_ZA/LC_COLLATE is owned by uid 1001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] I think the question is whether that's still the same underlying cause, or is a *different* possible cause of host contamination. I do have a vague memory that glibc-locale had some stuff in it that I expected to potentially behave badly, in that it was copying or linking stuff in from another part of the glibc build. So, vmeson looked at this a bit more, and I think this is still alive, so I'm gonna try to get another look at it real soon now. Thanks, I still see this once a day in glibc-locale. On tmpfs, if it makes a difference. I found a *different* bug while trying to reproduce this, but I can't make that pseudo-test package generate the apparent failures. I did find and fix the symlink problem with log.do_install, which is a bug we've apparently had for at *least* three years. I was testing with latest revision fddbe854c9db058d5a05830d3bcdd4233d95ee2e and this issues is still reproducible (got at least 2 builds failing with glibc-locale and one with qml-webos-framework - https://github.com/webosose/meta-webosose/blob/master/meta-webos/recipes-webos/qml-webos-framework/qml-webos-framework.bb) Do we have any indication of what the requirements are for hitting this? My thought is that it's weird that only a couple of packages are hitting it. With the new patches, if nothing else, pseudo.log *should* be more informative. (And the symlink fix might actually reduce some of the problems we were hitting.) I've finished all 1000 iterations of the loop from comment 3 over the weekend with latest pseudo revision fddbe854c9db058d5a05830d3bcdd4233d95ee2e and it failed in 54 cases, so it's still hard to reproduce. on Friday I've shared glibc-locale workdir from the first failure: 157M glibc-locale-workdir-003.tar.bz2 https://drive.google.com/open?id=1_aeVGu7-vz_0aZEGX9wiZdlYDLvUlVjG I wasn't able to reproduce it with the simple pseudo-test and pseudo-test2 recipes in over 100 iterations. pseudo.log starts out: debug_logfile: fd 2 pid 30285 [parent 30275], doing new pid setup and server start Setup complete, sending SIGUSR1 to pid 30275. creat for '/jenkins/mjansa/build-nodistro-master/BUILD/work/i586-oe-linux/glibc-locale/2.27-r0/package/usr/lib/gconv/stKXhfIT' replaces existing 73631829 ['/jenkins/mjansa/build-nodistro-master/BUILD/work/i586-oe-linux/glibc-locale/2.27-r0/package/usr/lib/gconv/IBM4899.so']. Okay, so. this has some interesting implications. One is, when this happens, it is actually happening on the first build since the pseudo directory was created, I believe. So this *should* be reproducible in a fresh build of glibc-locale. The second observation is that the errors aren't the same files every time, or all the files, just *some* files. I'm still trying to find anything smaller I can do that ever triggers it. Given that the trivial test case no longer does it, I'm betting on, say, a specific syscall that's being handled wrong, which is being used by something used in glibc-locale, but not in the smaller test case. I've added few possible reproducers to http://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/pseudo qtquickcontrols triggers this quite often in some of my builds, now I'm re-executing do_package 1000 times (like with glibc-locale before) to see what failure rate we get with qtquickcontrols, sofar no failure in first 27 iterations. qtquickcontrols didn't fail once in 1000 iteration, either something is missing to reproduce it or the failure rate is really low for this one. I've added 2 more possible reproducers (luna-init, qml-webos-framework) from meta-webosose to: http://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/pseudo Now doing the 1000 iterations for both to find out failure rate. For the glibc-locale it's really just some random file (usually just one per glibc-locale failure, even more rarely 2 files, e.g. in build 811 and 944) from on of glibc-binary-localedata-* packages, here are the files from my for loop in January: libc-locale-logerr.qa.019:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-es-cr/usr/lib/locale/es_CR/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.019:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-to-to/usr/lib/locale/to_TO/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.266:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-bi-vu/usr/lib/locale/bi_VU/LC_NAME is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.293:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-es-es/usr/lib/locale/es_ES/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.311:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-hu-hu/usr/lib/locale/hu_HU/LC_NUMERIC is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.343:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-hu-hu/usr/lib/locale/hu_HU/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.362:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-en-bw/usr/lib/locale/en_BW/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.386:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-mg-mg/usr/lib/locale/mg_MG/LC_NAME is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.420:ERROR: glibc-locale-2.26-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-be-by/usr/lib/locale/be_BY/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] This time: glibc-locale-logerr.qa.003:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-nl-be/usr/lib/locale/nl_BE/ LC_CTYPE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.034:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ar-kw.iso-8859-6/usr/lib/lo cale/ar_KW.ISO-8859-6/LC_CTYPE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.035:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-kw-gb/usr/lib/locale/kw_GB/LC_MONETARY is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.036:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-es-pe/usr/lib/locale/es_PE/LC_PAPER is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.058:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-so-so/usr/lib/locale/so_SO/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.064:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ar-lb.iso-8859-6/usr/lib/locale/ar_LB.ISO-8859-6/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.079:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ar-iq/usr/lib/locale/ar_IQ/LC_NUMERIC is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.088:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-be-by+latin/usr/lib/locale/be_BY@latin/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.109:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-doi-in/usr/lib/locale/doi_IN/LC_MONETARY is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.168:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-id-id.iso-8859-1/usr/lib/locale/id_ID.ISO-8859-1/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.235:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-es-hn/usr/lib/locale/es_HN/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.355:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-fi-fi/usr/lib/locale/fi_FI/LC_NAME is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.358:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-bho-in/usr/lib/locale/bho_IN/LC_NUMERIC is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.382:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ti-er/usr/lib/locale/ti_ER/LC_MONETARY is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.411:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-byn-er/usr/lib/locale/byn_ER/LC_TELEPHONE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.433:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-so-so.iso-8859-1/usr/lib/locale/so_SO.ISO-8859-1/LC_MEASUREMENT.tmp is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.435:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-it-ch/usr/lib/locale/it_CH/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.567:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-mag-in/usr/lib/locale/mag_IN/LC_PAPER is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.580:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-nn-no.iso-8859-1/usr/lib/locale/nn_NO.ISO-8859-1/LC_NUMERIC is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.604:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-nan-tw/usr/lib/locale/nan_TW/LC_MONETARY is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.618:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-hsb-de/usr/lib/locale/hsb_DE/LC_MONETARY is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.652:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-zu-za/usr/lib/locale/zu_ZA/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.672:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-br-fr.iso-8859-1/usr/lib/locale/br_FR.ISO-8859-1/LC_NAME is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.680:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-hne-in/usr/lib/locale/hne_IN/LC_NUMERIC is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.703:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-nr-za/usr/lib/locale/nr_ZA/LC_PAPER is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.717:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-en-ie/usr/lib/locale/en_IE/LC_NAME is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.720:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-es-hn.iso-8859-1/usr/lib/locale/es_HN.ISO-8859-1/LC_PAPER.tmp is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.742:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-oc-fr/usr/lib/locale/oc_FR/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.751:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-en-zw/usr/lib/locale/en_ZW/LC_NUMERIC is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.782:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-es-gt/usr/lib/locale/es_GT/LC_NAME is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.797:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-so-et/usr/lib/locale/so_ET/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.803:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-el-cy/usr/lib/locale/el_CY/LC_NUMERIC is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.804:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ca-it.iso-8859-15/usr/lib/locale/ca_IT.ISO-8859-15/LC_NAME is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.811:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-an-es/usr/lib/locale/an_ES/LC_MONETARY is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.811:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-en-il/usr/lib/locale/en_IL/LC_PAPER is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.816:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-it-ch/usr/lib/locale/it_CH/LC_PAPER is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.835:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-de-lu/usr/lib/locale/de_LU/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.838:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-unm-us/usr/lib/locale/unm_US/LC_PAPER is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.856:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ca-fr/usr/lib/locale/ca_FR/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.888:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-kk-kz/usr/lib/locale/kk_KZ/LC_CTYPE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.898:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-eu-es/usr/lib/locale/eu_ES/LC_CTYPE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.902:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ayc-pe/usr/lib/locale/ayc_PE/LC_TELEPHONE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.903:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-kw-gb/usr/lib/locale/kw_GB/LC_PAPER is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.906:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-gu-in/usr/lib/locale/gu_IN/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.910:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ru-ua/usr/lib/locale/ru_UA/LC_MESSAGES/SYS_LC_MESSAGES is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.915:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-de-de+euro.iso-8859-15/usr/lib/locale/de_DE@euro.ISO-8859-15/LC_NAME is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.919:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-es-pe.iso-8859-1/usr/lib/locale/es_PE.ISO-8859-1/LC_CTYPE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.930:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-nl-nl/usr/lib/locale/nl_NL/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.934:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-tn-za/usr/lib/locale/tn_ZA/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.938:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-tcy-in/usr/lib/locale/tcy_IN/LC_MONETARY is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.944:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ar-lb/usr/lib/locale/ar_LB/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.944:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-sr-rs+latin/usr/lib/locale/sr_RS@latin/LC_COLLATE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.946:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-en-ng/usr/lib/locale/en_NG/LC_PAPER is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.950:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-fr-ca.iso-8859-1/usr/lib/locale/fr_CA.ISO-8859-1/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.971:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-el-gr+euro.iso-8859-7/usr/lib/locale/el_GR@euro.ISO-8859-7/LC_CTYPE is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] glibc-locale-logerr.qa.984:ERROR: glibc-locale-2.27-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ar-bh/usr/lib/locale/ar_BH/LC_MEASUREMENT is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] I had no luck reproducing it with qml-webos-framework and luna-init when re-executing just do_package task like with glibc-locale, but now re-executing do_install I got failure with qml-webos-framework just after 45 iterations. I've updated jansa/pseudo branch once more, last commit has the exact command I'm using to reproduce this. Created attachment 4291 [details]
pseudo.log from failing qml-webos-framework builds, the error message appended at the end
Okay, that's really interesting, because that's *totally* different from the glibc-locale thing. So I think we have two different things here. What's particularly interesting is, note that we're seeing the same numbers show up more than once; we have several references to 75722572 vs. 75722816. And we don't see many "creat replaces existing", but we have some "creat ignored for existing", which... well that sure is something. Huh. I'm also not sure why there's so many different runs of the daemon here. > I'm also not sure why there's so many different runs of the daemon here.
Because it's being rebuilt again and again in the same WORKDIR until it fails when it's copied out and the for loop continues.
I'm rather surprised there aren't more of them, this pseudo.log is from 123th iteration and there are only 29 "^db cleanup finished" "^Setup complete" messages in it.
With the qml-webos-framework I got it reproduced 2 times in 140 iterations on one builder and only once in 156 iteration on slower builder. But at least the iteration is much faster than glibc-locale (on slow busy builder I got those 156 iterations in around 4 hours). Have we ever seen it on a *first* build? I recognize it's rare, and thus hard to tell. What's actually doing the unpacking that's triggering this? Because a *single* tar process shouldn't be able to hit this, I don't think, but if we had two things unpacking *at the same time*, I could see it becoming an issue. So, looking at that workdir. We have an error: glibc-locale: /glibc-binary-localedata-nl-be/usr/lib/locale/nl_BE/LC_CTYPE is owned by uid 2001, which is the same as the user running bitbake. We have a parallel thing in pseudo.log: creat for '/jenkins/mjansa/build-nodistro-master/BUILD/work/i586-oe-linux/glibc-locale/2.27-r0/locale-tree/usr/lib/locale/ru_UA/LC_NUMERIC' replaces existing 73330495 ['/jenkins/mjansa/build-nodistro-master/BUILD/work/i586-oe-linux/glibc-locale/2.27-r0/locale-tree/usr/lib/locale/nl_BE/LC_CTYPE']. Meanwhile, in files.db, we find that this inode is in fact in use for ... nl_BE/LC_NUMERIC. But that's a file with like 50+ links. 101777|/jenkins/mjansa/build-nodistro-master/BUILD/work/i586-oe-linux/glibc-locale/2.27-r0/locale-tree/usr/lib/locale/nl_BE/LC_NUMERIC|64514|73330495|0|0|33188|0|0 And LC_CTYPE exists: 101773|/jenkins/mjansa/build-nodistro-master/BUILD/work/i586-oe-linux/glibc-locale/2.27-r0/locale-tree/usr/lib/locale/nl_BE/LC_CTYPE|64514|73330492|2001|2001|33188|0|0 ... but with the wrong UID. So that's an important thing: It's not that we're *losing* the pseudo database entry. We're *creating* one with the wrong UID! What's interesting here is that the inode was being used by LC_CTYPE. So *something* happened to cause that inode to be in the database for LC_CTYPE. My suspicion is that it was used for that file previously, and then got removed outside of pseudo. Currently, if a CREAT comes in for a file, and there's a non-matching entry, but no exact-match entry, we delete everything with the old inode number. And that *should* be correct -- but only if we're sure we're never generating a CREAT for a file which is really just a new link. But I also note that there's a *ton* of messages involving this, and files which have multiple hard links. So I'm trying to figure out what's happening to lead up to this. The fact that only some packages are triggering it makes me think they're doing something that's causing it, but I'm still not sure what's up. After 160 iterations, I've restarted the build to do clean build every single time (so that the pseudo.log doesn't get accumulated messages from multiple builds).
Currently running this:
export COMP=qml-webos-framework; export WORK=tmp-glibc/work/qemux86_64-oe-linux/${COMP}/; for i in `seq -w 160 999`; do echo $i; bitbake -c cleansstate ${COMP} >/dev/null; bitbake -v -v -DDDD -f -c package_qa ${COMP} 2>${COMP}-logerr.qa.$i >${COMP}-log.qa.$i && mv ${WORK} ${COMP}-workdir-ok-$i || mv ${WORK} ${COMP}-workdir-bad-$i; done
It might take a while until I hit first failure, but IIRC I was getting failures even from clean WORKDIR before as well.
Not sure what you mean with unpacking mentioned in last comment, luna-init is calling tar to unpack the font files which end up with wrong UID, but I haven't reproduced that one yet, with qml-webos-framework I don't think there is any unpacked involved (as I'm not reusing sstate for this one, nor calling tar during the build AFAIK).
In 750 iterations sofar I got 10 failures. In one case (build 627 it was from 2 files, otherwise always only one file) qml-webos-framework-logerr.qa.039:ERROR: qml-webos-framework-1.0.0-1-r23 do_package_qa: QA Issue: qml-webos-framework: /qml-webos-framework/usr/share/ls2/roles/prv/com.webos.qml-app.json is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] qml-webos-framework-logerr.qa.082:ERROR: qml-webos-framework-1.0.0-1-r23 do_package_qa: QA Issue: qml-webos-framework: /qml-webos-framework/usr/share/dbus-1/system-services/com.webos.qml-app.service is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] qml-webos-framework-logerr.qa.100:ERROR: qml-webos-framework-1.0.0-1-r23 do_package_qa: QA Issue: qml-webos-framework: /qml-webos-framework/usr/share/dbus-1/services/com.webos.qml-app.service is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] qml-webos-framework-logerr.qa.228:ERROR: qml-webos-framework-1.0.0-1-r23 do_package_qa: QA Issue: qml-webos-framework: /qml-webos-framework/usr/share/dbus-1/services/com.webos.qml-app.service is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] qml-webos-framework-logerr.qa.472:ERROR: qml-webos-framework-1.0.0-1-r23 do_package_qa: QA Issue: qml-webos-framework: /qml-webos-framework/usr/share/ls2/roles/pub/com.webos.qml-app.json is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] qml-webos-framework-logerr.qa.556:ERROR: qml-webos-framework-1.0.0-1-r23 do_package_qa: QA Issue: qml-webos-framework: /qml-webos-framework/usr/share/ls2/roles/pub/com.webos.qml-app.json is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] qml-webos-framework-logerr.qa.590:ERROR: qml-webos-framework-1.0.0-1-r23 do_package_qa: QA Issue: qml-webos-framework: /qml-webos-framework/usr/share/ls2/roles/pub/com.webos.qml-app.json is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] qml-webos-framework-logerr.qa.627:ERROR: qml-webos-framework-1.0.0-1-r23 do_package_qa: QA Issue: qml-webos-framework: /qml-webos-framework/usr/share/dbus-1/system-services/com.webos.qml-app.service is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination qml-webos-framework-logerr.qa.627:qml-webos-framework: /qml-webos-framework/usr/share/ls2/roles/prv/com.webos.qml-app.json is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] qml-webos-framework-logerr.qa.641:ERROR: qml-webos-framework-1.0.0-1-r23 do_package_qa: QA Issue: qml-webos-framework: /qml-webos-framework/usr/share/ls2/roles/pub/com.webos.qml-app.json is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] qml-webos-framework-logerr.qa.655:ERROR: qml-webos-framework-1.0.0-1-r23 do_package_qa: QA Issue: qml-webos-framework: /qml-webos-framework/usr/share/ls2/roles/pub/com.webos.qml-app.json is owned by uid 2001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] The pseudo.log files are in most failed cases relatively short: 17 qml-webos-framework-workdir-bad-039/1.0.0-1-r23/pseudo/pseudo.log 17 qml-webos-framework-workdir-bad-082/1.0.0-1-r23/pseudo/pseudo.log 17 qml-webos-framework-workdir-bad-100/1.0.0-1-r23/pseudo/pseudo.log 17 qml-webos-framework-workdir-bad-228/1.0.0-1-r23/pseudo/pseudo.log 24 qml-webos-framework-workdir-bad-472/1.0.0-1-r23/pseudo/pseudo.log 17 qml-webos-framework-workdir-bad-556/1.0.0-1-r23/pseudo/pseudo.log 17 qml-webos-framework-workdir-bad-590/1.0.0-1-r23/pseudo/pseudo.log 34 qml-webos-framework-workdir-bad-627/1.0.0-1-r23/pseudo/pseudo.log 17 qml-webos-framework-workdir-bad-641/1.0.0-1-r23/pseudo/pseudo.log 17 qml-webos-framework-workdir-bad-655/1.0.0-1-r23/pseudo/pseudo.log pseudo.log from builds 228, 472, 627 attached. pseudo.log from successful builds often has only 6 lines like this: debug_logfile: fd 2 pid 47189 [parent 47188], doing new pid setup and server start Setup complete, sending SIGUSR1 to pid 47188. db cleanup for server shutdown, 10:24:40.892 memory-to-file backup complete, 10:24:40.893. db cleanup finished, 10:24:40.893 But not always, in 263 builds (from 750) it has more than 6 lines. E.g. 54 successful builds have 7 lines long pseudo.log from these 54 the 7th line is '^creat for ' in 4 cases '^creat ignored for existing file ' in 47 cases '^inode mismatch' in 3 cases but "typical" create ignored looks like this and usually is about the files which often end with wrong UID: creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/ls2/roles/pub/com.webos.qml-app.json'. Every failed build has "^creat ignored for existing file " for the files which ends up with wrong UID: grep "^creat ignored for existing file " qml-webos-framework-workdir-bad-*/1.0.0-1-r23/pseudo/pseudo.log | wc -l 11 qml-webos-framework-workdir-bad-039/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/ls2/roles/prv/com.webos.qml-app.json'. qml-webos-framework-workdir-bad-082/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/dbus-1/system-services/com.webos.qml-app.service'. qml-webos-framework-workdir-bad-100/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/dbus-1/services/com.webos.qml-app.service'. qml-webos-framework-workdir-bad-228/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/dbus-1/services/com.webos.qml-app.service'. qml-webos-framework-workdir-bad-472/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/ls2/roles/pub/com.webos.qml-app.json'. qml-webos-framework-workdir-bad-556/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/ls2/roles/pub/com.webos.qml-app.json'. qml-webos-framework-workdir-bad-590/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/ls2/roles/pub/com.webos.qml-app.json'. qml-webos-framework-workdir-bad-627/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/dbus-1/system-services/com.webos.qml-app.service'. qml-webos-framework-workdir-bad-627/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/ls2/roles/prv/com.webos.qml-app.json'. qml-webos-framework-workdir-bad-641/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/ls2/roles/pub/com.webos.qml-app.json'. qml-webos-framework-workdir-bad-655/1.0.0-1-r23/pseudo/pseudo.log:creat ignored for existing file '/jenkins/mjansa/build-nodistro-master/BUILD/work/qemux86-oe-linux/qml-webos-framework/1.0.0-1-r23/image/usr/share/ls2/roles/pub/com.webos.qml-app.json'. But this message in log isn't the only reason for failure, because 83 (from 740 successful builds) have the same message for the same files, but doesn't trigger the issue with UID grep "^creat ignored for existing file " qml-webos-framework-workdir-ok-*/1.0.0-1-r23/pseudo/pseudo.log | wc -l 83 Let me know if you're able to reproduce with qml-webos-framework locally or if you need any other information from me. I've seen this once when building mariadb but of course it wasn't reproducible. I haven't really taken enough time to try to reproduce this yet. It's been around for several releases so I doubt it will gate the 2.6 release. Move it to M4 for now. I've seen this type of failure twice in the last week in the Wind River automated builds. Both times it was glibc-locale that encountered the QA error. The first error was on September 30, 2018 but I don't have a specific commit yet. We'll be investigating in the coming weeks but it's quite a rare error so it won't get much attention next week. Is therea log file that I could get the automated builds to save? I could also add a local commit, even an intrusive one after we branch thud. Potentially resolved by: http://git.openembedded.org/openembedded-core/commit/?id=92fdb64ac9689b9cac8a1229b1928b50338969be but it's a rare failure so I plan to monitor Wind River internal builds for a month or so to see if the error happens. It isn't resolved by that change, as reported recently: http://lists.openembedded.org/pipermail/openembedded-core/2019-February/279606.html http://lists.openembedded.org/pipermail/openembedded-core/2019-February/279576.html http://git.yoctoproject.org/cgit.cgi/pseudo/commit/?id=ed20f8323a9676e4e2c14bda06aa07fb1a1ba8ae might be the problem. There is probably a second piece to this we still need to figure out but this would help fix it. I'm using latest master (3fa7c853e0bcd6fe23f7524c2a3c9e3af90901c3). I've had problems with the base-files IPK, generated by Yocto, containing one or multiple files owned by 1000:1000. For my case, the "partial fix" did not help. However I noticed that if I disable memory DB (#undef USE_MEMORY_DB) and clean (bitbake -c clean) base-files, the permissions in the output IPK always look good (37 out of 37 times so far). If I enable memory DB, then bitbake clean base-files and run again, I get 1 good IPK (owner 0) and then only bad IPKs (owner 1000) (at least 8 bad IPKs at the moment). My test is as follows, in a loop: bitbake base-files -C package -c package_write_ipk FILES=$(ar p $IPK data.tar.xz | tar tvJ) if echo "$FILES" | grep -m1 1000; then echo BAD; else echo GOOD; fi I hit the problem originally on a higher level (build image) - running "base-files -C package -c package_write_ipk" is just to speed up the testing. However, I have only tested the "partial fix" with this test. (In reply to comment #43) > I'm using latest master (3fa7c853e0bcd6fe23f7524c2a3c9e3af90901c3). I've had > problems with the base-files IPK, generated by Yocto, containing one or > multiple files owned by 1000:1000. > > For my case, the "partial fix" did not help. However I noticed that if I > disable memory DB (#undef USE_MEMORY_DB) and clean (bitbake -c clean) > base-files, the permissions in the output IPK always look good (37 out of 37 > times so far). If I enable memory DB, then bitbake clean base-files and run > again, I get 1 good IPK (owner 0) and then only bad IPKs (owner 1000) (at > least 8 bad IPKs at the moment). > > My test is as follows, in a loop: > > bitbake base-files -C package -c package_write_ipk > FILES=$(ar p $IPK data.tar.xz | tar tvJ) > if echo "$FILES" | grep -m1 1000; then > echo BAD; > else > echo GOOD; > fi > > > I hit the problem originally on a higher level (build image) - running > "base-files -C package -c package_write_ipk" is just to speed up the > testing. However, I have only tested the "partial fix" with this test. I suspect that w/o memory DB pseudo runs much slower, which helps reducing running into race conditions. I tried pseudo w/o memory DB, but it still generated the error. ( I tried with looping "bitbake glibc-locale" , there's way more work for pseudo involved) With new pseudo from current oe-core/master and warrior (SRCREV = "3fa7c853e0bcd6fe23f7524c2a3c9e3af90901c3") I did 2000 glibc-locale rebuilds and didn't trigger the issue once (before it was usually triggered at least once in first few hundred) But I have started seeing host-user-contamination again from files installed by qmake (it was quite common with qmake 5.9, then got resolved with 5.10 and newer until recent rework to be compatible with new pseudo, details in: https://github.com/meta-qt5/meta-qt5/issues/187#issuecomment-484730210 ) So I believe some issue which resulted in host-user-contamination was fixed, but not all (and qmake difference is probably the best lead atm, but still no reliable reproducer based on that). (In reply to comment #45) > With new pseudo from current oe-core/master and warrior (SRCREV = > "3fa7c853e0bcd6fe23f7524c2a3c9e3af90901c3") I did 2000 glibc-locale rebuilds > and didn't trigger the issue once (before it was usually triggered at least > once in first few hundred) > > But I have started seeing host-user-contamination again from files installed > by qmake (it was quite common with qmake 5.9, then got resolved with 5.10 > and newer until recent rework to be compatible with new pseudo, details in: > https://github.com/meta-qt5/meta-qt5/issues/187#issuecomment-484730210 ) > > So I believe some issue which resulted in host-user-contamination was fixed, > but not all (and qmake difference is probably the best lead atm, but still > no reliable reproducer based on that). I should have probably mentioned I used an 88 core machine for stress testing. I can still detect the problem in glibc-locale, typically within 15 iterations. I used the latest "pseudo" master ( top commit http://git.yoctoproject.org/cgit/cgit.cgi/pseudo/commit/?id=3fa7c853e0bcd6fe23f7524c2a3c9e3af90901c3 ) We've seen this error 3 times since April 22th, 2019 in our automated build farm (builds in docker, lots of cpus and lots of load). I made some progress today on understanding what's special about the recipes which use qmake for installation (and often trigger this bug), sorry for bad formatting, I'm using the same comment in internal Jira and don't know how to properly noformat in bugzilla. 1) I've created simple reproducer: http://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/pseudo2 cherry-pick last 2 commits and just bitbake pseudo-test, should trigger that warning on ~10 files 2) it's failing for recipes installed with qmake since qt 5.9, because 5.9.0 is where qmake -install was introduced as QINSTALL in Makefiles with this commit: https://code.qt.io/cgit/qt/qtbase.git/commit/?id=2ad7f6ddf5042d7442c97a89b083ca2853cf5721 e.g. Qt5Gui_QTuioTouchPlugin.cmake file from ERROR: QA Issue: qtbase: /qtbase-dev/usr/lib/cmake/Qt5Gui/Qt5Gui_QTuioTouchPlugin.cmake is owned by uid 1101, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] is in Qt 5.6 installed with /src/plugins/generic/tuiotouch/Makefile {noformat} install_cmake_qt5_plugin_file: first FORCE @test -d $(INSTALL_ROOT)/usr/lib/cmake/Qt5Gui || mkdir -p $(INSTALL_ROOT)/usr/lib/cmake/Qt5Gui -$(INSTALL_FILE) /jenkins/mjansa/build-webos-ose-thud/BUILD/work/qemux86-webos-linux/qtbase/5.6.3+gitAUTOINC+e6f8b072d2-r0/build/lib/cmake/Qt5Gui/Qt5Gui_QTuioTouchPlugin.cmake $(INSTALL_ROOT)/usr/lib/cmake/Qt5Gui/ {noformat} where {noformat} INSTALL_FILE = install -m 644 -p INSTALL_PROGRAM = install -m 755 -p INSTALL_DIR = cp -f -R {noformat} In Qt 5.9 and newer it's: {noformat} install_cmake_qt5_plugin_file: first FORCE @test -d $(INSTALL_ROOT)/usr/lib/cmake/Qt5Gui || mkdir -p $(INSTALL_ROOT)/usr/lib/cmake/Qt5Gui -$(QINSTALL) /jenkins/mjansa/build-nodistro-master/BUILD/work/i586-oe-linux/qtbase/5.12.3+gitAUTOINC+b527725766-r0/build/lib/cmake/Qt5Gui/Qt5Gui_QTuioTouchPlugin.cmake $(INSTALL_ROOT)/usr/lib/cmake/Qt5Gui/Qt5Gui_QTuioTouchPlugin.cmake {noformat} {noformat} INSTALL_FILE = install -m 644 -p INSTALL_PROGRAM = install -m 755 -p INSTALL_DIR = cp -f -R QINSTALL = /jenkins/mjansa/build-nodistro-master/BUILD/work/i586-oe-linux/qtbase/5.12.3+gitAUTOINC+b527725766-r0/recipe-sysroot-native/usr/bin/qmake -install qinstall QINSTALL_PROGRAM = /jenkins/mjansa/build-nodistro-master/BUILD/work/i586-oe-linux/qtbase/5.12.3+gitAUTOINC+b527725766-r0/recipe-sysroot-native/usr/bin/qmake -install qinstall -exe {noformat} 3) it's still failing with latest 5.12 even after the renameat2 wrapper (doing the direct syscall) was removed in: https://code.qt.io/cgit/qt/qtbase.git/commit/src/corelib/io/qfilesystemengine_unix.cpp?id=b7887f9b4faad2227691a2af589e9d7680d6ae08 4) when the renameat2 is explicitly enabled (with -feature-renameat2 and 0018-Avoid-renameeat2-for-native-sdk-builds.patch removed from qtbase-native.bb), then it fails installing the files: But for working renameat2 (without using the removed wrapper for direct syscall) you need to use host OS with newer glibc. Ubuntu 18.04 we're currently using has: ii libc6:amd64 2.27-3ubuntu1 amd64 ii coreutils 8.28-1ubuntu1 amd64 18.10: ii libc6:amd64 2.28-0ubuntu1 amd64 ii coreutils 8.28-1ubuntu2 amd64 19.04: ii libc6:amd64 2.29-0ubuntu2 amd64 ii coreutils 8.30-1ubuntu1 amd64 https://github.com/meta-qt5/meta-qt5/issues/187#issuecomment-484296968 With 18.04 qtbase-native 5.12.3 do_configure fails if you try to explicitly enable -feature-renameat2 without having the call in glibc {noformat} ERROR: Feature 'renameat2' was enabled, but the pre-condition 'config.linux && tests.renameat2' failed. {noformat} because the config.test for it failed (as expected): {noformat} > main.cpp:9:53: error: �RENAME_NOREPLACE� was not declared in this scope > main.cpp:9:72: error: �RENAME_WHITEOUT� was not declared in this scope {noformat} With Ubuntu 19.04 which has glibc 2.29-0ubuntu2, you can explicitly enable renameat2, but then it will fails in do_install, because pseudo returns ENOSYS while https://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/io/qfilesystemengine_unix.cpp#n1256 checks for EINVAL, causing: {noformat} Error copying /OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/pseudo-test/1.0-r0/pseudo-test-1.0/005.txt to /OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/pseudo-test/1.0-r0/image/005.txt: Cannot create /OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/pseudo-test/1.0-r0/image/005.txt for output {noformat} This error is from QFile::copy https://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/io/qfile.cpp#n843 where QFile::rename -> QFileSystemEngine::renameFile is called in cases where the destination already exists (that's why the mv in the recipe might be needed to trigger this issue). In some cases the do_install fails even with renameat2 disabled, e.g.: {noformat} Error copying /OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/pseudo-test/1.0-r0/pseudo-test-1.0/101.txt to /OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/pseudo-test/1.0-r0/image/101.txt: Destination file exists make: *** [Makefile:203: R101] Error 3 make: *** Waiting for unfinished jobs.... Error copying /OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/pseudo-test/1.0-r0/pseudo-test-1.0/118.txt to /OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/pseudo-test/1.0-r0/image/118.txt: Cannot create /OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/pseudo-test/1.0-r0/image/118.txt for output make: *** [Makefile:237: R118] Error 3 {noformat} I need to debug qmake -install qinstall a bit more to narrow down the paths when the do_install doesn't fail, but the pseudo db still gets into inconsistent state. For that I'll add more debug output in qtbase-native first, to see what's really going on in various combinations of new/old pseudo, new/old glibc, qtbase with/without syscall wrapper. Interesting! Note that renameat2() in pseudo is quite special and I think that pseudo's behaviour changes depending on what the host glibc is when it was built. We merged: 0fb257121b pseudo: Update to gain key bugfixes CommitDate: Thu Apr 11 15:22:44 2019 +0100 into our build environment on 2019-04-15. There have been 8 glibc-locale gid/uid failures after the merge: 4/22/19, 9:10 AM ET 5/1/19, 5:40 AM ET 5/8/19, 9:10 AM ET 5/10/19, 12:20 AM ET 5/21/19, 10:50 PM ET 5/24/19, 5:00 PM ET 6/21/19, 8:40 AM ET 6/23/19, 3:00 AM ET as of 6/24/2019 10:40 AM ET. There is a claim that another test environment can more easily reproduce the error so I'm trying to get a developer to have access to that environment. |