| Summary: | glibc-locale warning [host-user-contaminated] | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Juro Bystricky <juro.bystricky> | ||||
| Component: | core | Assignee: | Chen Qi <Qi.Chen> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | Medium+ | CC: | akuster, alex.kanavin, joshuagloe, liezhi.yang, meta.mr.watcher, meta.watcher, opello, randy.macleod, richard.purdie, ross.burton | ||||
| Version: | 2.3 | ||||||
| Target Milestone: | 3.1 | ||||||
| Hardware: | x86 | ||||||
| OS: | Multiple | ||||||
| See Also: | https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434 | ||||||
| Whiteboard: | |||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||
| Attachments: |
|
||||||
|
Description
Juro Bystricky
2017-04-05 20:19:23 UTC
Seems similar problem has been observed on rare occasions even in past releases, most likely pseudo related. FWIW, I observed this building in a container on a 88 core system /Ubuntu 14. The plan here is to first find a way to reproduce the problem reliably. Created attachment 3704 [details]
package claimed to have problems by sanity check.
Ran into the issue again today, clean build (wiped sstate/tmpdir): PACKAGE_CLASSES ?= "package_rpm" pokyuser@2b9618dd63e8:/workdir/poky/build-repro-1$ bitbake core-image-minimal WARNING: Host distribution "ubuntu-14.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Parsing recipes: 100% |##################################################################################################################################################################| Time: 0:00:05 Parsing of 830 .bb files complete (0 cached, 830 parsed). 1298 targets, 48 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.33.3" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "ubuntu-14.04" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.2+snapshot-20170410" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:633ad6c9f436f5d2b6ee1a005b697661a054a394" Initialising tasks: 100% |###############################################################################################################################################################| Time: 0:00:04 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks WARNING: glibc-locale-2.25-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-de-lu.iso-8859-1/usr/lib/locale/de_LU.ISO-8859-1/LC_COLLATE is owned by uid 1001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] But, curiously enough, the package seems to be OK. (attached) Again: pokyuser@2b9618dd63e8:/workdir/poky/build-repro-1$ bitbake glibc-locale WARNING: Host distribution "ubuntu-14.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Loading cache: 100% |####################################################################################################################################################################| Time: 0:00:00 Loaded 1298 entries from dependency cache. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.33.3" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal-4.8" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.2+snapshot-20170411" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:633ad6c9f436f5d2b6ee1a005b697661a054a394" Initialising tasks: 100% |###############################################################################################################################################################| Time: 0:00:00 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks WARNING: glibc-locale-2.25-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-mk-mk/usr/lib/locale/mk_MK/LC_TELEPHONE is owned by uid 1001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] Another observation: <buildir>tmp/work/i586-poky-linux/glibc-locale/2.25-r0/pseudo/pseudo.log contains 18733 various "path mismatch" messages. That cannot be right. So it turns out it is fairly easy to reproduce the problem with a stress test, on a 88 thread server I get more than a 10% failure rate. I use the poky git repo: meta-yocto-bsp = "master:1423508b29fc557d8a1305f39c33de33e28d9003" FWIW, the commit c5269fd2108d66623515291481c4c24e93be805b (master-next:pseudo: Backport two upstream fixes) did not help. just verified still happens with "pyro": $ bitbake glibc-locale Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal-4.8" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "pyro:7a0e795373653886452a7a2992ced10080711c26" Initialising tasks: 100% |###############################################################################################################################################################| Time: 0:00:00 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks WARNING: glibc-locale-2.25-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-ca-ad.iso-8859-15/usr/lib/locale/ca_AD.ISO-8859-15/LC_NUMERIC is owned by uid 1001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] *** Bug 8447 has been marked as a duplicate of this bug. *** I can verify that this isn't a weird caching problem: Buildhistory says: diff --git a/packages/corei7-64-poky-linux/glibc-locale/glibc-binary-localedata-lb-lu/files-in-package.txt b/packages/corei7-64-poky-linux/glibc-locale/glibc-binary-localedata-lb-lu/files-in-package.txt index 07a4873..70f2ed3 100644 --- a/packages/corei7-64-poky-linux/glibc-locale/glibc-binary-localedata-lb-lu/files-in-package.txt +++ b/packages/corei7-64-poky-linux/glibc-locale/glibc-binary-localedata-lb-lu/files-in-package.txt @@ -9,7 +9,7 @@ drwxr-xr-x root root 280 ./usr/lib/locale/lb_LU --rw-r--r-- root root 294 ./usr/lib/locale/lb_LU/LC_MONETARY +-rw-r--r-- 1000 1000 294 ./usr/lib/locale/lb_LU/LC_MONETARY $ dpkg -c glibc-binary-localedata-lb-lu*ipk | grep MON -rw-r--r-- 1000/1000 294 2017-05-11 17:27 ./usr/lib/locale/lb_LU/LC_MONETARY The actual package has the incorrect permissions written in. (In reply to comment #9) > I can verify that this isn't a weird caching problem: > > Buildhistory says: > > diff --git > a/packages/corei7-64-poky-linux/glibc-locale/glibc-binary-localedata-lb-lu/ > files-in-package.txt > b/packages/corei7-64-poky-linux/glibc-locale/glibc-binary-localedata-lb-lu/ > files-in-package.txt > index 07a4873..70f2ed3 100644 > --- > a/packages/corei7-64-poky-linux/glibc-locale/glibc-binary-localedata-lb-lu/ > files-in-package.txt > +++ > b/packages/corei7-64-poky-linux/glibc-locale/glibc-binary-localedata-lb-lu/ > files-in-package.txt > @@ -9,7 +9,7 @@ drwxr-xr-x root root 280 > ./usr/lib/locale/lb_LU > --rw-r--r-- root root 294 > ./usr/lib/locale/lb_LU/LC_MONETARY > +-rw-r--r-- 1000 1000 294 > ./usr/lib/locale/lb_LU/LC_MONETARY > > $ dpkg -c glibc-binary-localedata-lb-lu*ipk | grep MON > -rw-r--r-- 1000/1000 294 2017-05-11 17:27 > ./usr/lib/locale/lb_LU/LC_MONETARY > > The actual package has the incorrect permissions written in. Thanks, I saved some builds for post-morem analysis for later, so I can go back and probably retract my statement the actual package has correct ownership. Also, I can reproduce it if needs to be. Since there are actual errors, I will pull this from the back-burner to the front-burner. Something is different in my case. I used RPM packages, don't think it matters. In my preserved tmp folder, I have file log.do_package_qa with: NOTE: Checking Package: glibc-binary-localedata-gl-es.iso-8859-1 WARNING: QA Issue: glibc-locale: /glibc-binary-localedata-gl-es.iso-8859-1/usr/lib/locale/gl_ES.ISO-8859-1/LC_MEASUREMENT is owned by uid 1001, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated] So I pulled glibc-binary-localedata-gl-es.iso-8859-1-2.25-r0.i586.rpm from the tmp folder and checked: $ rpm -qlpv ./glibc-binary-localedata-gl-es.iso-8859-1-2.25-r0.i586.rpm drwxr-xr-x 2 root root 0 Apr 24 03:31 /usr drwxr-xr-x 2 root root 0 Apr 24 03:31 /usr/lib drwxr-xr-x 2 root root 0 Apr 24 03:31 /usr/lib/locale drwxr-xr-x 2 root root 0 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1 -rw-r--r-- 1 root root 154 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_ADDRESS -rw-r--r-- 1 root root 19459 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_COLLATE -rw-r--r-- 1 root root 274132 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_CTYPE -rw-r--r-- 1 root root 369 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_IDENTIFICATION -rw-r--r-- 1 root root 28 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_MEASUREMENT drwxr-xr-x 2 root root 0 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_MESSAGES -rw-r--r-- 1 root root 64 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_MESSAGES/SYS_LC_MESSAGES -rw-r--r-- 1 root root 299 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_MONETARY -rw-r--r-- 1 root root 67 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_NAME -rw-r--r-- 1 root root 59 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_NUMERIC -rw-r--r-- 1 root root 39 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_PAPER -rw-r--r-- 1 root root 54 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_TELEPHONE -rw-r--r-- 1 root root 2351 Apr 24 03:31 /usr/lib/locale/gl_ES.ISO-8859-1/LC_TIME So in this case, the ownership is OK. I will redo this again with .ipk packages. I reproduced the problem again with .ipk package, the warning being: WARNING: glibc-locale-2.25-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-nn-no.iso-8859-1/usr/lib/locale/nn_NO.ISO-8859-1/LC_MEASUREMENT is owned by uid 1001 verifying the package contents I got this: dpkg -c glibc-binary-localedata-nn*ipk | grep MEA -rw-r--r-- 1001/100 28 2017-05-12 01:08 ./usr/lib/locale/nn_NO.ISO-8859-1/LC_MEASUREMENT So LC_MEASUREMENT does have wrong ownership. I'll retest the RPM again so I have something to think about. I suspect the RPM packages end up having correct ownership because the RPM packager will set correct ownership after QA, fixing any incorrect owners we were warned about as a by-product. Just some additional observations: 1. in case of the ownership error, the database file "files.db" contains a record which has the incorrect owner (the owner is not root). 2. Modyfing the pseudo code (pseudo_db.c) to wait for file flush via: "PRAGMA synchronous = EXTRA;" did not solve the problem. My suspicion is that we delete files outside pseudo context, the inode numbers is reused and some of the attributes of a deleted file are mapped to a newly created file which has the same inode number. Proving this is the case is tricky but it might help people investigate further. (In reply to comment #15) > My suspicion is that we delete files outside pseudo context, the inode > numbers is reused and some of the attributes of a deleted file are mapped to > a newly created file which has the same inode number. Proving this is the > case is tricky but it might help people investigate further. I tend to think now pseudo is not the guilty part. There is locale file manipulation going on outside pseudo and pseudo gets confused. Generally, all locale file accesses should be done under pseudo, not just do_install. But that creates other problems. BTW, I can reproduce the problem (stress testing) even with current master. So adding 'fakeroot' to all the tasks that do file mangling isn't enough? (In reply to comment #17) > So adding 'fakeroot' to all the tasks that do file mangling isn't enough? I tried that a while ago, but all kinds of files ended up being owned by root and that created some build errors. But in principle, it should fix the problem, I think. I was curious as to when this became an issue. At first it seemed jethro did not have the problem, so I ran a series of lengthy bisects and got this: b95c3404432cb8986533c52a16a13f68b200c7a3 is the first bad commit (insane.bbclass: handle tests which need fakeroot) So basically once we had the QA test, the error showed up right away. So not seeing this in jethro means there simply was no QA test to detect the ownership problem. It is safe to assume the problem was there all along, but went undetected. Having an incorrect/stale record in the database could be in principle caused by sqlite. So I tried pseudo (Rocko) with modified settings, such as: "PRAGMA locking_mode = NORMAL;", "PRAGMA busy_timeout = 2000;" Still observed failures. I updated sqlite to the latest version (3.21.0) Still observed failures. BB_VERSION = "1.37.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.4+snapshot-20171215" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "master:b73e96e7f3f5d1ba3a221d99792a7a3c7ef42c21" The problem is still present in "sumo". *** Bug 12976 has been marked as a duplicate of this bug. *** After reading through the mailing list[1] threads[2] I could find on the host-user-contaminated warning and seeing it with many (most? all? not certain.) -dbg packages I spot tested with one recipe (libepoxy) that reliably showed the issue for me. The patch from this post[3] appears to resolve the issue. The pseudo/files.db in the work directory doesn't contain the 'uid <> 0' entries that were there when the warnings were displayed after the patch is applied. Since this is from some time ago I wasn't sure if the most appropriate place for the fix was pseudo or if it was worked around in some other way that I might not be seeing. [1]: http://lists.openembedded.org/pipermail/openembedded-core/2016-April/thread.html#238260 [2]: http://lists.openembedded.org/pipermail/openembedded-core/2016-January/thread.html#234085 [3]: http://lists.openembedded.org/pipermail/openembedded-core/2016-April/119863.html I just met this warning for latest master.
Checking at the recipe, the 'cp' command seems to be a candidate.
cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
Maybe we should also put 'ownership' in the preserve list?
--preserve=mode,links,ownership
Unfortunately, I don't have any reliable way to reproduce the problem...
(In reply to comment #24) > I just met this warning for latest master. > Checking at the recipe, the 'cp' command seems to be a candidate. > cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* > ${D}${localedir} > > Maybe we should also put 'ownership' in the preserve list? > --preserve=mode,links,ownership > > Unfortunately, I don't have any reliable way to reproduce the problem... After checking the history and did more tests, my above comments are totally wrong. http://git.yoctoproject.org/cgit.cgi/pseudo/commit/?id=ed20f8323a9676e4e2c14bda06aa07fb1a1ba8ae might be the problem. There is probably a second piece to this we still need to figure out but this would help fix it. (In reply to comment #26) > http://git.yoctoproject.org/cgit.cgi/pseudo/commit/ > ?id=ed20f8323a9676e4e2c14bda06aa07fb1a1ba8ae might be the problem. There is > probably a second piece to this we still need to figure out but this would > help fix it. I tried the patch with "thud", running a stress test program. The problem remains. (I added the patch to "pseudo" recipe, the last commit there being "also make sure inodes are 64-bit values to SQL") Even with pseudo update, the problem is still there. (In reply to comment #29) > http://git.yoctoproject.org/cgit.cgi/poky/commit/ > ?id=8102c55bc1851233d3c5632e47e0adfddc4b23f8 This looks very promising. I have been running stress tests on 88cores for the last hour or so and no problems so far. I will run few more hours. If it works than hats off to Jason. We were seeing a glibc-locale uid/gid build error every two days on average on the master branch builds. The have stopped happening on August 21 just after we rebased our oe-core tree with Jason's fix and haven't happened in the past 15 days at all so I'm convinced. I ran stress tests for over 24 hours on a 88 core machine. No problems were detected. (The test used to find a problem within 10-15 minutes). So it is pretty safe to say the problem was fixed. Kudos to Jason Wessel. |