7724 2015-05-06 18:38:24 +0000 useradd.bbclass users cannot be safely referred to by dependent recipes due to sstate-induced race with base-passwd 2017-04-13 17:52:37 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 1.7 All Multiple RESOLVED FIXED TC needed Medium major 2.4 M1 1 rich.tollerton leonardo.sandoval.gonzalez alexandru.c.georgescu ben.shelton bluelightning haris.okanovic jose.a.lamego juan.m.cruz.alcaraz leonardo.sandoval.gonzalez maxin.john meta.mr.watcher meta.watcher randy.macleod rich.tollerton richard.purdie costin.c.constantin --- --- Don't know oldest_to_newest 50831 0 rich.tollerton 2015-05-06 18:38:24 +0000 I have observed an organic situation today (not doing anything wrong AFAIK) where - base-passwd gets rebuilt due to a nominally unrelated change (specifically adding a couple of CVS patches to the glibc build) - a useradd.bbclass-inheriting package does *not* get rebuilt, and its setscene runs before base-passwd gets rebuilt - a dependent package fails because the new user is missing from /etc/passwd This build failure is persistent; it will continue until the useradd-inheriting recipe gets force-rebuilt. (Also note that this recipe will not appear in the build failure logs.) AFAIK, the root problem seems to be that if base-passwd repopulates sysroot and a useradd package is cached, there isn't anything that forces useradd_preint() to be run again. To reproduce: 1) Build a clean openembedded-core (I built core-image-minimal from master) 2) Create meta/recipes-core/useraddtest-a.bb: SUMMARY = "useradd test A" LICENSE = "MIT" LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690 \ file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" ALLOW_EMPTY_${PN} = "1" inherit allarch useradd USERADD_PACKAGES = "${PN}" GROUPADD_PARAM_${PN} = " --system foo" USERADD_PARAM_${PN} = "-N -g foo foo" 3) Create meta/recipes-core/useraddtest-b_1.0.bb: DEPENDS += "useraddtest-a" LICENSE = "MIT" LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690 \ file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" FILES_${PN} = "${datadir}/foo" do_install () { mkdir -p ${D}${datadir} echo foo > ${D}${datadir}/foo chown foo:foo ${D}${datadir}/foo } 4) Set CORE_IMAGE_EXTRA_INSTALL="useraddtest-b" in local.conf. 5) Rebuild core-image-minimal. 6) Make trivial changes to *both* meta/recipes-core/base-passwd/base-passwd_3.5.29.bb and meta/recipes-core/useraddtest/useraddtest-b_1.0.bb to invalidate their sstate caches and trigger a rebuild. For base-passwd_3.5.29.bb, I added a space between URIs in SRC_URI. For useraddtest-b_1.0.bb, I changed "echo foo" to "echo foo2". 7) Rebuild core-image-minimal. This will fail with the following error: Log data follows: | DEBUG: Executing shell function do_install | chown: invalid user: 'foo:foo' | WARNING: /ssd/rtollert/oe-core-build/tmp-glibc/work/i586-oe-linux/useraddtest-b/1.0-r0/temp/run.do_install.15042:1 exit 1 from | chown foo:foo /ssd/rtollert/oe-core-build/tmp-glibc/work/i586-oe-linux/useraddtest-b/1.0-r0/image/usr/share/foo | ERROR: Function failed: do_install (log file is located at /ssd/rtollert/oe-core-build/tmp-glibc/work/i586-oe-linux/useraddtest-b/1.0-r0/temp/log.do_install.15042) ERROR: Task 261 (/mnt/rtollert/src/openembedded-core/meta/recipes-core/useraddtest/useraddtest-b_1.0.bb, do_install) failed with exit code '1' Workarounds: - Invalidate sstate cache of useradd-inheriting package - Delete sstate cache - Avoid referring to users/groups created by useradd-inheriting packages in other packages. (This may not be possible depending on the situation.) 50854 1 sgw 2015-05-07 15:07:16 +0000 Please assign to someone in GDC to become new owner for useradd issues, this also needs unit tests. 51806 2 leonardo.sandoval.gonzalez 2015-06-18 19:04:05 +0000 Patch is now on master: 5d426794f9a9d3ca4ffd21b0c5a300e3daeef4b9 55744 3 costin.c.constantin 2015-10-26 14:31:04 +0000 Hello, Just tested this on latest master: on an Ubuntu machine: 505a82673ac2487df5ea343a6422c2fc47018831 I followed comment#1 directions and i get the same error message as described: ~/costin-poky/build$ bitbake core-image-minimal ... NOTE: Executing RunQueue Tasks ERROR: Function failed: do_install (log file is located at /home/autobuilder/costin-poky/build/tmp/work/i586-poky-linux/useraddtestb/1.0-r0/temp/log.do_install.22591) ERROR: Logfile of failure stored in: /home/autobuilder/costin-poky/build/tmp/work/i586-poky-linux/useraddtestb/1.0-r0/temp/log.do_install.22591 Log data follows: | DEBUG: Executing shell function do_install | chown: invalid user: 'foo:foo' | WARNING: exit code 1 from a shell command. | ERROR: Function failed: do_install (log file is located at /home/autobuilder/costin-poky/build/tmp/work/i586-poky-linux/useraddtestb/1.0-r0/temp/log.do_install.22591) ERROR: Task 256 (/home/autobuilder/costin-poky/meta/recipes-core/useradd_tests/useraddtestb.bb, do_install) failed with exit code '1' NOTE: Tasks Summary: Attempted 2023 tasks of which 2017 didn't need to be rerun and 1 failed. 55768 4 leonardo.sandoval.gonzalez 2015-10-26 16:20:25 +0000 (In reply to comment #3) > Hello, > > Just tested this on latest master: > > on an Ubuntu machine: 505a82673ac2487df5ea343a6422c2fc47018831 > > I followed comment#1 directions and i get the same error message as > described: > there must be something on a configuration file that do not match between mine and yours. I just created a core image with those two recipes included, started from a clean tmp folder, and did not notice anything bad after baking. I have pushed my dev branch on http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=lsandov1/useradd can you give it a try on a clean tmp folder? You can either use my branch or use yours, it should not matter. In the other hand, this is how my local.conf looks like MACHINE ??= "qemux86-64" DISTRO ?= "poky" PACKAGE_CLASSES ?= "package_rpm package_ipk package_deb" EXTRA_IMAGE_FEATURES = "debug-tweaks" USER_CLASSES ?= "buildstats image-mklibs image-prelink" BB_DISKMON_DIRS = "\ STOPTASKS,${TMPDIR},1G,100K \ STOPTASKS,${DL_DIR},1G,100K \ STOPTASKS,${SSTATE_DIR},1G,100K \ ABORT,${TMPDIR},100M,1K \ ABORT,${DL_DIR},100M,1K \ ABORT,${SSTATE_DIR},100M,1K" PACKAGECONFIG_append_pn-qemu-native = " sdl" PACKAGECONFIG_append_pn-nativesdk-qemu = " sdl" ASSUME_PROVIDED += "libsdl-native" CONF_VERSION = "1" INHERIT += "own-mirrors" SSTATE_MIRRORS ?= "file://.* http://yocto-ab-master.jf.intel.com/sstate/PATH" SOURCE_MIRROR_URL ?= "file:///srv/current_sources" CORE_IMAGE_EXTRA_INSTALL = "useraddtest-b" > ~/costin-poky/build$ bitbake core-image-minimal > ... > NOTE: Executing RunQueue Tasks > ERROR: Function failed: do_install (log file is located at > /home/autobuilder/costin-poky/build/tmp/work/i586-poky-linux/useraddtestb/1. > 0-r0/temp/log.do_install.22591) > ERROR: Logfile of failure stored in: > /home/autobuilder/costin-poky/build/tmp/work/i586-poky-linux/useraddtestb/1. > 0-r0/temp/log.do_install.22591 > Log data follows: > | DEBUG: Executing shell function do_install > | chown: invalid user: 'foo:foo' > | WARNING: exit code 1 from a shell command. > | ERROR: Function failed: do_install (log file is located at > /home/autobuilder/costin-poky/build/tmp/work/i586-poky-linux/useraddtestb/1. > 0-r0/temp/log.do_install.22591) > ERROR: Task 256 > (/home/autobuilder/costin-poky/meta/recipes-core/useradd_tests/useraddtestb. > bb, do_install) failed with exit code '1' > NOTE: Tasks Summary: Attempted 2023 tasks of which 2017 didn't need to be > rerun and 1 failed. 56441 5 sgw 2015-11-12 16:31:30 +0000 It appears that Leo assigned this to Constin to check an updated version in a private branch, please take a look. 56448 6 sjolley.yp.pm 2015-11-12 18:33:49 +0000 Question asked. 57139 7 haris.okanovic 2015-12-08 23:35:02 +0000 *** Bug 8759 has been marked as a duplicate of this bug. *** 57743 8 costin.c.constantin 2016-01-12 13:51:49 +0000 Hi Leo, Sorry for the late reply. I just re-tested this and still get the same output as on comment #3. This time i did everything on a clean env. by freshly cloning poky. The only things that may relate to something already existing are these two lines i use to speed up build in local.conf: ... DL_DIR ?= "/media/sstate/downloads" SSTATE_DIR ?= "/media/sstate" ... The rest stays the same as when clone done. My build screen looks like this: BB_VERSION = "1.28.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "Ubuntu-15.04" TARGET_SYS = "i686-poky-linux" MACHINE = "genericx86" DISTRO = "poky" ... meta-yocto-bsp = "master:1724ffd32d1c169a2de66f2cb88ba18c56da6146" I will give it a shot without sstate and downloads remote dirs. Will post the result of that shortly. 57747 9 costin.c.constantin 2016-01-12 14:53:59 +0000 Hi, Did the same as per last comment without external DL_DIR and SSTATE_DIR and got the same issue. 58686 10 leonardo.sandoval.gonzalez 2016-02-05 22:00:59 +0000 Missed 2.0.1. Moved to 2.0.2 60806 11 leonardo.sandoval.gonzalez 2016-04-05 15:20:53 +0000 Problem is indeed present on current master [1], although problem is reported for 2.0.2. Working on it. [1] commit 1ca71e5178ed8514655524f714877e02f6db90af Refs: {origin/HEAD}, {origin/master}, 2.1_M1-2261-g1ca71e5 Author: Richard Purdie <richard.purdie@linuxfoundation.org> AuthorDate: Tue Mar 29 16:09:52 2016 +0100 60823 12 leonardo.sandoval.gonzalez 2016-04-05 21:34:05 +0000 The error show is actually expected: the reason is that the useraddtest-b change does not trigger the prefunc do_install function which actually add the groups and users needed. Under this scenario, the task useraddtest-a's do_install must be tainted, so when building core-image-minimal again (including useraddtest-b), useraddtest-a's do_install (and its prefunc) is executed, then useraddtest-b is built again and properly added into the final image. These are the commands to be used for this scenario once [1] is done: bitbake -f -c install useraddtest-a bitbake core-image-minimal [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=7724#c0 60825 13 leonardo.sandoval.gonzalez 2016-04-05 21:52:24 +0000 Reopening to review the current status and the proposed workaround. 60963 14 maxin.john 2016-04-08 07:58:52 +0000 (In reply to comment #12) > The error show is actually expected: the reason is that the useraddtest-b > change does not trigger the prefunc do_install function which actually add > the groups and users needed. > > Under this scenario, the task useraddtest-a's do_install must be tainted, so > when building core-image-minimal again (including useraddtest-b), > useraddtest-a's do_install (and its prefunc) is executed, then useraddtest-b > is built again and properly added into the final image. > > These are the commands to be used for this scenario once [1] is done: > > bitbake -f -c install useraddtest-a > bitbake core-image-minimal > > > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=7724#c0 Almost the same observation for similar scenario in Bug #8078. It can be achieved by adding the below line in useradd.bbclass. do_install[nostamp] = "1" However, this causes longer rebuild time. 60981 15 leonardo.sandoval.gonzalez 2016-04-08 17:16:07 +0000 > Almost the same observation for similar scenario in Bug #8078. > It can be achieved by adding the below line in useradd.bbclass. > > do_install[nostamp] = "1" Right, that is another option. > > However, this causes longer rebuild time. The reason of this behavior was introduced by this commit: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=0aed04a49ec13c3331169ca4fd2c820b03faf19c 60982 16 richard.purdie 2016-04-08 17:24:39 +0000 To give some better information about what is going on here, the issue is that base-passwd is listed in layer.conf in SIGGEN_EXCLUDERECIPES_ABISAFE. This was done since base-passwd is not allarch and without this, you couldn't have an allarch recipe which added users. If we agree that allarch recipes shouldn't be able to add users, you could revert that however I suspect people would have some issues with that. The next solution could be to save copies of the original master/group entries, then upon sstate uninstall, remove these, upon addition, add in the new entries and never overwrite the files completely. Another option would be to find some cleverer way of handling SIGGEN_EXCLUDERECIPES_ABISAFE such that some types of changes are reflected in the sstate hash but I can't see a way to make this work. 61527 17 leonardo.sandoval.gonzalez 2016-04-21 19:00:34 +0000 This problem could not be resolved on time, so moving it to 2.2 M1. Also, it is present on jethro AND master (krogoth). 70022 18 richard.purdie 2017-01-23 15:56:48 +0000 I'd suggest this is retested now recipe specific sysroots has merged as I'm hoping this solves the problem. 71832 19 leonardo.sandoval.gonzalez 2017-03-28 21:10:17 +0000 Following the steps on current HEAD, now the error is produced on step 4 with the following log: | NOTE: gettext-minimal-native exists in sysroot, skipping | DEBUG: Python function extend_recipe_sysroot finished | DEBUG: Executing shell function do_install | chown: invalid user: ‘foo:foo’ | WARNING: /home/lsandov1/poky/build/tmp/work/i586-poky-linux/useraddtest-b/1.0-r0/temp/run.do_install.48180:1 exit 1 from 'chown foo:foo /home/lsandov1/poky/build/tmp/work/i586-poky-linux/useraddtest-b/1.0-r0/image/usr/share/foo' | ERROR: Function failed: do_install (log file is located at /home/lsandov1/poky/build/tmp/work/i586-poky-linux/useraddtest-b/1.0-r0/temp/log.do_install.48180) ERROR: Task (/home/lsandov1/poky/meta-useradd/recipes-core/useradd/useraddtest-b_1.0.bb:do_install) failed with exit code '1' NOTE: Tasks Summary: Attempted 2404 tasks of which 2396 didn't need to be rerun and 1 failed. Summary: 1 task failed: /home/lsandov1/poky/meta-useradd/recipes-core/useradd/useraddtest-b_1.0.bb:do_install Summary: There was 1 ERROR message shown, returning a non-zero exit code. next step is to check if this is expected base on comment 18 71911 20 leonardo.sandoval.gonzalez 2017-03-30 15:11:14 +0000 useradd related issues will be targeted at 2.4, so moving it. 72485 21 leonardo.sandoval.gonzalez 2017-04-13 17:52:37 +0000 Fixed by [1]. For further reference [2], the useradd test recipes were created as provided in the comment 0 but 1) including a base folder (useradd) and including DEPENDS += "shadow-native" on the b recipe. [1] http://lists.openembedded.org/pipermail/openembedded-core/2017-April/135611.html [2] http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=lsandov1/useradd-bug7724-scenario