Created attachment 3859 [details] useradd not found multilib In *some* multilib builds, useradd is isn't found in the RSS: Adding dependency on libxdmcp-native Considering setscene task: ['libxslt-native', 'do_populate_sysroot'] considering dependency: ['libxslt-native', 'do_populate_sysroot'] considering dependency: ['lib32-xkeyboard-config', 'do_populate_sysroot'] Skipping setscene dependency virtual:native:/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/layers/oe-core/meta/recipes-support/libxslt/libxslt_1.1.29.bb:do_populate_sysroot for installation into the sysroot NOTE: quilt-native exists in sysroot, skipping NOTE: sed -e 's:^[^/]*/:/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-avahi/0.6.32-r0/lib32-recipe-sysroot/:g' /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/armv7ahf-neon/lib32-libtool-cross/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sys roots-components/armv7ahf-neon/lib32-dbus/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/armv7ahf-neon/lib32-gobject-introspection/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/armv7ahf-neon/lib32-base-passwd/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/ tmp/sysroots-components/armv7ahf-neon/lib32-python3/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/armv7ahf-neon/lib32-systemd/fixmepath | xargs sed -i -e 's:FIXMESTAGINGDIRTARGET:/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-avahi/0.6.32-r0/lib32-recipe-sysroot:g; s:FIXMESTAGINGDIRHOST:/home/wrlbuil d/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-avahi/0.6.32-r0/recipe-sysroot-native:g' -e 's:FIXME_COMPONENTS_DIR:/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components:g' -e 's:FIXME_HOSTTOOLS_DIR:/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/hosttools:g' -e 's:FIXME_PKG DATA_DIR:/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/pkgdata/bcm-587xx:g' NOTE: sed -e 's:^[^/]*/:/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-avahi/0.6.32-r0/recipe-sysroot-native/:g' /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/python3-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-compo nents/x86_64/libtool-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/gobject-introspection-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/gettext-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64 /intltool-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/gnu-config-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/autoconf-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/lib32-gcc-cross-arm/ fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/automake-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/pkgconfig-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/bison-native/fixmepath /home/wrlbuild/ builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/gtk-doc-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/glib-2.0-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/elfutils-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-f c7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/gmp-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/openssl-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/libsdl-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrli nux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/libpcre-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/ncurses-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/libx11-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_worl d_10/tmp/sysroots-components/x86_64/libpng-native/fixmepath /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components/x86_64/perl-native/fixmepath | xargs sed -i -e 's:FIXMESTAGINGDIRTARGET:/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-avahi/0.6.32-r0/lib32-recipe-sysroot:g; s:FIXMESTAGINGDIRHOST:/home/wrlbuild/builds/bu ilds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-avahi/0.6.32-r0/recipe-sysroot-native:g' -e 's:FIXME_COMPONENTS_DIR:/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/sysroots-components:g' -e 's:FIXME_HOSTTOOLS_DIR:/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/hosttools:g' -e 's:FIXME_PKGDATA_DIR:/h ome/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/pkgdata/bcm-587xx:g' Can't find 'useradd' in $PATH. ERROR: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:extend_recipe_sysroot(d) 0003: File: '/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/layers/oe-core/meta/classes/staging.bbclass', lineno: 521, function: extend_recipe_sysroot 0517: else: 0518: staging_processfixme(fixme[f], multilibs[f].getVar("RECIPE_SYSROOT"), recipesysroot, recipesysrootnative, d) 0519: 0520: for p in postinsts: *** 0521: subprocess.check_output(p, shell=True) 0522: 0523: for dep in manifests: 0524: c = setscenedeps[dep][0] 0525: os.symlink(manifests[dep], depdir + "/" + c + ".complete") File: '/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/bin/buildtools.buildtools-standalone-20161208/sysroots/x86_64-wrlinuxsdk-linux/usr/lib/python3.5/subprocess.py', lineno: 626, function: check_output 0622: # empty string. That is maintained here for backwards compatibility. 0623: kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b'' 0624: 0625: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, *** 0626: **kwargs).stdout 0627: 0628: 0629:class CompletedProcess(object): 0630: """A process that has finished running. File: '/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/bin/buildtools.buildtools-standalone-20161208/sysroots/x86_64-wrlinuxsdk-linux/usr/lib/python3.5/subprocess.py', lineno: 708, function: run 0704: raise 0705: retcode = process.poll() 0706: if check and retcode: 0707: raise CalledProcessError(retcode, process.args, *** 0708: output=stdout, stderr=stderr) 0709: return CompletedProcess(process.args, retcode, stdout, stderr) 0710: 0711: 0712:def list2cmdline(seq): Exception: subprocess.CalledProcessError: Command '/home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-avahi/0.6.32-r0/lib32-recipe-sysroot/usr/bin/postinst-lib32-base-passwd' returned non-zero exit status 1 Subprocess output: /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-systemd/1_232-r0/recipe-sysroot-native/usr/sbin/useradd Running groupadd commands... NOTE: lib32-systemd: Performing groupadd with [--root /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-avahi/0.6.32-r0/lib32-recipe-sysroot -r lock] NOTE: lib32-systemd: Performing groupadd with [--root /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-avahi/0.6.32-r0/lib32-recipe-sysroot -r systemd-journal] Running useradd commands... NOTE: lib32-systemd: Performing useradd with [--root /home/wrlbuild/builds/builds-e2b2e75e-fc7b-4db7-9626-7d90f026ce2e/wrlinux-std-sato_bcm-587xx_world_10/tmp/work/armv7ahf-neon-wrsmllib32-linux-gnueabi/lib32-avahi/0.6.32-r0/lib32-recipe-sysroot --system -d / -M --shell /bin/nologin systemd-timesync] ERROR: lib32-systemd: useradd command did not succeed.
Most of the times that this happens, the error is around: ERROR: lib32-systemd: useradd command did not succeed.
This is from a build with oe-core, meta-openembedded and ohter layers, I can try to reproduce it with just oe-core if needed.
This is most likely not relevant to this bug, but I believe adduser can be called from various tasks in random order, as these task can run in parallel. This is based on the contents of /etc/password. Sometimes (core-image-sato-sdk) I see: distcc:x:993:65534::/dev/null:/bin/sh pulse:x:994:1001::/var/run/pulse:/bin/false other times I see: pulse:x:993:1001::/var/run/pulse:/bin/false distcc:x:994:65534::/dev/null:/bin/sh
This build failure signature happens for both useradd and groupadd commands. It happens most of the time for the lib32-systemd package but also once for lib32-dbus and once for lib32-mariadb. It only happens for 64 bit world multilib builds but both x86-64 and arm64 targets are both seen as you'd expect. There have been 40 such build failures since they appear to have started around April 19, 11:20 PM ET, 2017. WR had rebased our branch to a newer version of oe-core on: 2017-04-17 and after that these rare build failures started. The previous rebase was: 2017-04-12 so it's possible that with only 4 build failures happening in April on: 4-19, 4-20, 4-24, 4-28, the problem MAY have been introduced before 4-17. There were 7 build failures in May, 19 in June, and 10 in July so far. The increasing frequency is due to the number of builds/day being increased by a factor of 5 on June 26. There were roughly 2 build failures a day from June 27 to July 5; this branch gets built about 750 times, with roughly 100 of those builds being world builds. All to say that the problem is relatively rare in a normal world build. This rebase added 135 new commits: $ git log --oneline a6152dd..ba2884f | wc -l 135 I'll continue a bit tomorrow but then I'm on vacation for a week.
This error stopped happening around July 18th. Since it's a rare build failure, I don't know the specific commit that that fixed it.