14636 2021-11-18 15:47:20 +0000 Frequent pseudo aborts with tcp-wrappers, complains about hardlinks in packages-split 2021-11-22 15:03:00 +0000 1 1 6 Yocto Project Subprojects Pseudo pseudo unspecified x86 Multiple RESOLVED INVALID Medium+ normal 3.1.14 1 ernstp steve randy.macleod richard.purdie yp.pseudo.watcher yp.watcher --- --- No (bug/feature does not impact docs) oldest_to_newest 92035 0 ernstp 2021-11-18 15:47:20 +0000 I seem to be getting semi-repeatable pseudo abort errors on tcp-wrappers right now. I'm getting this on other packages sometimes also. I'm using Dunfell. It seems to happen both with Dunfell 3.1.7 and the almost 3.1.12 from today, 0839888394 build-appliance-image: Update to dunfell head revision It's strange that it's complaining that package/usr/src/debug/tcp-wrappers/7.6-r10/tcp_wrappers_7.6/diag.c and packages-split/tcp-wrappers-src/usr/src/debug/tcp-wrappers/7.6-r10/tcp_wrappers_7.6/diag.c have the same inode... They should! They are hardlinks. | DEBUG: Removing manifest: /mnt/scratch/ernst.sjostrand/cu1/work/cortexa7t2hf-neon-fslc-linux-gnueabi/tcp-wrappers/7.6-r10/recipe-sysroot-native/usr/ | NOTE: Installed into sysroot: ['dwarfsrcfiles-native', 'rpm-native', 'dbus-native', 'openssl-native', 'file-native', 'python3-native', 'popt-native', 'libarchive-native', 'db-native', 'elfutils-native', 'bzip2-native', 'perl-native', 'pkgconfig-native', 'expat-native', 'sqlite3-native', 'libnsl2-native', 'libffi-native', 'gdbm-native', 'util-linux-native', 'libtirpc-native', 'readline-native', 'lzo-native', 'e2fsprogs-native', 'libpcre2-native', 'ncurses-native', 'libcap-ng-native'] | NOTE: Skipping as already exists in sysroot: ['glibc', 'pseudo-native', 'libnsl2', 'quilt-native', 'patch-native', 'gcc-cross-arm', 'gcc-runtime', 'linux-libc-headers', 'libtirpc', 'automake-native', 'autoconf-native', 'attr-native', 'gnu-config-native', 'libtool-native', 'xz-native', 'gettext-minimal-native', 'libmpc-native', 'flex-native', 'binutils-cross-arm', 'mpfr-native', 'texinfo-dummy-native', 'gmp-native', 'zlib-native', 'libgcc', 'm4-native'] | DEBUG: sed -e 's:^[^/]*/:/mnt/scratch/ernst.sjostrand/cu1/work/cortexa7t2hf-neon-fslc-linux-gnueabi/tcp-wrappers/7.6-r10/recipe-sysroot-native/:g' /mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/rpm-native/fixmepath /mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/dbus-native/fixmepath /mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/openssl-native/fixmepath /mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/python3-native/fixmepath /mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/elfutils-native/fixmepath /mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/perl-native/fixmepath /mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/pkgconfig-native/fixmepath /mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/e2fsprogs-native/fixmepath /mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/libpcre2-native/fixmepath /mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/ncurses-native/fixmepath | xargs sed -i -e 's:FIXMESTAGINGDIRTARGET:/mnt/scratch/ernst.sjostrand/cu1/work/cortexa7t2hf-neon-fslc-linux-gnueabi/tcp-wrappers/7.6-r10/recipe-sysroot:g; s:FIXMESTAGINGDIRHOST:/mnt/scratch/ernst.sjostrand/cu1/work/cortexa7t2hf-neon-fslc-linux-gnueabi/tcp-wrappers/7.6-r10/recipe-sysroot-native:g' -e 's:FIXME_PSEUDO_SYSROOT:/mnt/scratch/ernst.sjostrand/cu1/sysroots-components/x86_64/pseudo-native:g' -e 's:FIXME_HOSTTOOLS_DIR:/mnt/scratch/ernst.sjostrand/cu1/hosttools:g' -e 's:FIXME_PKGDATA_DIR:/mnt/scratch/ernst.sjostrand/cu1/pkgdata/cu2:g' -e 's:FIXME_PSEUDO_LOCALSTATEDIR:/mnt/scratch/ernst.sjostrand/cu1/work/cortexa7t2hf-neon-fslc-linux-gnueabi/tcp-wrappers/7.6-r10/pseudo/:g' -e 's:FIXME_LOGFIFO:/mnt/scratch/ernst.sjostrand/cu1/work/cortexa7t2hf-neon-fslc-linux-gnueabi/tcp-wrappers/7.6-r10/temp/fifo.10567:g' | DEBUG: Python function extend_recipe_sysroot finished | abort()ing pseudo client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this. | Check logfile: /mnt/scratch/ernst.sjostrand/cu1/work/cortexa7t2hf-neon-fslc-linux-gnueabi/tcp-wrappers/7.6-r10/pseudo//pseudo.log | ERROR: Command '['rm', '-rf', '/mnt/scratch/ernst.sjostrand/cu1/work/cortexa7t2hf-neon-fslc-linux-gnueabi/tcp-wrappers/7.6-r10/packages-split']' died with <Signals.SIGABRT: 6>. ERROR: Task (/home/ernst.sjostrand/cu1/meta/poky/meta/recipes-extended/tcp-wrappers/tcp-wrappers_7.6.bb:do_package) failed with exit code '1' NOTE: Tasks Summary: Attempted 3242 tasks of which 1516 didn't need to be rerun and 1 failed. Summary: 1 task failed: /home/ernst.sjostrand/cu1/meta/poky/meta/recipes-extended/tcp-wrappers/tcp-wrappers_7.6.bb:do_package Summary: There were 3 WARNING messages shown. Summary: There was 1 ERROR message shown, returning a non-zero exit code. Pseduo.log: path mismatch [2 links]: ino 6176280 db '/mnt/scratch/ernst.sjostrand/cu1/work/cortexa7t2hf-neon-fslc-linux-gnueabi/tcp-wrappers/7.6-r10/package/usr/src/debug/tcp-wrappers/7.6-r10/tcp_wrappers_7.6/diag.c' req '/mnt/scratch/ernst.sjostrand/cu1/work/cortexa7t2hf-neon-fslc-linux-gnueabi/tcp-wrappers/7.6-r10/packages-split/tcp-wrappers-src/usr/src/debug/tcp-wrappers/7.6-r10/tcp_wrappers_7.6/diag.c'. Build Configuration: BB_VERSION = "1.46.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "arm-fslc-linux-gnueabi" MACHINE = "cu2" DISTRO = "verilix" DISTRO_VERSION = "master" TUNE_FEATURES = "arm vfp cortexa7 neon thumb callconvention-hard" TARGET_FPU = "hard" meta meta-poky = "HEAD:0839888394a6e42e96f9f0d201376eb38bc79b24" meta-oe meta-multimedia meta-python meta-networking = "HEAD:c38d2a74f762a792046f3d3c377827b08aade513" meta-freescale = "HEAD:2da53ce8330d4bb0f7cb7a2ae186b7475ba02d52" meta-selinux = "HEAD:7af62c91d7d00a260cf28e7908955539304d100d" find * -name diag.c | xargs stat File: package/usr/src/debug/tcp-wrappers/7.6-r10/tcp_wrappers_7.6/diag.c Size: 1466 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 6176280 Links: 2 Access: (0644/-rw-r--r--) Uid: (12527211/ernst.sjostrand@verisure.com) Gid: (12527211/ernst.sjostrand@verisure.com) Access: 2021-11-18 15:05:21.358980637 +0000 Modify: 1994-12-28 16:42:20.000000000 +0000 Change: 2021-11-18 15:11:06.340869096 +0000 Birth: - File: packages-split/tcp-wrappers-src/usr/src/debug/tcp-wrappers/7.6-r10/tcp_wrappers_7.6/diag.c Size: 1466 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 6176280 Links: 2 Access: (0644/-rw-r--r--) Uid: (12527211/ernst.sjostrand@verisure.com) Gid: (12527211/ernst.sjostrand@verisure.com) Access: 2021-11-18 15:05:21.358980637 +0000 Modify: 1994-12-28 16:42:20.000000000 +0000 Change: 2021-11-18 15:11:06.340869096 +0000 Birth: - File: tcp_wrappers_7.6/diag.c Size: 1466 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 8524642 Links: 1 Access: (0444/-r--r--r--) Uid: (12527211/ernst.sjostrand@verisure.com) Gid: (12527211/ernst.sjostrand@verisure.com) Access: 2021-11-18 15:17:35.489209195 +0000 Modify: 1994-12-28 16:42:20.000000000 +0000 Change: 2021-11-18 15:11:06.384871389 +0000 Birth: - Host machine: 5.4.0-1063-azure /dev/sdb1 on /mnt type ext4 (rw,relatime,x-systemd.requires=cloud-init.service) 92046 1 steve 2021-11-19 15:20:30 +0000 I'd like to find some way to reproduce this issue, as I've never seen it in my own testing. I do regular builds with layer sets quite similar to what you are using, though I mostly use the poky distro (or a distro that inherits poky). So your distro is one potential area to investigate. Also none of my testing uses an azure host machine, so that is another area of difference. Have you tried using other host machines to see if it might be azure related? 92054 2 ernstp 2021-11-22 09:39:50 +0000 Looks like you need to do two builds in a row without sstate caching you. So build, tweak something, build again. The error comes from: def exec_func(func, d, dirs = None): <...> cleandirs = flags.get('cleandirs') if flags else None if cleandirs: for cdir in d.expand(cleandirs).split(): + bb.warn("My extra print: running cleandir %s" % cdir) bb.utils.remove(cdir, True) bb.utils.mkdirhier(cdir) So this "rm -rf" is running under Pseudo... seems a bit meaningless? Can it be disabled? I actually don't think this is Dunfell specific, it's probably the same on all up to date Yocto.. ? I just managed to trigger this somehow. There's a been a couple of mentions of similar problems: https://lists.yoctoproject.org/g/yocto/topic/78528061?p=Created,,,20,2,0,0::recentpostdate%2Fsticky,,,20,2,40,78528061 My distro is based on Poky. Yeah I also suspected Azure a bit. Looks like a normal ext4 filesystem etc, couldn't find anything very special about it... No Docker involved. Decently modern kernel. 92055 3 4844 ernstp 2021-11-22 09:52:04 +0000 Created attachment 4844 Patch to disable pseudo during bb.utils.remove() 92056 4 ernstp 2021-11-22 09:52:30 +0000 This patch solve my issue. 92057 5 ernstp 2021-11-22 13:49:42 +0000 Ignore the patch. This seems to be caused by adding --enable-xattrdb to pseudo. The pseudo database is missing a lot of files. 92058 6 richard.purdie 2021-11-22 15:03:00 +0000 xattrdb is an experimental feature in pseudo and known not to work well, we'd not recommend it. 4844 2021-11-22 09:52:04 +0000 2021-11-22 09:52:04 +0000 Patch to disable pseudo during bb.utils.remove() 0001-bitbake-Disable-pseudo-during-remove.patch text/plain 997 ernstp RnJvbSBhZTliYWFiYzc0YjgwYmYwNjA1ZmFhYWJkOGQxZGZkMmY2YmU3YTZhIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiA9P1VURi04P3E/RXJuc3Q9MjBTaj1DMz1CNnN0cmFuZD89IDxl cm5zdC5zam9zdHJhbmRAdmVyaXN1cmUuY29tPgpEYXRlOiBNb24sIDIyIE5vdiAyMDIxIDA5OjQx OjA2ICswMDAwClN1YmplY3Q6IFtQQVRDSF0gYml0YmFrZTogRGlzYWJsZSBwc2V1ZG8gZHVyaW5n IHJlbW92ZQoKW1lPQ1RPICMxNDYzNl0KCkNoYW5nZS1JZDogSTYxMjEwYjIxM2E1MDg2Y2M0OWJl MTQ1ZGU2YzRmMmNmOTk1YmZjYjEKLS0tCiBiaXRiYWtlL2xpYi9iYi91dGlscy5weSB8IDMgKyst CiAxIGZpbGUgY2hhbmdlZCwgMiBpbnNlcnRpb25zKCspLCAxIGRlbGV0aW9uKC0pCgpkaWZmIC0t Z2l0IGEvYml0YmFrZS9saWIvYmIvdXRpbHMucHkgYi9iaXRiYWtlL2xpYi9iYi91dGlscy5weQpp bmRleCA1ZjU3NjdjMWRhLi4yOGZiMmZkMzJkIDEwMDY0NAotLS0gYS9iaXRiYWtlL2xpYi9iYi91 dGlscy5weQorKysgYi9iaXRiYWtlL2xpYi9iYi91dGlscy5weQpAQCAtNjk3LDcgKzY5Nyw4IEBA IGRlZiByZW1vdmUocGF0aCwgcmVjdXJzZT1GYWxzZSwgaW9uaWNlPUZhbHNlKToKICAgICAgICAg Y21kID0gW10KICAgICAgICAgaWYgaW9uaWNlOgogICAgICAgICAgICAgY21kID0gWydpb25pY2Un LCAnLWMnLCAnMyddCi0gICAgICAgIHN1YnByb2Nlc3MuY2hlY2tfY2FsbChjbWQgKyBbJ3JtJywg Jy1yZiddICsgZ2xvYi5nbG9iKHBhdGgpKQorICAgICAgICAjIE5vIG5lZWQgdG8gaGF2ZSBwc2V1 ZG8gY2hlY2sgZHVyaW5nIHJlbW92ZQorICAgICAgICBzdWJwcm9jZXNzLmNoZWNrX2NhbGwoY21k ICsgWydybScsICctcmYnXSArIGdsb2IuZ2xvYihwYXRoKSwgZW52PXsiUFNFVURPX0RJU0FCTEVE IjogIjEifSkKICAgICAgICAgcmV0dXJuCiAgICAgZm9yIG5hbWUgaW4gZ2xvYi5nbG9iKHBhdGgp OgogICAgICAgICB0cnk6Ci0tIAoyLjM0LjAKCg==