Bug 15190

Summary: Race between fitimage and runtime_test.TestImage.test_testimage_install
Product: [QA/Testing] Functional (self) Testing Reporter: Richard Purdie <richard.purdie>
Component: oe-selftestAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Undecided    
Version: unspecified   
Target Milestone: ---   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2023-08-14 15:25:54 UTC 
We occasionally see issues with runtime_test.TestImage.test_testimage_install where socat isn't found.

https://autobuilder.yocto.io/pub/failed-builds-data/socat/log.do_testimage.3424338

is the test image log and it shows it installing an arm socat package into the target and trying to run it getting an exec error, unsurprisingly as it is an x86_64 image.

The package seems to be found via a symlink out the recipe's package data into the main tmp/deploy/rpm directory:

okybuild@debian11-ty-3:~/yocto-worker/oe-selftest-debian/build/build-st-2844859/tmp/work/qemux86_64-poky-linux/core-image-full-cmdline/1.0/oe-rootfs-repo$ ls -la
total 504
drwxr-xr-x  6 pokybuild pokybuild   4096 Aug 14 05:02 .
drwxr-xr-x 20 pokybuild pokybuild   4096 Aug 14 05:02 ..
drwxr-xr-x  2 pokybuild pokybuild   4096 Aug 14 04:55 all
drwxr-xr-x  2 pokybuild pokybuild 393216 Aug 14 04:55 core2-64
drwxr-xr-x  2 pokybuild pokybuild 102400 Aug 14 04:55 qemux86_64
drwxr-xr-x  2 pokybuild pokybuild   4096 Aug 14 05:02 repodata
lrwxrwxrwx  1 pokybuild pokybuild     85 Aug 14 05:02 rpm -> /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2844859/tmp/deploy/rpm
pokybuild@debian11-ty-3:~/yocto-worker/oe-selftest-debian/build/build-st-2844859/tmp/work/qemux86_64-poky-linux/core-image-full-cmdline/1.0/oe-rootfs-repo$ ls -la /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2844859/tmp/deploy/rpm
total 944
drwxr-xr-x 8 pokybuild pokybuild   4096 Aug 14 05:03 .
drwxr-xr-x 8 pokybuild pokybuild   4096 Aug 14 04:54 ..
drwxr-xr-x 2 pokybuild pokybuild  57344 Aug 14 04:38 beaglebone_yocto
drwxr-xr-x 2 pokybuild pokybuild 401408 Aug 14 05:03 core2_64
drwxr-xr-x 2 pokybuild pokybuild 401408 Aug 11 09:07 cortexa8hf_neon
drwxr-xr-x 2 pokybuild pokybuild   4096 May  5 14:36 noarch
-rw-r--r-- 2 pokybuild pokybuild   1700 Mar 26 04:24 PACKAGEFEED-GPG-KEY-poky-mickledore
drwxr-xr-x 2 pokybuild pokybuild   4096 Aug 14 04:43 qemuarm
drwxr-xr-x 2 pokybuild pokybuild  77824 Aug 11 08:29 qemux86_64
pokybuild@debian11-ty-3:~/yocto-worker/oe-selftest-debian/build/build-st-2844859/tmp/work/qemux86_64-poky-linux/core-image-full-cmdline/1.0/oe-rootfs-repo$ cd /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2844859/tmp/deploy/rpm
pokybuild@debian11-ty-3:~/yocto-worker/oe-selftest-debian/build/build-st-2844859/tmp/deploy/rpm$ find -name socat*
./cortexa8hf_neon/socat-dev-1.7.4.4-r0.cortexa8hf_neon.rpm
./cortexa8hf_neon/socat-1.7.4.4-r0.cortexa8hf_neon.rpm
./cortexa8hf_neon/socat-dbg-1.7.4.4-r0.cortexa8hf_neon.rpm
./cortexa8hf_neon/socat-src-1.7.4.4-r0.cortexa8hf_neon.rpm
./cortexa8hf_neon/socat-doc-1.7.4.4-r0.cortexa8hf_neon.rpm
./core2_64/socat-dev-1.7.4.4-r0.core2_64.rpm
./core2_64/socat-src-1.7.4.4-r0.core2_64.rpm
./core2_64/socat-doc-1.7.4.4-r0.core2_64.rpm
./core2_64/socat-1.7.4.4-r0.core2_64.rpm
./core2_64/socat-dbg-1.7.4.4-r0.core2_64.rpm
pokybuild@debian11-ty-3:~/yocto-worker/oe-selftest-debian/build/build-st-2844859/tmp/work/qemux86_64-poky-linux/core-image-full-cmdline/1.0/oe-rootfs-repo$ find -name socat*
./core2-64/socat-dev_1.7.4.4-r0_amd64.deb
./core2-64/socat-src_1.7.4.4-r0_amd64.deb
./core2-64/socat-dbg_1.7.4.4-r0_amd64.deb
./core2-64/socat-doc_1.7.4.4-r0_amd64.deb
./core2-64/socat_1.7.4.4-r0_amd64.deb
pokybuild@debian11-ty-3:~/yocto-worker/oe-selftest-debian/build/build-st-2844859/tmp/work/qemux86_64-poky-linux/core-image-full-cmdline/1.0/oe-rootfs-repo$ 

I'd hypothesise that since fitimage contains the only beaglebone-yocto build and this matches the beaglebone package architecture, the command to reproduce would be:

oe-selftest -j 1 -r fitimage runtime_test.TestImage.test_testimage_install,
Comment 1 Richard Purdie 2023-08-14 15:33:19 UTC 
I confirmed the reproducer did reproduce the issue on the worker in question. Only remaining potential issue with reproducing may be timestamp order on the rpms as I don't know how it picks one when it returns two possibilities from a repo query but we should be able to debug and fix it from here.
Comment 2 Richard Purdie 2023-08-15 13:19:43 UTC 
Stripped down to two tests, the reproducer is:

oe-selftest -j 1 -r fitimage.FitImageTests.test_initramfs_bundle runtime_test.TestImage.test_testimage_install
Comment 3 Richard Purdie 2023-08-15 13:49:58 UTC 
I suspect it is testexport triggering the BB_LIMITED_DEPS code in lib/oe/package_manager/__init__.py which generates the symlink