I bisected it in opkg-utils as well. This is the reason for the regression

https://git.yoctoproject.org/opkg-utils/commit/?id=8d9953dd8d589e9b740307976cbe474e0ce292a0

Sadly, it seems the only significant change between 0.6.2 and 0.6.3 release as well. So either we can manually revert it in oe-core via a patch or maybe opkg-utils should be fixed and then we just from 0.6.2 to 0.6.4 when its fixed

ACK the revert.

Though it isn't clear to me what that opkg-build commit would be doing *wrongly*, assuming that it is getting sane values from OE. So I think the investigation here should focus on what args OE is setting, and validating that they make sense in context.

(In reply to Khem Raj from comment #0)
> Recently, ptests for cpio package started to fail because they are run as
> non-root ( ptest ) user and files in /usr/lib/cpio/ptest became non-writable
> for ptest:ptest uid/gid combo and as a result the test would fail. Same
> happens for quilt package ptests as well. The issue is that GID for ptest
> group comes out to be 999 when the cpio package is built and is same in
> cpio-ptest ipk. However, when image ( core-image-ptest-cpio ) is built the
> value for GID of ptest group that is in etc/group is computed to be 995 and
> there is 999 group present as well ( messagebus ) and all files under
> /usr/lib/cpio/ptest inherit the group messagegroup as a result. When ptests
> are run, then it fails because of wrong group permissions since the ptest
> script tries to write log files into /usr/lib/cpio/ptest and can not do it
> any more.
> 
> I have traced this to be a regression in opkg-utils upgrade
> https://git.yoctoproject.org/poky/commit/
> ?id=a856192be5dd78c621478ed29871191c580686c4
> 
> ```
> commit a856192be5dd78c621478ed29871191c580686c4
> Author: Wang Mingyu <wangmy@fujitsu.com>
> Date:   Tue Dec 19 17:34:57 2023 +0800
> 
>     opkg-utils: upgrade 0.6.2 -> 0.6.3
> 
>     Changelog:
>     =========
>     -Add acls and xattrs support
>     -whitespace and styling
>     -update markdown styling
>     -update opkg mailing list
>     -add SECURITY.md
>     -refresh document
> 
>     (From OE-Core rev: f6bee62670c3a3f9bbc11b104f64249ca32072c9)
> 
>     Signed-off-by: Wang Mingyu <wangmy@fujitsu.com>
>     Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
>     Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_0.6.2.bb
> b/meta/recipes-devtools/opkg-utils/opkg-utils_0.6.3.bb
> similarity index 97%
> rename from meta/recipes-devtools/opkg-utils/opkg-utils_0.6.2.bb
> rename to meta/recipes-devtools/opkg-utils/opkg-utils_0.6.3.bb
> index eb88b9b734c..b509227e1a6 100644
> --- a/meta/recipes-devtools/opkg-utils/opkg-utils_0.6.2.bb
> +++ b/meta/recipes-devtools/opkg-utils/opkg-utils_0.6.3.bb
> @@ -10,7 +10,7 @@ PROVIDES += "${@bb.utils.contains('PACKAGECONFIG',
> 'update-alternatives', 'virtu
>  SRC_URI =
> "git://git.yoctoproject.org/opkg-utils;protocol=https;branch=master \
>             file://0001-update-alternatives-correctly-match-priority.patch \
>             "
> -SRCREV = "67994e62dc598282830385da75ba9b1abbbda941"
> +SRCREV = "589880d01969eb9af1e66120e731d43193504718"
> 
>  S = "${WORKDIR}/git"
> ```
> 
> To reproduce this
> 
> set local.conf to have
> 
> PACKAGE_CLASSES = "package_ipk"
> DISTRO_FEATURES:append = " ptest"
> 
> $ bitbake core-image-ptest-cpio
> $ bitbake -ctestimage core-image-ptest-cpio
> 
> 
> we should revert this upgrade and report this issue to opkg upstream.

I have downloaded newest master 1bbdc674aef8abada28d130a23752c1d8a4620b9 and I'm unable to run

$ bitbake -ctestimage core-image-ptest-cpio

as I'm getting:

ERROR: Task do_testimage does not exist for target core-image-ptest-cpio (virtual:mcextend:cpio:/home/plobacz/workspace/poky/meta/recipes-core/images/core-image-ptest.bb:do_testimage). Close matches:                                                                                                      | ETA:  0:00:00
  do_image
ERROR: Command execution failed: 1

Could you please provide more details on how to run it?

maybe add

IMAGE_CLASSES += "testimage"

in conf/local.conf

OK I have my test result which looks like this:

plobacz@latitude ~/workspace/poky/build $ bitbake -ctestimage core-image-ptest-cpio
WARNING: Host distribution "gentoo-2.14" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Loading cache: 100% |########################################################################################################################################################################################################################################################################################| Time: 0:00:00
Loaded 1848 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "2.6.1"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "4.3+snapshot-1bbdc674aef8abada28d130a23752c1d8a4620b9"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "master:1bbdc674aef8abada28d130a23752c1d8a4620b9"

Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 381 (0% match, 100% complete)#######################################################################################################################################################################################                             | ETA:  0:00:00
Initialising tasks: 100% |###################################################################################################################################################################################################################################################################################| Time: 0:00:00
NOTE: Executing Tasks
QMP Available for connection at /home/plobacz/workspace/poky/build/tmp/.7_l5c9p2
QMP connected to QEMU at 01/05/24 10:09:56 and took 0.95 seconds
QMP released QEMU at 01/05/24 10:09:56 and took 0.07 seconds from connect
WARNING: core-image-ptest-cpio-1.0-r0 do_testimage: There were failing ptests.
Traceback (most recent call last):
  File "/home/plobacz/workspace/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/plobacz/workspace/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/plobacz/workspace/poky/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/plobacz/workspace/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail
    self.do_ptestrunner()
  File "/home/plobacz/workspace/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 120, in do_ptestrunner
    self.fail(failmsg)
AssertionError: 
ptests which had no test results:
['cpio']

Cannot run ptests without @expectedFailure as ptests are expected to fail
Output from runqemu:
runqemu - INFO - Received signal: 15
runqemu - INFO - Cleaning up
runqemu - INFO - Host uptime: 136905.33

tput: No value for $TERM and no -T specified

RESULTS:
RESULTS - parselogs.ParseLogsTest.test_get_context: PASSED (0.00s)
RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (2.57s)
RESULTS - ping.PingTest.test_ping: PASSED (0.01s)
RESULTS - ssh.SSHTest.test_ssh: PASSED (11.14s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s)
RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: EXPECTEDFAIL (0.53s)
SUMMARY:
core-image-ptest-cpio () - Ran 6 tests in 14.251s
core-image-ptest-cpio - OK - All required tests passed (successes=4, skipped=1, failures=0, errors=0)
NOTE: Tasks Summary: Attempted 1288 tasks of which 1286 didn't need to be rerun and all succeeded.

Summary: There were 2 WARNING messages.

good. Now you can revert the aforementioned patch in opkg-utils and try the same test again and it will pass this time. The problem I have already explained the reason being change in GID for ptest group in cpio-ptest ipk and final image rootfs. if you need more clatification feel free to ask here.

OK I have found out the issue and from my point of view it is not in opkg-build script but in yocto itself.

Generally the updated opkg-build script is using --numeric-owner parameter at line:

( cd $pkg_dir && tar $attributesargs $ogargs $tsortargs --numeric-owner --no-recursion $mtime_args -c $tarformat -T $tmp_dir/file_list | $compressor $compressorargs > $tmp_dir/data.tar.$cext )

which is causing the packages to differ between storing values meaning that in case without update data.tar.zst file stored in cpio-ptest_2.14-r0_core2-64.ipk looks like this:

drwxr-xr-x  root/root         0 2023-05-01 20:09 ./usr/
drwxr-xr-x  root/root         0 2023-05-01 20:09 ./usr/lib/
drwxr-xr-x  root/root         0 2023-05-01 20:09 ./usr/lib/cpio/
drwxrwxr-x  root/ptest        0 2023-05-01 20:09 ./usr/lib/cpio/ptest/
-rwxrwxr-x  root/ptest       33 2023-05-01 20:09 ./usr/lib/cpio/ptest/run-ptest
-rwxrwxr-x  root/ptest      443 2023-05-01 20:09 ./usr/lib/cpio/ptest/test.sh
drwxrwxr-x  root/ptest        0 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/
-rwxrwxr-x  root/ptest      461 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/atconfig
-rwxrwxr-x  root/ptest      256 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/atlocal
-rwxrwxr-x  root/ptest   114416 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/genfile
-rwxrwxr-x  root/ptest    84883 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/testsuite

and in case of patched version it looks like this:

drwxr-xr-x  0/0               0 2023-05-01 20:09 ./usr/
drwxr-xr-x  0/0               0 2023-05-01 20:09 ./usr/lib/
drwxr-xr-x  0/0               0 2023-05-01 20:09 ./usr/lib/cpio/
drwxrwxr-x  0/999             0 2023-05-01 20:09 ./usr/lib/cpio/ptest/
-rwxrwxr-x  0/999            33 2023-05-01 20:09 ./usr/lib/cpio/ptest/run-ptest
-rwxrwxr-x  0/999           443 2023-05-01 20:09 ./usr/lib/cpio/ptest/test.sh
drwxrwxr-x  0/999             0 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/
-rwxrwxr-x  0/999           461 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/atconfig
-rwxrwxr-x  0/999           256 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/atlocal
-rwxrwxr-x  0/999        114416 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/genfile
-rwxrwxr-x  0/999         84883 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/testsuite

meaning that it is storing some random rubbish used in sysroot-native for image of cpio-ptest build, not the corect one which is autogenerated for the proper rootfs. I would rather blame for this issue usage of non-static user ids instead of this script...

(In reply to Piotr Łobacz from comment #7)
> OK I have found out the issue and from my point of view it is not in
> opkg-build script but in yocto itself.
> 
> Generally the updated opkg-build script is using --numeric-owner parameter
> at line:
> 
> ( cd $pkg_dir && tar $attributesargs $ogargs $tsortargs --numeric-owner
> --no-recursion $mtime_args -c $tarformat -T $tmp_dir/file_list | $compressor
> $compressorargs > $tmp_dir/data.tar.$cext )
> 
> which is causing the packages to differ between storing values meaning that
> in case without update data.tar.zst file stored in
> cpio-ptest_2.14-r0_core2-64.ipk looks like this:
> 
> drwxr-xr-x  root/root         0 2023-05-01 20:09 ./usr/
> drwxr-xr-x  root/root         0 2023-05-01 20:09 ./usr/lib/
> drwxr-xr-x  root/root         0 2023-05-01 20:09 ./usr/lib/cpio/
> drwxrwxr-x  root/ptest        0 2023-05-01 20:09 ./usr/lib/cpio/ptest/
> -rwxrwxr-x  root/ptest       33 2023-05-01 20:09
> ./usr/lib/cpio/ptest/run-ptest
> -rwxrwxr-x  root/ptest      443 2023-05-01 20:09 ./usr/lib/cpio/ptest/test.sh
> drwxrwxr-x  root/ptest        0 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/
> -rwxrwxr-x  root/ptest      461 2023-05-01 20:09
> ./usr/lib/cpio/ptest/tests/atconfig
> -rwxrwxr-x  root/ptest      256 2023-05-01 20:09
> ./usr/lib/cpio/ptest/tests/atlocal
> -rwxrwxr-x  root/ptest   114416 2023-05-01 20:09
> ./usr/lib/cpio/ptest/tests/genfile
> -rwxrwxr-x  root/ptest    84883 2023-05-01 20:09
> ./usr/lib/cpio/ptest/tests/testsuite
> 
> and in case of patched version it looks like this:
> 
> drwxr-xr-x  0/0               0 2023-05-01 20:09 ./usr/
> drwxr-xr-x  0/0               0 2023-05-01 20:09 ./usr/lib/
> drwxr-xr-x  0/0               0 2023-05-01 20:09 ./usr/lib/cpio/
> drwxrwxr-x  0/999             0 2023-05-01 20:09 ./usr/lib/cpio/ptest/
> -rwxrwxr-x  0/999            33 2023-05-01 20:09
> ./usr/lib/cpio/ptest/run-ptest
> -rwxrwxr-x  0/999           443 2023-05-01 20:09 ./usr/lib/cpio/ptest/test.sh
> drwxrwxr-x  0/999             0 2023-05-01 20:09 ./usr/lib/cpio/ptest/tests/
> -rwxrwxr-x  0/999           461 2023-05-01 20:09
> ./usr/lib/cpio/ptest/tests/atconfig
> -rwxrwxr-x  0/999           256 2023-05-01 20:09
> ./usr/lib/cpio/ptest/tests/atlocal
> -rwxrwxr-x  0/999        114416 2023-05-01 20:09
> ./usr/lib/cpio/ptest/tests/genfile
> -rwxrwxr-x  0/999         84883 2023-05-01 20:09
> ./usr/lib/cpio/ptest/tests/testsuite
> 
> meaning that it is storing some random rubbish used in sysroot-native for
> image of cpio-ptest build, not the corect one which is autogenerated for the
> proper rootfs. I would rather blame for this issue usage of non-static user
> ids instead of this script...

Our builds do not require that the numeric ownership stays the same so that is incorrect and not what we support. The addition of that is a regression for us unfortunately.

In case of supporting acls/xattrs you won't be able to do that otherwise unfortunately...

OK I have just verified that ACLs will still work without this parameter in here. The more important are in different places.

Alex do you want my to apply a patch for that or will you do that on your own?

@piotr; Thanks for investigating. Since you have a reproducing setup, it would be best if you authored the patchset.

https://git.yoctoproject.org/poky/commit/?id=351737169a4be5ce2aab77ff30c89364fdcf5cf8