Bug 14428 - eSDK: native pseudo broken
Summary: eSDK: native pseudo broken
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: High major
Target Milestone: 3.4 M2
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2021-06-11 14:00 UTC by Andrej Valek
Modified: 2025-02-14 16:48 UTC (History)
9 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andrej Valek 2021-06-11 14:00:30 UTC
Hello,

I've observed an bug during eSDK installation. Looks like, that eSDK archive can be installed only on the same distribution as it was built. It works on Debian-10:Debian-10, but it filed on Debian-10:Ubuntu-20 even if uninative is enabled.

> Preparing SDK for system-image-sdk, meta-extsdk-toolchain:do_populate_sysroot...
> NOTE: Resolving any missing task queue dependencies
> Sstate summary: Wanted 2280 Found 2016 Missed 264 Current 0 (88% match, 0% complete)
> NOTE: Executing Tasks
> NOTE: Running setscene task 3 of 2282 (.../dropbear/dropbear_2020.80.bb:do_populate_sysroot_setscene)
> NOTE: Running setscene task 4 of 2282 (.../firewalld/firewalld_0.7.5.bb:do_populate_sysroot_setscene)
> NOTE: Running setscene task 5 of 2282 (.../networkmanager/networkmanager_1.28.0.bb:do_populate_sysroot_setscene)
> ...
> NOTE: recipe dropbear-2020.80-r0: task do_populate_sysroot_setscene: Started
> NOTE: recipe firewalld-0.7.5-r0: task do_populate_sysroot_setscene: Started
> NOTE: recipe networkmanager-1.28.0-r0: task do_populate_sysroot_setscene: Started
> ...
>  '.../layers/poky/meta/recipes-support/popt/popt_1.16.bb:do_package',
>  '.../layers/poky/meta/recipes-support/rng-tools/rng-tools_6.9.bb:do_package',
>  'virtual:native:.../layers/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_deploy_source_date_epoch',
>  'virtual:native:.../layers/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot'}
> ERROR: Task (.../layers/poky/meta/recipes-core/systemd/systemd_244.5.bb:do_write_config) failed with exit code 'setscene whitelist'
> NOTE: recipe systemd-systemctl-native-1.0-r6: task do_deploy_source_date_epoch_setscene: Succeeded
> ...
> NOTE: Tasks Summary: Attempted 39 tasks of which 13 didn't need to be rerun and 1 failed.
> NOTE: Build completion summary:
> NOTE:   do_populate_sysroot: 100.0% sstate reuse(361 setscene, 0 scratch)
> NOTE:   do_deploy_source_date_epoch: 100.0% sstate reuse(361 setscene, 0 scratch)
> NOTE:   do_package_qa: 100.0% sstate reuse(255 setscene, 0 scratch)
> NOTE:   do_packagedata: 100.0% sstate reuse(255 setscene, 0 scratch)
> NOTE:   do_package_write_ipk: 100.0% sstate reuse(255 setscene, 0 scratch)
> NOTE:   do_populate_lic: 100.0% sstate reuse(419 setscene, 0 scratch)

There is no hashes difference in locked-sigs.inc files, but native string for pseudo component is different.

SIGGEN_LOCKEDSIGS_t-x86-64-debian-10 vs SIGGEN_LOCKEDSIGS_t-x86-64-ubuntu-20.04

So there has to be something wrong regarding to pseudo.
Comment 1 Richard Purdie 2021-06-11 14:02:53 UTC
The problem is that http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3ecf5d9692fec97b37af6a4e6c747a4e2c2ca292 made pseudo host specific, so the eSDK also became host specific. Unintentionally, this means eSDK can't be used on other hosts. We probably need to find a way to make pseudo work on all hosts to remove the above change but I couldn't see any straightforward way to do that when I last looked.
Comment 2 Robert Berger 2021-06-12 08:44:41 UTC
My workaround for this is to build the eSDK in a poky build container[1] and to install the eSDK in a similar SDK container[2].

[1] https://github.com/RobertBerger/poky-container/tree/2021-06-09-master-local-gcc-9-gui-icecc-ub18

[2] https://github.com/RobertBerger/sdk-container/tree/2021-05-31-master-local-gcc-9-ub18
Comment 3 Mark Hatle 2021-06-13 17:15:44 UTC
Thinking out loud.

I guess the first question is, are there any Linux host systems without xattr support anymore?  If the answer is "no", then we can always enable it -- revert the patch and go on with life.  If there are other functions that are dynamic, we could make the same kind of call and just say "your host must have the following functions available."

(I suspect thought with Debian/Ubuntu, we may still need to support no xattr...)

Could we do something to make the xattr handling run-time specific?

For instance, split the xattr function(s) into a second .so.

So we end up with:

pseudo.so (LD_PRELOADed)
pseudo-xattr.so (just the xattr wrappers)

Switch the various 'PSEUDO_XATTRDB' ifdefs would go away and always be "on".  (This might be problematic as it requires xattr.h.)

pseudo.so when the ctor is called will then do a runtime check to see if xattr support is available.  (Is there such a way to do this and do this quickly, or maybe it only does it on the first use of something that MIGHT need xattr?). If the support is enabled, it would then dlopen the "new" pseudo-xattr.so plugin, it would have it's own init function to setup the xattr wrappers.

If the system did NOT have xattr support, it would then run in the same way as w/o the plugin.
Comment 4 Robert Berger 2021-06-13 18:51:33 UTC
Mark, it might be even more interesting.

I don't quite understand what you mean with:

>> (I suspect thought with Debian/Ubuntu, we may still need to support no xattr...)

What is the problem with/without xattr?

1) It seems to depend also on the Ubuntu version.

It looks like from Ubuntu 18.x it's supported:

https://packages.ubuntu.com/search?keywords=xattr

... and Ubuntu 16 is kind of - out of standard support:

https://wiki.ubuntu.com/Releases

$ cat /etc/issue
Ubuntu 20.04.1 LTS \n \l

$ sudo apt-cache search xattr | grep "xattr - tool"
xattr - tool for manipulating filesystem extended attributes

My "standard" build container is Ubuntu 18.x but I don't have xattr installed.
Will that help to fix the eSDK?

2) I thought for cases where tools are missing, or are not in the proper version we have we have buildtools-extended-nativesdk.
Comment 5 Robert Berger 2021-06-14 07:19:11 UTC
Neither my (working) sdk container nor my build container have xattr installed.

Both of them are derived from Ubuntu 18.

Once I installed the eSDK via a build container I can even utilize a couple of things from the host without using the container.

e.g. cross compile a cmake project with the eSDK
Comment 6 Mark Hatle 2021-07-01 15:24:03 UTC
I am _NOT_ working on this issue.  I commented as a possible solution.

What we need to understand, are the xattr functions now available on all supported Linux hosts?  If so, then we should just remove the logic that makes this support dynamic and always enable it.

Otherwise, I don't have enough understanding to know what the actual issue is here or why the code was done originally the way it was (other then the interfaces were not consistent between different distributions.)
Comment 7 Randy MacLeod 2021-07-01 15:34:40 UTC
As Mark said, he's not working on this so I've moved it to unassigned and added Mark and Saul to the CC list.
Comment 8 Robert Berger 2021-07-01 16:14:43 UTC
I guess it would be easy for me to install xattr and rebuild the eSDK, but I am not sure if this will fix it.

I believe I already tried and it did not make any difference for me.

If someone thinks that this might fix it I can give it a try and report back here what's happening.
Comment 9 Richard Purdie 2021-07-10 22:15:57 UTC
The big issue here was that the statx struct was only available on new libc versions so if pseudo was built on old libc systems, the function wasn't intercepted. I've added a copy of the struct into the code which resolves that issue.

The second worries were whether any of the other port code changed between systems we run on. I've added sanity tests for that but I think it is all old enough that it is present on anything we now build upon/care about.

Those two fixes combined should be enough to build pseudo-native generically again.
Comment 11 Randy MacLeod 2025-02-14 16:46:29 UTC
bulk change to add Saul's non-WR address.
Comment 12 Randy MacLeod 2025-02-14 16:48:27 UTC
Bulk change: Remove Saul's old WR address.