| Summary: | eSDK: python3.5.real: No such file or directory | ||
|---|---|---|---|
| Product: | [Yocto Project Subprojects] eSDK | Reporter: | Robert Berger <pokylinux> |
| Component: | eSDK | Assignee: | Richard Purdie <richard.purdie> |
| Status: | RESOLVED FIXED | QA Contact: | Francisco Pedraza <francisco.j.pedraza.gonzalez> |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | randy.macleod, richard.purdie |
| Version: | 3.1.2 | ||
| Target Milestone: | 3.4 M3 | ||
| Hardware: | All | ||
| OS: | Multiple | ||
| Whiteboard: | Backport to 3.1.x | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Yes (doc changes required) | |
|
Description
Robert Berger
2020-09-18 02:42:38 UTC
It's even more interesting, since those files are available in the SDK, but the exe tries to link against system libs /workdir/sysroots/x86_64-resysdk-linux/usr/bin/python3.8.real tries to find libpython3.8.so.1.0 => not found libcrypt.so.2 => not found so my first thought was to install them on my host linux (UB 18 container). libpython3.8.so.1.0 is actually available and can be installed libcrypt.so.2 not so much from standard UB 18 repositories they are available in: libpython3.8.so.1.0 => /workdir/sysroots/x86_64-resysdk-linux/usr/lib/libpython3.8.so.1.0 (0x00007f1bee9fa000) libcrypt.so.2 => /workdir/sysroots/x86_64-resysdk-linux/usr/lib/libcrypt.so.2 (0x00007f1beef0f000) --- sudo ln -sf /workdir/sysroots/x86_64-resysdk-linux/usr/lib/libpython3.8.so.1.0 /lib/x86_64-linux-gnu/libpython3.8.so.1.0 sudo ln -sf /workdir/sysroots/x86_64-resysdk-linux/usr/lib/libcrypt.so.2 /lib/x86_64-linux-gnu/libcrypt.so.2 did not help This: $ /workdir/sysroots/x86_64-resysdk-linux/usr/bin/python3.8.real bash: /workdir/sysroots/x86_64-resysdk-linux/usr/bin/python3.8.real: No such file or directory Implies that the SDK relocation code hasn't worked, or the uninative shim isn't present. Running strings on python3.8.real would show a path to the ld-linux.so.2 loader which might help us figure out which of those it is. Could you run that? Richard asked a question below. (In reply to comment #2) > This: > > $ /workdir/sysroots/x86_64-resysdk-linux/usr/bin/python3.8.real > bash: /workdir/sysroots/x86_64-resysdk-linux/usr/bin/python3.8.real: No such > file or directory > > Implies that the SDK relocation code hasn't worked, or the uninative shim > isn't present. Running strings on python3.8.real would show a path to the > ld-linux.so.2 loader which might help us figure out which of those it is. > Could you run that? https://pastebin.com/Mt41y1QU /opt/resy/3.1.2/sysroots/x86_64-resysdk-linux/lib/ld-linux-x86-64.so.2 is definitely wrong in the container where I install the SDK. Please note this error while installing: sdk@1507ee466e39:/projects/sdk-install/sdk$ ./resy-glibc-x86_64-core-image-minimal-qt5-armv7at2hf-vfp-multi-v7-ml-toolchain-ext-3.1.2.sh Resy (Reliable Embedded Systems Reference Distro) Extensible SDK installer version 3.1.2 ======================================================================================== Enter target directory for SDK (default: ~/resy_sdk): /sdk/resy_sdk_ext You are about to install the SDK to "/sdk/resy_sdk_ext". Proceed [Y/n]? Y Unable to access "/sdk/resy_sdk_ext", will not attempt to use sudo as as extensible SDK cannot be used as root. sdk@1507ee466e39:/projects/sdk-install/sdk$ cd /sdk bash: cd: /sdk: No such file or directory sdk@1507ee466e39:/projects/sdk-install/sdk$ ./resy-glibc-x86_64-core-image-minimal-qt5-armv7at2hf-vfp-multi-v7-ml-toolchain-ext-3.1.2.sh Resy (Reliable Embedded Systems Reference Distro) Extensible SDK installer version 3.1.2 ======================================================================================== Enter target directory for SDK (default: ~/resy_sdk): /projects/sdk/3.1.2/resy_sdk_ext You are about to install the SDK to "/projects/sdk/3.1.2/resy_sdk_ext". Proceed [Y/n]? Y Extracting SDK................................................................................................................................................................................................................................................................................................................................done Setting it up... Extracting buildtools... Preparing build system... Parsing recipes: 100% |#######################################################################################################################################| Time: 0:01:45 Initialising tasks: 100% |####################################################################################################################################| Time: 0:00:03 Checking sstate mirror object availability: 100% |############################################################################################################| Time: 0:00:00 Loading cache: 100% |#########################################################################################################################################| Time: 0:00:00 Parsing recipes: 100% |#######################################################################################################################################| Time: 0:00:00 Initialising tasks: 100% |####################################################################################################################################| Time: 0:00:00 done SDK environment now set up; additionally you may now run devtool to perform development tasks. Run devtool --help for further details. Fatal Python error: init_import_size: Failed to import the site module Python runtime state: initialized Traceback (most recent call last): File "/projects/sdk/3.1.2/resy_sdk_ext/sysroots/x86_64-resysdk-linux/usr/lib/python3.8/site.py", line 580, in <module> main() File "/projects/sdk/3.1.2/resy_sdk_ext/sysroots/x86_64-resysdk-linux/usr/lib/python3.8/site.py", line 567, in main known_paths = addsitepackages(known_paths) File "/projects/sdk/3.1.2/resy_sdk_ext/sysroots/x86_64-resysdk-linux/usr/lib/python3.8/site.py", line 348, in addsitepackages for sitedir in getsitepackages(prefixes): File "/projects/sdk/3.1.2/resy_sdk_ext/sysroots/x86_64-resysdk-linux/usr/lib/python3.8/site.py", line 338, in getsitepackages sitepackages.append(os.path.join(prefix, sys.lib, AttributeError: module 'sys' has no attribute 'lib' post-relocate command "/projects/sdk/3.1.2/resy_sdk_ext/sysroots/x86_64-resysdk-linux/post-relocate-setup.d/meson-setup.py /projects/sdk/3.1.2/resy_sdk_ext" failed with status 1 SDK has been successfully set up and is ready to be used. Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g. $ . /projects/sdk/3.1.2/resy_sdk_ext/environment-setup-armv7at2hf-vfp-resy-linux-gnueabi -------- In order to reach so far, I did those hacks: # hack to fix missing libs RUN apt-get -y install libpython3.8 libarchive13 xz-utils # hack to fix missing libs COPY lib/x86_64-linux-gnu/libcrypt.so.2.0.0 /lib/x86_64-linux-gnu/libcrypt.so.2.0.0 RUN ln -s /lib/x86_64-linux-gnu/libcrypt.so.2.0.0 /lib/x86_64-linux-gnu/libcrypt.so.2 --- I will try to come up with a smaller testcase, since this one involves qt5, which might do nasty things. FYI: If I install the "classic" SDK in the same container things look like this: sdk@09b24244c587:/workdir/build/multi-v7-ml-qt5/tmp/deploy/sdk$ ./resy-glibc-x86_64-core-image-minimal-qt5-armv7at2hf-vfp-multi-v7-ml-toolchain-3.1.2.sh Resy (Reliable Embedded Systems Reference Distro) SDK installer version 3.1.2 ============================================================================= Enter target directory for SDK (default: /opt/resy/3.1.2): /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk You are about to install the SDK to "/projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk". Proceed [Y/n]? Y Extracting SDK.............................................................................................................................................................................................................................................................................done Setting it up...done SDK has been successfully set up and is ready to be used. Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g. $ . /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk/environment-setup-armv7at2hf-vfp-resy-linux-gnueabi sdk@09b24244c587:/workdir/build/multi-v7-ml-qt5/tmp/deploy/sdk$ The relocation seems to work with the "classic" SDK: strings ./sysroots/x86_64-resysdk-linux/usr/bin/python3.8.real /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk/sysroots/x86_64-resysdk-linux/lib/ld-linux-x86-64.so.2 libpython3.8.so.1.0 I removed this: # hack to fix missing libs RUN apt-get -y install libpython3.8 libarchive13 xz-utils # hack to fix missing libs COPY lib/x86_64-linux-gnu/libcrypt.so.2.0.0 /lib/x86_64-linux-gnu/libcrypt.so.2.0.0 RUN ln -s /lib/x86_64-linux-gnu/libcrypt.so.2.0.0 /lib/x86_64-linux-gnu/libcrypt.so.2 --- "classic" sdk installs without my hack: sdk@8403dfb9ef3a:/workdir/build/multi-v7-ml-qt5/tmp/deploy/sdk$ ./resy-glibc-x86_64-core-image-minimal-qt5-armv7at2hf-vfp-multi-v7-ml-toolchain-3.1.2.sh Resy (Reliable Embedded Systems Reference Distro) SDK installer version 3.1.2 ============================================================================= Enter target directory for SDK (default: /opt/resy/3.1.2): /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk You are about to install the SDK to "/projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk". Proceed [Y/n]? Y Extracting SDK.............................................................................................................................................................................................................................................................................done Setting it up...done SDK has been successfully set up and is ready to be used. Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g. $ . /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk/environment-setup-armv7at2hf-vfp-resy-linux-gnueabi sdk@8403dfb9ef3a:/workdir/build/multi-v7-ml-qt5/tmp/deploy/sdk$ --- esdk is unhappy: sdk@8403dfb9ef3a:/workdir/build/multi-v7-ml-qt5/tmp/deploy/sdk$ ./resy-glibc-x86_64-core-image-minimal-qt5-armv7at2hf-vfp-multi-v7-ml-toolchain-ext-3.1.2.sh Resy (Reliable Embedded Systems Reference Distro) Extensible SDK installer version 3.1.2 ======================================================================================== Enter target directory for SDK (default: ~/resy_sdk): /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext You are about to install the SDK to "/projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext". Proceed [Y/n]? Y Extracting SDK................................................................................................................................................................................................................................................................................................................................done Setting it up... Extracting buildtools... Preparing build system... Parsing recipes: 100% |#######################################################################################################################################| Time: 0:01:47 Initialising tasks: 100% |####################################################################################################################################| Time: 0:00:03 Checking sstate mirror object availability: 100% |############################################################################################################| Time: 0:00:00 Loading cache: 100% |#########################################################################################################################################| Time: 0:00:00 Parsing recipes: 100% |#######################################################################################################################################| Time: 0:00:00 Initialising tasks: 100% |####################################################################################################################################| Time: 0:00:00 done SDK environment now set up; additionally you may now run devtool to perform development tasks. Run devtool --help for further details. /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext/sysroots/x86_64-resysdk-linux/usr/bin/python3: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory post-relocate command "/projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext/sysroots/x86_64-resysdk-linux/post-relocate-setup.d/meson-setup.py /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext" failed with status 127 SDK has been successfully set up and is ready to be used. Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g. $ . /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext/environment-setup-armv7at2hf-vfp-resy-linux-gnueabi sdk@8403dfb9ef3a:/workdir/build/multi-v7-ml-qt5/tmp/deploy/sdk$ My core-image-minimal-qt5 recipe looks like this: https://pastebin.com/Xjmvdsk5 common-img.inc should not do any harm, it looks like this: # DISTRO, DISTRO_VERSION are already there by default # PART_NUMBER EXTRA_PKGS not (yet) defined IMAGE_BUILDINFO_VARS_append = " DATETIME DISTRO_NAME IMAGE_BASENAME MACHINE TUNE_PKGARCH" IMAGE_BUILDINFO_VARS_append = " MACHINE_FEATURES DISTRO_FEATURES COMMON_FEATURES IMAGE_FEATURES" IMAGE_BUILDINFO_VARS_append = " TUNE_FEATURES TARGET_FPU" IMAGE_BUILDINFO_VARS_append = " APP_URI_PREFIX APP_URI_BRANCH" --- Will try now a core-image-minimal classic and ext sdk without "inherit populate_sdk_qt5" but I will use systemd. Like this we should be able to see if the "wrong relocation" comes from "inherit populate_sdk_qt5". Also we should see if meson-setup.py (from systemd?) is still unhappy. ---- "classic" SDK installed happily: sdk@444f0a306041:/workdir/build/multi-v7-ml-qt5/tmp/deploy/sdk$ ./resy-glibc-x86_64-core-image-minimal-armv7at2hf-vfp-multi-v7-ml-toolchain-3.1.2.sh Resy (Reliable Embedded Systems Reference Distro) SDK installer version 3.1.2 ============================================================================= Enter target directory for SDK (default: /opt/resy/3.1.2): /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk You are about to install the SDK to "/projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk". Proceed [Y/n]? Y Extracting SDK...........................................................................done Setting it up...done SDK has been successfully set up and is ready to be used. Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g. $ . /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk/environment-setup-armv7at2hf-vfp-resy-linux-gnueabi -------- sdk@444f0a306041:/workdir/build/multi-v7-ml-qt5/tmp/deploy/sdk$ ./resy-glibc-x86_64-core-image-minimal-armv7at2hf-vfp-multi-v7-ml-toolchain-ext-3.1.2.sh Resy (Reliable Embedded Systems Reference Distro) Extensible SDK installer version 3.1.2 ======================================================================================== Enter target directory for SDK (default: ~/resy_sdk): /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext You are about to install the SDK to "/projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext". Proceed [Y/n]? Y Extracting SDK...............................................................done Setting it up... Extracting buildtools... Preparing build system... Parsing recipes: 100% |#######################################################################################################################################| Time: 0:01:41 Initialising tasks: 100% |####################################################################################################################################| Time: 0:00:02 Checking sstate mirror object availability: 100% |############################################################################################################| Time: 0:00:00 Loading cache: 100% |#########################################################################################################################################| Time: 0:00:00 Parsing recipes: 100% |#######################################################################################################################################| Time: 0:00:00 Initialising tasks: 100% |####################################################################################################################################| Time: 0:00:00 done SDK has been successfully set up and is ready to be used. Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g. $ . /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext/environment-setup-armv7at2hf-vfp-resy-linux-gnueabi sdk@444f0a306041:/workdir/build/multi-v7-ml-qt5/tmp/deploy/sdk$ ---- $ strings /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext/buildtools/sysroots/x86_64-resysdk-linux/usr/bin/python3.8.real /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext/buildtools/sysroots/x86_64-resysdk-linux/lib/ld-linux-x86-64.so.2 libpython3.8.so.1.0 The eSDK looks good, so the problems comes most likely from "inherit populate_sdk_qt5". This seems to be correct: strings /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext/buildtools/sysroots/x86_64-resysdk-linux/usr/bin/python3.8.real /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext/buildtools/sysroots/x86_64-resysdk-linux/lib/ld-linux-x86-64.so.2 ... But this is wrong strings /projects/sdk/3.1.2/qt5-armv7at2hf-vfp/sdk_ext/sysroots/x86_64-resysdk-linux/usr/bin/python3.8.real /opt/resy/3.1.2/sysroots/x86_64-resysdk-linux/lib/ld-linux-x86-64.so.2 ... Thanks for the info Robert. I've moved this bug out of NEEDINFO and given it back to Paul. As far as I can tell meta-qt5 is to blame for this bug. I opened a bug report at meta-qt5[1] but as far as I tell the motivation of the community to work on meta-qt5 is close to zero since the Qt Company does not seem to value community contributions.[2] [1] https://github.com/meta-qt5/meta-qt5/issues/356 [2] https://github.com/meta-qt5/meta-qt5/issues/349 I"m not sure if this is still a problem but bumping the defect to 3.4-M1. I was finally able to look at this a bit. The issue isn't meta-qt5 but any eSDK that pulls in nativesdk-meson. You can reproduce on OE-Core alone with: TOOLCHAIN_HOST_TASK:append = " nativesdk-meson nativesdk-sdk-provides-dummy" bitbake core-image-minimal -c populate_sdk_ext bitbake core-image-minimal -c testsdkext The issue is the post-relocation python script inside the nativesdk-meson recipe which tries to use the python which was just installed. In the eSDK case this isn't setup yet. I haven't got further with debugging/fixing it but wanted to summarise the narrowed down cause and how to reproduce. Further info - eSDK clears out TOOLCHAIN_HOST_TASK and doesn't support installing nativesdk recipes into the eSDK, it doesn't really need them. The :append bypasses this and puts things into the eSDK which shouldn't be there. They aren't relocated properly and this triggers the bug. Where the bug is, is arguable. meta-qt5 adding things to TOOLCHAIN_HOST_TASK with blanket append which the bbclass can't override is the problem. I have a patch which rewrites the way TOOLCHAIN_HOST_TASK is handled for eSDK which should resolve this. |