Bug 14866

Summary: testsdk logs missing information
Product: [QA/Testing] Runtime Testing Reporter: Richard Purdie <richard.purdie>
Component: testsdkAssignee: Deepesh Varatharajan <deepesh.varatharajan>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium+ CC: Harish.Sadineni, mathieu.dubois-briand, randy.macleod, sundeep.kokkonda, tgamblin, tim.orling, Yash.Shinde
Version: unspecified   
Target Milestone: 6.0 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2022-07-25 15:49:07 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/44/builds/5597/steps/26/logs/stdio is puzzling as it shows an error but no reason why.

Logging into the failed worker and looking at the do_testsdk log shows:

NOTE: ERROR
NOTE: ======================================================================
NOTE: ERROR: setUpClass (rust.RustCompileTest)
NOTE: ----------------------------------------------------------------------
NOTE: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/multilib/build/meta/lib/oeqa/core/case.py", line 39, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/pokybuild/yocto-worker/multilib/build/meta/lib/oeqa/sdk/cases/rust.py", line 20, in setUpClass
    shutil.copytree(os.path.join(self.tc.sdk_files_dir, "rust/hello"),
  File "/usr/lib/python3.9/shutil.py", line 565, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/usr/lib/python3.9/shutil.py", line 466, in _copytree
    os.makedirs(dst, exist_ok=dirs_exist_ok)
  File "/usr/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/pokybuild/yocto-worker/multilib/build/build/tmp/work/qemumips64-poky-linux-gnun32/core-image-minimal/1.0-r0/testimage-sdk/hello'

NOTE: ----------------------------------------------------------------------
NOTE: Ran 11 tests in 187.765s
NOTE: FAILED
NOTE:  (errors=1, skipped=1)

which makes it clear the copy rerunning in a muliblib build broke things. This should be in the main autobuilder log!
Comment 1 Randy MacLeod 2023-02-02 15:24:26 UTC
Is this still happening?
Comment 2 Randy MacLeod 2023-07-26 21:21:56 UTC
Bulk move of 64 bugs to 4.3 M3 after a quick review.
If a bug is actually fixed, please add a commit link and resolve it.
-- Randy for YP bug team.
Comment 3 Trevor Gamblin 2023-09-14 16:46:07 UTC
Moved to M4.
Comment 4 Randy MacLeod 2023-10-30 15:29:07 UTC
Bulk move to 5.0 M1. -- Randy
Comment 5 Tim Orling 2023-12-17 20:10:25 UTC
Local test of the rust.RustCompileTest.test_cargo_build (SDKMACHINE = 'x86_64', MACHINE = 'qemuarm64'):

Traceback (most recent call last):                                                                                                      
  File "/path/to/poky/meta/lib/oeqa/sdk/cases/rust.py", line 35, in test_cargo_build                      
    self._run('cd %s/hello; cargo build' % self.tc.sdk_dir)                                                                             
  File "/path/to/poky/meta/lib/oeqa/sdk/case.py", line 14, in _run                                        
    return subprocess.check_output(". %s > /dev/null; %s;" % \                                                                          
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output                                                                   
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,                                                                    
  File "/usr/lib/python3.10/subprocess.py", line 526, in run                                                                            
    raise CalledProcessError(retcode, process.args,                                                                                     
oeqa.utils.subprocesstweak.OETestCalledProcessError: Command '. /path/to/build-arm64/tmp/work/qemuarm64-poky-linux/core-image-full-cmdline/1.0/testimage-sdk/environment-setup-cortexa57-poky-linux > /dev/null; cd /path/to/build-arm64/tmp/work/qemuarm64-poky-linux/core-image-full-cmdline/1.0/testimage-sdk//hello; cargo build;' returned non-zero exit status 101                                                                                                                            
Standard Output:    Compiling hello v0.1.0 (/path/to/build-arm64/tmp/work/qemuarm64-poky-linux/core-image-full-cmdline/1.0/testimage-sdk/hello)                                                                                                   
error[E0464]: multiple candidates for `dylib` dependency `std` found                                                                    
  |                                                                                                                                     
  = note: candidate #1: /path/to/build-arm64/tmp/work/qemuarm64-poky-linux/core-image-full-cmdline/1.0/testimage-sdk/sysroots/cortexa57-poky-linux/usr/lib/rustlib/aarch64-poky-linux-gnu/lib/libstd-905a0b38e33a26a5.so                          
  = note: candidate #2: /path/to/build-arm64/tmp/work/qemuarm64-poky-linux/core-image-full-cmdline/1.0/testimage-sdk/sysroots/cortexa57-poky-linux/usr/lib/rustlib/aarch64-poky-linux-gnu/lib/libstd.so                                           
                                                                                                                                        
error: cannot find macro `println` in this scope                                                                                        
 --> src/main.rs:2:5                                                                                                                    
  |                                                                                                                                     
2 |     println!("Hello, OpenEmbedded world!");                                                                                         
  |     ^^^^^^^                                                                                                                         
                                                                                                                                        
error: requires `sized` lang_item                                                                                                       
                                                                                                                                        
For more information about this error, try `rustc --explain E0464`.                                                                     
error: could not compile `hello` (bin "hello") due to 3 previous errors
Comment 6 Randy MacLeod 2023-12-21 23:28:26 UTC
Bulk move of 52 5.0-M1 bugs to M2.
Comment 7 Randy MacLeod 2024-01-25 14:28:43 UTC
Bulk move of bugs owned by "unassigned" from 5.0-M2 to 5.0-M3.
Comment 8 Randy MacLeod 2024-09-11 20:58:42 UTC
I'm not sure if this error is still happening so I've left it at M3 for discussion at tomorrow's bug review meeting.
Comment 9 Randy MacLeod 2024-09-26 14:58:40 UTC
Sundeep, 
can you have someone review this bug and see if there are still issues.
Comment 10 Deepesh Varatharajan 2024-11-07 06:07:15 UTC
Hi,
 
I tried to reproduce the issue with the following modifications in conf/local.conf file.
 
==============================
IMAGE_CLASSES += "testimage testsdk"
TESTIMAGE_AUTO:qemuall = "1"
SDKMACHINE ?= "x86_64"
==============================
 
Ran the command "bitbake core-image-minimal:do_testsdk -k" for qemumips64 and qemuarm64 targets without any issues.
Hence, it seems like the issue is not present with the latest sources.
Can anyone please let me know if there is any other configurations to be added/modified to check the issue?
 
Thanks in Advance,
Deepesh
Comment 11 Deepesh Varatharajan 2025-02-05 03:51:04 UTC
Hi,
 
I tried to reproduce the issue with the following modifications in conf/local.conf file.
 
==============================
IMAGE_CLASSES += "testimage testsdk"
TESTIMAGE_AUTO:qemuall = "1"
SDKMACHINE ?= "x86_64"
MACHINE ??= "qemuarm64"
==============================
 
Ran the command "bitbake core-image-minimal:do_testsdk -k" for qemuarm64 target without any issues.
For rust we are not supporting mips targets anymore . So not tested for that target.
Hence, it seems like the issue is not present with the latest sources.
Can someone close this issue ?

Thanks,
Deepesh
Comment 12 Sundeep Kokkonda 2025-02-05 04:03:57 UTC
@Randy,

We tried reproducing the issue but not occurring. Can we close this, If there is no new occurrences in autobuilder also?
Comment 13 Randy MacLeod 2025-02-06 15:12:46 UTC
This error has gone away but we want to be sure that if such a problem happens,
then sufficient logs will be gathered in the YP AB that someone will know there was a problem and be able to easily gather relevant data.

Introduce a deliberate error and ensure that it is visible on the bugzilla front-end without logging into the worker.
Comment 14 Randy MacLeod 2025-09-08 17:42:48 UTC
Add Mathieu.

Mathieu, 
Can you help Deepesh and Sundeep with the AB configuration?

Sundeep has privs to run a YP AB build and if they can introduce an 
error similar to this, I'd like to see that we'd be able to automatically 
report on relevant information without always copying over too much data 
that is stored by AB.

Sundeep, Deepesh,
Have you read: https://docs.yoctoproject.org/test-manual/intro.html#yocto-project-autobuilder-overview
and https://docs.yoctoproject.org/test-manual/understand-autobuilder.html

https://git.yoctoproject.org/yocto-autobuilder-helper
Comment 15 Mathieu Dubois-Briand 2025-09-09 13:53:20 UTC
Hi Deepesh, Sundeep,

So I'm not sure what changes are needed here, but yes, the main configuration entry point is in the https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json file. Changes here have to be submitted to the yocto-patches mailing list.

Do you already know the changes that have to be made, or do you have to experiment locally first. If needed, you can run a buildbot instance locally. I have a docker file that tries to be similar to the autobuilder: https://git.yoctoproject.org/yocto-autobuilder2/tree/docker/README.md .

Please tell me if you need any help.
Comment 16 Deepesh Varatharajan 2025-11-12 05:08:56 UTC
(In reply to Randy MacLeod from comment #13)
> This error has gone away but we want to be sure that if such a problem
> happens,
> then sufficient logs will be gathered in the YP AB that someone will know
> there was a problem and be able to easily gather relevant data.
> 
> Introduce a deliberate error and ensure that it is visible on the bugzilla
> front-end without logging into the worker.

Hi Randy,

The issue is no longer reproducing. We tested multiple times with the 
following configuration:

IMAGE_CLASSES += "testimage testsdk"
TESTIMAGE_AUTO:qemuall = "1"
SDKMACHINE ?= "x86_64"
MACHINE ??= "qemuarm64"

As you suggested, if there are any failures, it should be visible in the
console output. To verify this, we intentionally triggered a failure by
adding:

CORE_IMAGE_EXTRA_INSTALL += "rust"

With this change, the expected error appeared in the console:

 error[E0464]: multiple candidates for `rlib` dependency `core` found  |
 = note: candidate #1: /srv/pokybuild/yocto-worker/qemuarm64/build/build/tmp/work/qemuarm64-poky-linux/core-image-sato/1.0/testimage-sdk/.../lib/libcore-5353d19aa4643db9.rlib
 = note: candidate #2: /srv/pokybuild/yocto-worker/qemuarm64/build/build/tmp/work/qemuarm64-poky-linux/core-image-sato/1.0/testimage-sdk/.../lib/libcore-7bb53c140d13aee9.rmeta

 error[E0464]: multiple candidates for `rmeta` dependency `compiler_builtins` found  |
 = note: candidate #1: /srv/pokybuild/yocto-worker/qemuarm64/build/build/tmp/work/qemuarm64-poky-linux/core-image-sato/1.0/testimage-sdk/.../lib/libcompiler_builtins-510f4f2e7881e89f.rmeta
 = note: candidate #2: /srv/pokybuild/yocto-worker/qemuarm64/build/build/tmp/work/qemuarm64-poky-linux/core-image-sato/1.0/testimage-sdk/.../lib/libcompiler_builtins-96b5c563e4eef06d.rlib

https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/2710/steps/15/logs/stdio

The issue occurred because Rust was included in both the SDK and the target image.Including Rust in the image introduced additional files, including different versions of Rust libraries. When Rust is also part of the SDK, this results in duplicate library versions, causing errors during testing.

Since our goal is to validate the SDK, Rust should only be included in the SDK
and not in the target image. With this setup, the tests fails as expected and
errors printed in the console.

Can we close this issue ?

Regards,
Deepesh
Comment 17 Randy MacLeod 2025-11-17 17:45:45 UTC
Hi Deepesh,

I don't test the SDK often but we certainly do need to be able to 
include Rust in the target image since

a) there's no reason that a user shouldn't be able to do that, and 
b) one of our required workflows is to do self-hosted kernel development 
    and we're going to get that working this fall.
 
So, would that just mean that you change a configuration for the SDK or 
does the bug need to be fixed?

As an example, we can put the gcc x-compiler in the SDK and on target and I assume that we can do the same with go.
Comment 18 Deepesh Varatharajan 2026-01-21 06:52:58 UTC
Hello Randy,
 
This bug was raised a long ago and the issue is no more reproducible.
But it was discussed here https://bugzilla.yoctoproject.org/show_bug.cgi?id=14866#c13 that if such error happens error message should be logged on ab.
 
For that, We've made a few changes in SDK testing and introduced a bug and the ab log shows the error message.
 
And, for your question in last comment - Yes we will be able to add Rust in target image and nothing to be fixed.
Comment 19 Randy MacLeod 2026-01-21 21:09:52 UTC
Moving to 6.0-M1 since Medium+ bugs should be assigned a milestone and
so that we can discuss it at an upcoming bug review meeting that covers old milestones. 

I'm willing to close the issue as resolved/works for me.
Comment 20 Randy MacLeod 2026-02-10 19:20:34 UTC
Seems to be gone since this works for Deepesh and me.