Bug 10850 - esdk creation on Toaster does not show necessary file in artifacts
Summary: esdk creation on Toaster does not show necessary file in artifacts
Status: RESOLVED FIXED
Alias: None
Product: Toaster
Classification: Build System, Metadata & Runtime
Component: toaster (show other bugs)
Version: 2.3
Hardware: x86 Multiple
: Medium normal
Target Milestone: 2.3 M4
Assignee: David Reyna
QA Contact: Libertad
URL:
Whiteboard:
: 10851 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-12-22 20:43 UTC by brian avery
Modified: 2017-04-11 23:54 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description brian avery 2016-12-22 20:43:32 UTC
To replicate:
build core-image-minimal:do_populate_sdk_ext
machine=qemux86,system=x86-64

Goto Build Summary for the above build.
Artifacts listed are:
Image files - ext3,ext4,jff2,tar.bz

kernel artifacts -
bzImage--4.8.12+git0+926c93ae07_021b4aef55-r0-qemux86-20161222184253.bin (6.7 MB)
modules--4.8.12+git0+926c93ae07_021b4aef55-r0-qemux86-20161222184253.tgz (4.3 MB)

SDK artifacts:
x86_64-buildtools-nativesdk-standalone-2.2.host.manifest (5.1 KB)
x86_64-buildtools-nativesdk-standalone-2.2.sh (24.8 MB)
x86_64-buildtools-nativesdk-standalone-2.2.target.manifest (0 B)
x86_64-nativesdk-libc.host.manifest (311 B)
x86_64-nativesdk-libc.tar.bz2 (2.4 MB)
x86_64-nativesdk-libc.target.manifest (0 B)

----
Unfortunately, the file one actually needs to download in order to install and use the esdk is:
build-toaster-2/tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.2.sh

and it's associated manifest files.
Comment 1 David Reyna 2017-01-23 10:12:20 UTC
When I ran this from the command line under the debugger, all of the expected SDK files were found scan_sdk_artifacts() and added.

When I ran this from within Toaster, the ESDK files were indeed not found. However, after much debugging I discovered they were in fact _not_ there when the event handler scan_sdk_artifacts() was executed, even thought that handle ran at least one minute after the date/times of the ESDK files.

It seems that the ESDK files are being moved to the "sdk" directory _after_ the Toaster handle is executed, which would explain why the timestamps are correct but misleading.

This was the definitive test:

   def scan_sdk_artifacts(self, event):
       ...
       logger.info("SDK_DATE:"+str(subprocess.Popen("date", \
           stdout=subprocess.PIPE, shell=True, \
           executable="/bin/bash").stdout.read()))
       logger.info("SDK_LS:"+str(subprocess.Popen("ls -la /opt/dreyna \
           /toaster_master/poky/build-toaster-2/tmp/deploy/sdk", \
           stdout=subprocess.PIPE, shell=True, \
           executable="/bin/bash").stdout.read()))

The "date" says the files should have been found, the the "ls" shows that they were not there yet.
Comment 2 David Reyna 2017-01-26 04:26:53 UTC
In examining the event flow against the actual SDK_EXT file placement, plus the DEBUG information in "work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_populate_sdk_ext", it is clear that the population of "tmp/deploy/sdk" now happens after "do_populate_sdk_ext", which is why this is failing for Toaster. Specifically, the file placement now happens in the function "sstate_install()" in "meta/classes/sstate.bbclass".

Here is what appears to be the pertinent commits:

	git log meta/classes/populate_sdk_ext
	...
	commit e1de69667481749d4e1081210a3a216378d034c9
	Date:   Fri Sep 2 11:22:41 2016 +0100
		populate_sdk_ext: Put populate_sdk_ext under sstate control

	commit 3c3962d27e659c1da153c588948122dae20a9d93
	Date:   Thu Sep 1 11:56:02 2016 +0300
		populate_sdk_base: Put populate_sdk under sstate control
		
The solution appears to be to hook "sstate_install" instead of "do_populate_sdk_ext" (which now only populates the package and not the top-level deploy directory).
Comment 3 David Reyna 2017-04-08 03:51:15 UTC
*** Bug 10851 has been marked as a duplicate of this bug. ***
Comment 4 David Reyna 2017-04-08 03:55:35 UTC
The solution had three parts:

1) Enable the TaskArtifacts event. It had been mostly implemented but not completed.

2) Make a special case for naming the SDK manifest file, because the meta used to create the file is lost from the data_smart environment by the time TaskArtifacts executes.

3) Add a scan_task_artifacts handler to add the SDK files from the manifest to the target.
Comment 5 David Reyna 2017-04-11 23:54:48 UTC
commit ba1d8e36917f9d048e0a6b1bc3450b492497c63c

This patch provides the first half of the implementation of #10283.