Bug 10283

Summary: Consume TaskArtifacts metadata event created in bitbake
Product: [Build System, Metadata & Runtime] Toaster Reporter: brian avery <brian.avery>
Component: toasterAssignee: David Reyna <david.reyna>
Status: IN PROGRESS IMPLEMENTATION --- QA Contact:
Severity: normal    
Priority: Medium CC: belen.barros.pena, david.reyna, jessica.zhang, jose.perez.carranza, joshuagloe, stanciux.mihail
Version: 2.2   
Target Milestone: 5.99   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on: 9869    
Bug Blocks: 7240, 7502    

Description brian avery 2016-09-15 02:08:20 UTC
bitbake now generates TaskArtifacts metadata event for every artifact created.  Rather than walking the file system, toaster can now use the events generated as part of the change made for Bug 9869
Comment 1 David Reyna 2016-11-23 16:21:17 UTC
Would need to restructure build info helper. Would improve reliability and readability, especially around file types. Does not directly affect user interface.

This is the best way, versus guessing the file paths and file name types (e.g. ARM has a device tree, kernel files compress differently), regex cannot guess everything.
 
Worth considering because might affect new layer types, especially commercialized ones.
Comment 2 David Reyna 2017-01-30 05:43:24 UTC
I have instrumented the TaskArtifacts event handler, and I am getting the data I am expected for the kernel, rootfs, and eSDK. This may be a good fix for 2.3.

The exception is the SDK, because the manifest file name is not as expected "manifest-qemux86x86_64-core-image-minimal" (see below):

* Task Artifacts data:

NOTE: EVENT_NM:TaskArtifacts({
  'task': '/<pokydir>/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_deploy', 
  'artifacts': [
	'/<builddir>/tmp/deploy/images/qemux86/modules--4.8.12+git0+926c93ae07_021b4aef55-r0-qemux86-20170126071216.tgz', 
	'/<builddir>/tmp/deploy/images/qemux86/bzImage-qemux86.bin', 
	'/<builddir>/tmp/deploy/images/qemux86/modules-qemux86.tgz', 
	'/<builddir>/tmp/deploy/images/qemux86/bzImage', 
	'/<builddir>/tmp/deploy/images/qemux86/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt', 
	'/<builddir>/tmp/deploy/images/qemux86/bzImage--4.8.12+git0+926c93ae07_021b4aef55-r0-qemux86-20170126071216.bin'
	]})


NOTE: EVENT_NM:TaskArtifacts({
  'task': '/<pokydir>/meta/recipes-core/images/core-image-minimal.bb:do_image_complete', 
  'artifacts': [
	'/<builddir>/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170126093206.rootfs.manifest', 
	'/<builddir>/tmp/deploy/images/qemux86/core-image-minimal-qemux86.ext4', 
	'/<builddir>/tmp/deploy/images/qemux86/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt', 
	'/<builddir>/tmp/deploy/images/qemux86/core-image-minimal-qemux86.manifest', 
	'/<builddir>/tmp/deploy/images/qemux86/core-image-minimal-qemux86.tar.bz2', 
	'/<builddir>/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170126093206.rootfs.tar.bz2', 
	'/<builddir>/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170126093206.rootfs.ext4'
	]})


NOTE: EVENT_NM:TaskArtifacts({
  'task': '/<pokydir>/meta/recipes-core/images/core-image-minimal.bb:do_populate_sdk_ext', 
  'artifacts': [
	'/<builddir>/tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.2.host.manifest', 
	'/<builddir>/tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.2.target.manifest', 
	'/<builddir>/tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.2.sh'
	]})

* SDK Manifest name issue:

[build]$ ls /<builddir>/tmp/sstate-control | grep populate_sdk
manifest-qemux86-core-image-minimal.populate_sdk_ext
manifest-qemux86x86_64-core-image-minimal.populate_sdk
manifest-x86_64_linux-buildtools-tarball.populate_sdk
manifest-x86_64_linux-uninative-tarball.populate_sdk
[build]$ 
[build]$ cat /<builddir>/tmp/sstate-control/manifest-qemux86x86_64-core-image-minimal.populate_sdk
/<builddir>/tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-2.2+snapshot.host.manifest
/<builddir>/tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-2.2+snapshot.testdata.json
/<builddir>/tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-2.2+snapshot.target.manifest
/<builddir>/tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-2.2+snapshot.sh
[build]$ 
[build]$ ls /<builddir>/tmp/sstate-control | grep x86x86
index-qemux86x86_64
manifest-qemux86x86_64-core-image-minimal.populate_sdk
[build]$
Comment 3 David Reyna 2017-04-08 04:08:33 UTC
1) The SDK and ESDK artifacts via TaskArtifacts have been implemented in 10850. 

The trick around the SDK manifest name was that the meta data used to create the file's name (stamp-extra-info) is lost from the data_smart environment by the time TaskArtifacts executes, so that has to be hardcoded.

  do_populate_sdk[stamp-extra-info] = "${MACHINE}${SDKMACHINE}"
  do_deploy[stamp-extra-info] = "${MACHINE}"
  do_image_complete[stamp-extra-info] = "${MACHINE}"
  do_populate_sdk_ext[stamp-extra-info] = "${MACHINE}"

2) We will implement do_deploy and do_image_complete in the next release, as well as remove the dead do_populate_sdk[_ext] hooks and obsolete handlers (scan_image_artifacts and scan_sdk_artifacts).
Comment 4 David Reyna 2017-08-29 15:58:50 UTC
Moving this to YP-2.5 since it is not blocking.
Comment 5 Randy MacLeod 2023-10-24 14:24:45 UTC
Bulk move from 4.99 or 0.00 to 5.99