Bug 9038 - extensible sdk - generates empty manifests/needs file with what is installed
Summary: extensible sdk - generates empty manifests/needs file with what is installed
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: devtools / tool chain (show other bugs)
Version: 2.0
Hardware: All Multiple
: Medium+ normal
Target Milestone: 2.4 M1
Assignee: Aníbal Limón
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks: 9090
  Show dependency tree
 
Reported: 2016-02-03 00:18 UTC by Aníbal Limón
Modified: 2017-12-06 16:52 UTC (History)
9 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
target manifest logs (10.12 KB, text/cache-manifest)
2017-02-21 23:01 UTC, Francisco Pedraza
no flags Details
host manifest log (3.44 KB, text/cache-manifest)
2017-02-21 23:02 UTC, Francisco Pedraza
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Aníbal Limón 2016-02-03 00:18:06 UTC
Currently when execute populate_sdk_ext it's generate a empty target manifest and a host manifest only with buildtools.

We need to give information about what is provided/to install in the extensible SDK because another components needs that info. May be not in the form of manifest but we need another file.

For example the SDK compatibility test suite executes tests based on what SDK provide/install the eSDK and now that information isn't provided.
Comment 1 Francisco Pedraza 2017-02-03 06:32:20 UTC
Well today, one year ago this one was raised .
.
.................() 
..........()...._||_....() 
......()._||_(........)_||_.() 
....._||_(..................)_||_ 
...(_____________________________) 
I am working on it.
Comment 2 Francisco Pedraza 2017-02-03 18:45:54 UTC
As suggested by Anibal, I just added 
INHERIT+="buildhistory"
BUILDHISTORY_COMMIT = "1" to conf/local.conf to check what information displays. 
After that $ bitbake core-image-minimal -c populate_sdk_ext
Comment 3 Francisco Pedraza 2017-02-16 18:41:17 UTC
I created a function at oe.sdk called get_extra_sdkinfo
Inside get_extra_sdkinfo I am doind the following:

import operator
    import math

    extra_info = {}
    extra_info['tasksizes'] = {}
    extra_info['filesizes'] = {}
    for root, _, files in os.walk(sstate_dir):
        for fn in files:
            if fn.endswith('.tgz'):
                fsize = int(math.ceil(float(os.path.getsize(os.path.join(root, fn))) / 1024))
                task = fn.rsplit(':',1)[1].split('_',1)[1].split(',')[0]
                origtotal = extra_info['tasksizes'].get(task, 0)
                extra_info['tasksizes'][task] = origtotal + fsize
                extra_info['filesizes'][fn] = fsize
    return extra_info




I added a validation at python write_target_sdk_manifest{
if d.getVar('BB_CURRENTTASK') == 'populate_sdk_ext'
then I import the get_extra_sdkinfo
in order to write at sstate_dir the content to the variable SDK_EXT_TARGET_MANIFEST. I do the same for 

python write_sdk_test_data() {
but for this case I copare with BB_CURRENTTASK
d.getVar('BB_CURRENTTASK') == 'populate_sdk_ext': 
and write conent  to SDK_EXT_HOST_MANIFEST


Right now is getting an error and I am working on it.
Comment 4 Francisco Pedraza 2017-02-21 23:01:42 UTC
Created attachment 3638 [details]
target manifest logs
Comment 5 Francisco Pedraza 2017-02-21 23:02:20 UTC
Created attachment 3639 [details]
host manifest log
Comment 6 Francisco Pedraza 2017-02-21 23:03:01 UTC
Sent three patches to open embedded lists:

1 - lib/oe/sdk: Add get_extra_sdk_info to reuse code in build history [1]

This function is going to be used for generating the target and host manifest files packages for eSDK.


2 - meta/classes/populate_sdk_base: Only execute write manifest when is populate_sdk [2]

This function is going to be used for generating the eSDK, however the
eSDK package information comes from sstate artifacts instead of package files.

3 - Adds support to generate extensible sdk package manifest

This function writes on host and target manifest files for eSDK. [3]

[1] - http://lists.openembedded.org/pipermail/openembedded-core/2017-February/133110.html

[2] - http://lists.openembedded.org/pipermail/openembedded-core/2017-February/133109.html

[3] - http://lists.openembedded.org/pipermail/openembedded-core/2017-February/133111.html


Currently I tested this implementation and is displaying the installed packages on eSDK properly. I already attached the logs.
Comment 7 Francisco Pedraza 2017-02-22 21:55:31 UTC
Added update due feedback.

Add get_extra_sdk_info to reuse code in buildhistory [1]
This function is going to be used for generating the target and host manifest files packages for eSDK.


Adds support for generating eSDK manifest files [2]
The functionalities to generate SDK and eSDK manifest files are different,
the SDK comes from package information and the eSDK comes from sstate artifacts.
Only execute write_sdk_{host, target}_manifest when is on populate_sdk class.

Adds new functions write_sdk{host, target}_ext_manifest to execute on postprocess
in populate_sdk_ext because at the end we have all the sstate artifacts to generate the manifest.

1 - http://lists.openembedded.org/pipermail/openembedded-core/2017-February/133257.html
2 - http://lists.openembedded.org/pipermail/openembedded-core/2017-February/133258.html
Comment 8 Francisco Pedraza 2017-03-22 18:01:45 UTC
Patch is sent, I will move to M4 due delay in revision.
Comment 11 Francisco Pedraza 2017-06-12 17:38:17 UTC
Waiting for integration, moved to M2.
Comment 13 Mikko Ylinen 2017-07-03 07:27:27 UTC
the manifest is empty with '-c populate_sdk_ext' too (set to blocked by bug #11741)
Comment 14 Aníbal Limón 2017-07-04 20:48:43 UTC
(In reply to comment #13)
> the manifest is empty with '-c populate_sdk_ext' too (set to blocked by bug
> #11741)

When the SDK is constructed with minimal type no toolchain or other sstate components are within, so for run tests needs to be in full mode.

See explanation at,

https://bugzilla.yoctoproject.org/show_bug.cgi?id=11741#c12

Cheers,
Anibal