Bug 12884

Summary: Extensible SDK: devtool sdk-update fails while Preparing build system
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Vineeth Chowdary <vineeth.chowdary>
Component: Scripts and ToolsAssignee: Pritam Srichandan Sahoo <PritamSrichandan.Sahoo>
Status: NEW --- QA Contact:
Severity: normal    
Priority: Medium CC: alex.kanavin, jaewon, mark.hatle, mathieu.dubois-briand, PritamSrichandan.Sahoo, randy.macleod, ross.burton
Version: 2.4.5   
Target Milestone: 6.1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
sample recipe none

Description Vineeth Chowdary 2018-08-13 14:37:36 UTC
Created attachment 4359 [details]
sample recipe

Rocko ( default configuration ):

Steps:
=====

$ wget http://mirrors.kernel.org/yocto/yocto/yocto-2.4/poky-rocko-18.0.0.tar.bz2
$ tar xvf poky-rocko-18.0.0.tar.bz2
$ cd poky-rocko-18.0.0

(Add any sample recipe "shiva" to core-image-minimal)

$ source oe-init-build-env
$ bitbake core-image-minimal -c do_populate_sdk_ext

(publish eSDK over http )
$ oe-publish-sdk tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.4.sh ~/sdk/update/
$ cd ~/sdk
$ python -m SimpleHTTPServer 8090

(Install eSDK from installer)
$ tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.4.sh -p -y -d ~/sdk/sdk-1
$ . ~/sdk/sdk-1/environment-setup-i586-poky-linux


( Make changes to sample recipe )
( rebuilt e-SDK )
$ rm -rf cache/ tmp/
$ source oe-init-build-env
$ bitbake core-image-minimal -c do_populate_sdk_ext

(Publish new eSDK )
$ oe-publish-sdk tmp/deploy/sdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.4.sh ~/sdk/update/


(Update SDK with proper link )
$ cd ~/sdk/sdk-1
$ . environment-setup-i586-poky-linux
$ devtool sdk-update http://myip:8090/update


Observed Behaviour:
==================
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
Cloning into 'layers'...
NOTE: Preparing build system... (This may take some time.)
ERROR: Unexecuted tasks found in preparation log:
  NOTE: Running task 1309 of 1851 (/ws/ssd2/demo/opensource/sdk/sdk-2/sdk-2/layers/poky-rocko-18.0.0/meta/recipes-test/shiva/shiva.bb:do_fetch)
  NOTE: Running task 1843 of 1851 (/ws/ssd2/demo/opensource/sdk/sdk-2/sdk-2/layers/poky-rocko-18.0.0/meta/recipes-test/shiva/shiva.bb:do_unpack)
  NOTE: Running task 1844 of 1851 (/ws/ssd2/demo/opensource/sdk/sdk-2/sdk-2/layers/poky-rocko-18.0.0/meta/recipes-test/shiva/shiva.bb:do_prepare_recipe_sysroot)
  NOTE: Running task 1845 of 1851 (/ws/ssd2/demo/opensource/sdk/sdk-2/sdk-2/layers/poky-rocko-18.0.0/meta/recipes-test/shiva/shiva.bb:do_patch)
  NOTE: Running task 1846 of 1851 (/ws/ssd2/demo/opensource/sdk/sdk-2/sdk-2/layers/poky-rocko-18.0.0/meta/recipes-test/shiva/shiva.bb:do_configure)
  NOTE: Running task 1847 of 1851 (/ws/ssd2/demo/opensource/sdk/sdk-2/sdk-2/layers/poky-rocko-18.0.0/meta/recipes-test/shiva/shiva.bb:do_compile)
  NOTE: Running task 1848 of 1851 (/ws/ssd2/demo/opensource/sdk/sdk-2/sdk-2/layers/poky-rocko-18.0.0/meta/recipes-test/shiva/shiva.bb:do_install)
  NOTE: Running task 1849 of 1851 (/ws/ssd2/demo/opensource/sdk/sdk-2/sdk-2/layers/poky-rocko-18.0.0/meta/recipes-test/shiva/shiva.bb:do_package)
  NOTE: Running task 1850 of 1851 (/ws/ssd2/demo/opensource/sdk/sdk-2/sdk-2/layers/poky-rocko-18.0.0/meta/recipes-test/shiva/shiva.bb:do_populate_sysroot)
  NOTE: Running task 1851 of 1851 (/ws/ssd2/demo/opensource/sdk/sdk-2/sdk-2/layers/poky-rocko-18.0.0/meta/recipes-test/shiva/shiva.bb:do_packagedata)



Expected Behaviour:
==================
sdk update should update sstate from specified mirror or should build the changed recipe ( shiva )
Comment 1 Jaewon Lee 2019-02-14 19:15:03 UTC
without –skip-prepare, sdk-update is looping through STAMPS_DIR, which is /tmp/stamps, which is not populated if you are extracting esdk in publish mode.
Even extracting without the -p flag though it wasn't syncing sstate for me. After looking into it,

SSTATE_MIRRORS is set in local.conf of esdk, which causes if statement to skip this in sdk.py

  221             with open(os.path.join(conf_dir, 'site.conf'), 'a') as f:
  222                 f.write('SCONF_VERSION = "%s"\n' % site_conf_version)
  223                 f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)

if I get rid of the check to see SSTATE_MIRRORS is set, and make sure site.conf is populated with this SSTATE_MIRRORS_append it works as expected

This workaround wouldnt work for esdks updated in publish mode however, which is common especially if the esdk is shared on an nfs mount point(you cant run bitbake on nfs) what is the fix here? 

There used to be a call to install_sstate_objects before this patch https://gitenterprise.xilinx.com/Yocto/poky/commit/02986886fc40c0fcd33359a279ba9520fd5cac2a when local updates were supported which would work, but now its gone
Comment 2 Jaewon Lee 2019-02-26 00:07:56 UTC
woops here is the upstream patch that used call to install_sstate_objects 
 https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=02986886fc40c0fcd33359a279ba9520fd5cac2a
Comment 3 Paul Eggleton 2019-02-28 20:15:55 UTC
I think I see what happened - the code in sdk.py was written before SSTATE_MIRRORS got set in the eSDK's local.conf (OE-Core commits 7fefdc1e11ecbfafe36e232a57c08d1a31dae6c9 and 6b9e8b780dcd8d5ffba3df35cfe41674413ee26d). The fix would be instead of checking to see if it's set at all, we would check if the path we want to add is already in SSTATE_MIRRORS, although we may need to take care to avoid things like trailing slashes making a path seem like a "new" one.
Comment 4 Paul Eggleton 2019-02-28 20:16:37 UTC
Sorry that's 9cab832b1e85822fa9be5ea9fa0415f48fba86f2 not 7fefdc1e1...
Comment 5 Jaewon Lee 2019-02-28 21:10:23 UTC
cant seem to find commit 9cab832b1e85822fa9be5ea9fa0415f48fba86f2, but yeah that would fix it, but only for non-publish mode. Any ideas on what to do for publish mode? (extracting with -p flag)
Comment 6 Jaewon Lee 2019-03-08 21:28:15 UTC
Also, in response to your previous comment, is the regex check to see if sstate mirror is already in there necessary? Seems like doing a SSTATE_MIRRORS_append would be cleaner?
Comment 7 Randy MacLeod 2019-06-20 15:12:12 UTC
Bruce, can work with Vineeth and other Xilinx devs to figure out the plans and see if someone there can work on the issue.
Comment 8 Bruce Ashfield 2019-07-17 14:22:50 UTC
Sending this back to the original submitter to see if this is still behaviour in the latest. There's been a lot of changes to the eSDK and I'm not sure this is still an issue.
Comment 9 Randy MacLeod 2019-07-18 14:50:30 UTC
Vineeth, can you reply to Bruce and set the defect back to be owned by him?
Comment 10 Vineeth Chowdary 2019-07-26 03:27:27 UTC
I will check and update
Comment 11 Vineeth Chowdary 2019-07-26 03:30:22 UTC
Assigning to Jaewon
Comment 12 Randy MacLeod 2021-04-29 15:27:15 UTC
Ping?
Comment 13 Randy MacLeod 2021-11-11 15:57:53 UTC
Someone volunteered you for the bug Mark.
Comment 14 Randy MacLeod 2023-10-30 15:37:27 UTC
Build move to 5.0 -- ../Randy
Comment 15 Randy MacLeod 2024-04-25 01:13:55 UTC
Bulk move of 5.0 medium importance issues to 5.99.
Move to 5.1 if you want to actively work on an issue.
Comment 16 Randy MacLeod 2024-04-25 01:23:14 UTC
Move to 5.99 since no one is actively working on this issue.
Move to 5.x if you start on a task, please.
Comment 17 Mathieu Dubois-Briand 2026-04-30 15:06:07 UTC
Need to retest and see if this is still an issue
Comment 18 Mathieu Dubois-Briand 2026-04-30 15:06:10 UTC
Need to retest and see if this is still an issue