8883 2016-01-06 11:40:57 +0000 Extensible SDK: sdk-update does not work 2016-02-08 07:40:05 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain 2.0 x86 Multiple VERIFIED FIXED High normal 2.1 M2 1 bluelightning bluelightning bogdanx.a.voiculescu dmitry.rojkov mark.hatle meta.mr.watcher meta.watcher richard.purdie bogdanx.a.voiculescu --- --- No (bug/feature does not impact docs) oldest_to_newest 57571 0 bluelightning 2016-01-06 11:40:57 +0000 After fixing a number of minor issues with the devtool sdk-update process I've hit something more substantial. Do the following: 1) Apply the patchset I just sent - http://patchwork.openembedded.org/patch/111431/ (if not already merged) 2) Build the ext SDK for core-image-minimal: bitbake -c populate_sdk_ext core-image-minimal 3) In a separate shell session, install the produced SDK 4) Edit conf/local.conf in the original build session and add the following line: CORE_IMAGE_EXTRA_INSTALL += "lsof" 5) Build the ext SDK again: bitbake -c populate_sdk_ext core-image-minimal 6) Publish the SDK: oe-publish-sdk <path to SDK installer> <temp path> 7) In a third shell session, cd to <temp path> and share it using: python -m SimpleHTTPServer 8) In the SDK shell session, run: devtool -d sdk-update --skip-prepare http://localhost:8000 9) If you had left out the --skip-prepare, the update would have stalled while it rebuilt a bunch of things it shouldn't have. Instead, in the SDK shell session, run: bash . layers/<core layer>/oe-init-build-env . bitbake -S printdiff core-image-minimal In the output from the last step you can see it thinks it needs to build a bunch of things, and then errors out with: ERROR: Can't find a task we're supposed to have written out? (hash: 39926ae51234fabbfc320b1511d6f107)? The hash in the error turns out to be the hash for gcc-source do_fetch. I'm not sure what to make of this - it should be able to get what it needs to add lsof to the image from sstate, it shouldn't need to build anything. 57607 1 bluelightning 2016-01-07 04:37:04 +0000 This is really bizarre, and I'm convinced there's something going wrong with the task dependency calculation. In the SDK's bitbake environment if I run this it lists whole bunch of real tasks it's going to run: bitbake -n lsof:do_populate_lic lsof:do_package_qa lsof:do_package_write_rpm | cat However, if I drop *any* of the three tasks, or try them individually, I get a small reasonable list of setscene tasks. I suspect there's something funny going on with the locked signatures checking code and whether or not the sstate archives are on the local disk (which may link up with bug 8884), because if I turn SIGGEN_LOCKEDSIGS_CHECK_LEVEL back to "warn", with the three tasks I get a warning for lsof:do_package_write_rpm, with less than that I don't. Additionally, if I run these then bitbake -n lsof | cat just lists one task (lsof.do_build): bitbake lsof:do_package_qa bitbake lsof:do_package_write_rpm bitbake lsof:do_populate_sysroot bitbake lsof:do_populate_lic bitbake lsof:do_packagedata 57652 2 richard.purdie 2016-01-07 18:01:48 +0000 I did try and reproduce this and hit https://bugzilla.yoctoproject.org/show_bug.cgi?id=8897 instead. Will try again... 57670 3 richard.purdie 2016-01-08 11:36:46 +0000 What is odd is that despite that error the sigdata is there: $ ls tmp/stamps/work-shared/gcc-5.3.0-r0.do_fetch.sigdata.* tmp/stamps/work-shared/gcc-5.3.0-r0.do_fetch.sigdata.39926ae51234fabbfc320b1511d6f107 57671 4 richard.purdie 2016-01-08 12:07:10 +0000 The gcc-source stamp is a red herring and a bug in the find_sigs function. This patch "solves" the issue at the expense of being a bit of a hack: From 69b9c0d48dee14fe6f327393378bb13400b28f20 Mon Sep 17 00:00:00 2001 From: Richard Purdie <richard.purdie@linuxfoundation.org> Date: Fri, 8 Jan 2016 12:02:24 +0000 Subject: sstatesig: Handle special case of gcc-source shared-workdir for printsigs Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index 6d1be3e..d65586b 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -236,6 +236,10 @@ def find_siginfo(pn, taskname, taskhashlist, d): localdata.setVar('PR', '*') localdata.setVar('EXTENDPE', '') stamp = localdata.getVar('STAMP', True) + if pn.startswith("gcc-source"): + # gcc-source shared workdir is a special case :( + stamp = localdata.expand("${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}") + filespec = '%s.%s.sigdata.*' % (stamp, taskname) foundall = False import glob -- cgit v0.10.2 57674 5 richard.purdie 2016-01-08 13:42:47 +0000 To further update, the reason that all these tasks get listed by printdiff is that the siginfo data is stripped out of sstate. If I do something like: diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py index 15af4eb..c2b31c6 100644 --- a/meta/lib/oe/copy_buildsystem.py +++ b/meta/lib/oe/copy_buildsystem.py @@ -84,7 +84,7 @@ def prune_lockedsigs(allowed_tasks, excluded_targets, lockedsigs, pruned_output) if invalue: if line.endswith('\\\n'): splitval = line.strip().split(':') - if splitval[1] in allowed_tasks and not splitval[0] in excluded_targets: + if (not allowed_tasks or splitval[1] in allowed_tasks) and not splitval[0] in excluded_targets: f.write(line) else: f.write(line) diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass index 317043d..3ca94c4 100644 --- a/meta/classes/populate_sdk_ext.bbclass +++ b/meta/classes/populate_sdk_ext.bbclass @@ -172,6 +172,7 @@ python copy_buildsystem () { # Filter the locked signatures file to just the sstate tasks we are interested in allowed_tasks = ['do_populate_lic', 'do_populate_sysroot', 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm', 'do_package_write_deb', 'do_package_qa', 'do_deploy'] + allowed_tasks = [] excluded_targets = d.getVar('SDK_TARGETS', True) lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc' oe.copy_buildsystem.prune_lockedsigs(allowed_tasks, then the siginfo is added, the output from printdiff is then: NOTE: Checking sstate mirror object availability (for 19 objects) The differences between the current build and any cached tasks start at the following tasks: /home/richard/poky_sdk/layers/poky/meta/recipes-core/images/core-image-minimal.bb, do_rootfs NOTE: Reparsing files to collect dependency data Writing locked sigs to /home/richard/poky_sdk/locked-sigs.inc ERROR: Bitbake's cached basehash does not match the one we just generated (/home/richard/poky_sdk/layers/poky/meta/recipes-core/images/core-image-minimal.bb.do_image_tar)! ERROR: The mismatched hashes were d16794330387230bf90183853d8891cb and 5d0f4ac3fdd1b70b1059de907a3db0cf Task core-image-minimal:do_rootfs couldn't be used from the cache because: We need hash b4a1f38bf503250063fb252f91217f4e, closest matching task was 9d0abae2b5714928a6f6b9371939da55 Hash for dependent task imagescore-image-minimal.bb.do_packagedata changed from 3a8afff87a5f5b7aebe4aec3fed0d15c to b74e4f83f63bfa0e9bd80fe2440a5052 Unable to find matching sigdata for /home/richard/poky_sdk/layers/poky/meta/recipes-core/images/core-image-minimal.bb.do_packagedata with hashes 3a8afff87a5f5b7aebe4aec3fed0d15c or b74e4f83f63bfa0e9bd80fe2440a5052 I have my do_rootfs split patches applied to this and some fixes/hacked related to that and the eSDK too. The above seems reasonable, except perhaps for the packagedata which as a task probably shouldn't exist for images. Its possible that could cause a problem, in triggering something else. Its also possible that in adding in more sstate sigs, I masked the problem somehow. I need to compare the sstate and see what the differences are. FWIW I'm also experimenting with using xz for compression of the SDK rather than bzip2. 57688 6 richard.purdie 2016-01-08 17:12:54 +0000 Looking at the task log after this upgrade it appears two things happen: a) the sstate for lsof is mising causing it to be built b) the hacks which truncate do_rootfs and friends are missing so it runs, taking time The "core-image-minimal" build actually succeeds, but sets error code as lsof failed from sstate. So exactly what the bug is here I'm not sure, probaby that lsof doesn't make it into sstate and that the rootfs isn't truncated upon upgrade. Obviously adding a bitbake parameter for "setscene only" would go some way to making this easier to fix? 57689 7 richard.purdie 2016-01-08 18:29:25 +0000 I've posted some patches related to this. I believe the underlying issue is that the sdk-update doesn't copy in the lsof artefacts, not points the SDK at any sstate mirror containing them. The system therefore tries to build them when you run "bitbake core-image-minimal". I believe there is a bug in the error handling of setscene tasks that fail, and its those failures which cause tasks like gcc-source to run. This is likely a separate issue to the above and should be split into a separate bug. 57701 8 richard.purdie 2016-01-10 18:12:04 +0000 I've looked into the setscene problem and there is a bug in the code. The issue is that the do_package tasks are not present and it seems that is gcc-runtime:do_package isn't available, it would force gcc to rebuild. The do_package artefacts are deliberately missing from the SDK since nothing should need them. I traced the strange dependency to a bug in bitbake and have sent out a patch for it. This doesn't solve the problem of the lsof artefacts being missing. It does however mean when they are, *just* lsof rebuilds which would be the correct outcome in such a scenario. This should allow the remaining issue of the missing lsof artefacts to be solved more easily in the absence of the other problems. 57703 9 bluelightning 2016-01-11 02:21:41 +0000 OK, thanks for digging into this - it works a lot better after your patches. One thing though - the lsof artifacts *are* available, excluding do_package - they're part of the published SDK. When the update runs it adds SSTATE_MIRRORS_append to the config file pointing to the sstate-cache subdirectory of the update server path. Unfortunately though there's still an issue - it expects an sstate archive for do_package to be there and if it's not that's enough to trigger a full build of lsof; looking at the logs on the server side it doesn't even check whether an sstate archive for do_package_write_rpm is there. 58027 10 bluelightning 2016-01-20 22:42:57 +0000 So I think we've resolved the underlying issue here, but I want to get through the changes I'm currently working on with the update process before declaring victory. It's also worth noting we just discovered that SimpleHTTPServer is not very good at dealing with concurrent requests; it isn't just slow, it will drop connections on the floor and for serving sstate in test situations that's a big problem. For now I am using twisted until I can find something easier to use. 58065 11 richard.purdie 2016-01-21 15:27:07 +0000 Just to help when it comes to closing/testing this, the commits which fixed issues which are discussed above were: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=f0d47a6d9ae773f2c4975679e4dc8cd5ee2700fb http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=9e867efdc87ced0e2743a3c134f30d2637795654 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=43328feccc95a65e6774011aead5ae9f31a92275 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=e306d5495a33c006fc7281c412f70af53d6c41a4 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=1ee08426d792ce27213dd411a885595d8435ae33 58162 12 bluelightning 2016-01-24 20:05:36 +0000 I think we can now mark this as resolved. 58629 13 bogdanx.a.voiculescu 2016-02-05 10:31:43 +0000 I cannot finish the verification since: sdk-update argument is an invalid choice - see bellow the error. I added a comment also on Bug #9043. Still this could be a different bug. devtool sdk-update http://localhost:8000 ERROR: argument <subcommand>: invalid choice: 'sdk-update' (choose from 'create-workspace', 'deploy-target', 'undeploy-target', 'search', 'build', 'add', 'modify', 'extract', 'sync', 'update-recipe', 'status', 'reset', 'upgrade', 'edit-recipe', 'configure-help', 'build-image') usage: devtool [--basepath BASEPATH] [--bbpath BBPATH] [-d] [-q] [--color COLOR] [-h] <subcommand> ... 58698 14 bluelightning 2016-02-06 06:32:45 +0000 Where are you running this? If it's not after installing the extensible SDK and sourcing the environment setup script in a new shell session, then you're not in the right environment for that subcommand to be available. 58710 15 bogdanx.a.voiculescu 2016-02-08 07:38:19 +0000 After switching the env it worked. 58711 16 bogdanx.a.voiculescu 2016-02-08 07:39:20 +0000 sdk$ devtool -d sdk-update --skip-prepare http://localhost:8000 DEBUG: Using basepath /home/yocto/poky_sdk DEBUG: Using bitbake subdir layers/poky/bitbake DEBUG: Loading plugins from /home/yocto/poky_sdk/layers/poky/scripts/lib/devtool... DEBUG: Loading plugin package DEBUG: Loading plugin deploy DEBUG: Loading plugin runqemu DEBUG: Loading plugin search DEBUG: Loading plugin build DEBUG: Loading plugin standard DEBUG: Loading plugin upgrade DEBUG: Loading plugin utilcmds DEBUG: Loading plugin build-image DEBUG: Loading plugin sdk DEBUG: Loading plugins from /home/yocto/poky_sdk/workspace/lib/devtool... DEBUG: Loading plugins from /home/yocto/poky_sdk/layers/poky/meta-yocto/lib/devtool... DEBUG: Loading plugins from /home/yocto/poky_sdk/lib/devtool... DEBUG: Loading plugins from /home/yocto/poky_sdk/layers/poky/meta/lib/devtool... DEBUG: Loading plugins from /home/yocto/poky_sdk/layers/poky/meta-yocto-bsp/lib/devtool... DEBUG: Reading workspace in /home/yocto/poky_sdk/workspace DEBUG: updateserver: http://localhost:8000 DEBUG: basepath = /home/yocto/poky_sdk DEBUG: Found conf/locked-sigs.inc in /home/yocto/poky_sdk NOTE: Already up-to-date 58712 17 bogdanx.a.voiculescu 2016-02-08 07:40:05 +0000 Thank you Paul! I was in the wrong env.