Bug 9599

Summary: Reduce bundle overheads by not depending on os-core when generating bundle sysroots
Product: [Build System, Metadata & Runtime] Other YP Layers Reporter: Joshua Lock <joshuagloe>
Component: meta-swupdAssignee: Joshua Lock <joshuagloe>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium+ CC: igor.stoppa, jose.perez.carranza
Version: 2.2   
Target Milestone: 2.2 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Joshua Lock 2016-05-05 21:54:14 UTC
We currently generate the bundle contents by performing a diff between an os-core image and an image that consists of os-core + the bundle contents.

This adds significant overhead when adding bundles, especially noticeable with small bundles (an extreme example being a bundle which only installs a data file).

It could be possible to improve performance significantly by only installing the contents of the bundle into the swupd image directory for processing by swupd.
Comment 1 Igor Stoppa 2016-05-11 11:53:49 UTC
This would be probably the most important feature, to improve the overall developer's experience.
Comment 2 Joshua Lock 2016-05-11 18:53:33 UTC
(In reply to comment #1)
> This would be probably the most important feature, to improve the overall
> developer's experience.

Thanks for the feedback, the Medium+ Importance indicates that this feature should land in 2.2 M1 (but won't 'block' that milestone being completed) and therefore indicates it's a priority for the current development milestone.

I agree it will be a big improvement for developer workflow and will prioritise accordingly.
Comment 3 Joshua Lock 2016-05-18 15:34:21 UTC
I have a partial implementation for this, unfortunately verification fails as follows:

Verifying files
Hash mismatch for file: /bin
Hash mismatch for file: /etc
Hash mismatch for file: /lib
Hash mismatch for file: /lib/systemd
Hash mismatch for file: /lib/systemd/system
Hash mismatch for file: /lib/systemd/system/multi-user.target.wants
Hash mismatch for file: /lib/systemd/system/sockets.target.wants
Hash mismatch for file: /sbin
Hash mismatch for file: /usr
Hash mismatch for file: /usr/bin
Hash mismatch for file: /usr/sbin
Hash mismatch for file: /usr/share
Hash mismatch for file: /usr/share/dbus-1
Hash mismatch for file: /var
Inspected 2082 files
  14 files did not match

I've pushed the current WIP patch to the joshuagl/wip-rm-bundle-images:

http://git.yoctoproject.org/cgit/cgit.cgi/meta-swupd/commit/?h=joshuagl/wip-rm-bundle-images&id=7e59fdb48dc28c3a8df17282f632a460a05725df
Comment 4 Joshua Lock 2016-05-19 22:21:08 UTC
The proposed solution now produces images which succeed at verification on my relatively simple OS. I intend to do further testing of:
* swupdimage images (images consisting of os-core + N bundles)
* updates
* bundle addition
* bundle removal

I don't expect updates or bundle addition to be affected by the changes, but would like to have some certainty by testing.
Comment 5 Joshua Lock 2016-05-20 12:52:16 UTC
I realised there's an edge case we have to handle differently, when the bundle defines BUNDLE_FEATURES (which adds extra IMAGE_FEATURES for inclusion in that bundle) I think we must handle things the old way with a virtual image.

We presume that the user is aware that an IMAGE_FEATURE that doesn't map to a set of installed packages is not really supported (because a bundle's contents must only add files to the system, not change files in os-core, I realise now that I need to add this to docs/Guide.md) but we can't be sure that an IMAGE_FEATURE directly maps to an installable target (i.e. a packagegroup) — I believe that this is true for dbg, dev and ptest IMAGE_FEATURES, thus for bundles which utilise BUNDLE_FEATURES we must build a virtual image and copy out the files from the resulting image which aren't in os-core.

The good news here is that the copyxattrfiles() function we already added in this series makes copying files from a virtual image more efficient than before.
Comment 6 Joshua Lock 2016-06-01 16:44:51 UTC
This is now implemented in meta-swupd master, the change spans multiple commits with the main ones being:

http://git.yoctoproject.org/cgit/cgit.cgi/meta-swupd/commit/?id=714e77c917cd991e34fd8c7c53293cbf41e81563

and

http://git.yoctoproject.org/cgit/cgit.cgi/meta-swupd/commit/?id=57e636e999cd2598e907fe822a3cf548aac69555

to address the BUNDLE_FEATUREs case