Bug 8318 - Avoid unnecessary package feed churn due to packages rebuilding
Summary: Avoid unnecessary package feed churn due to packages rebuilding
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 1.8
Hardware: All Multiple
: Medium+ enhancement
Target Milestone: 2.2 M3
Assignee: Robert Yang
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-09-17 14:07 UTC by Paul Eggleton
Modified: 2017-12-06 16:48 UTC (History)
9 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Eggleton 2015-09-17 14:07:37 UTC 
When a dependency causes a recipe to effectively be rebuilt, its output may in fact not change; but new packages (with an increased PR value, if using the PR server) will be generated nonetheless. There's no practical way for us to predict whether or not this is going to be the case based solely on the inputs, but we could compare the package output and see if that is materially different and based upon that decide to replace the old package with the new one.

I've created a class to do this but it needs further work. Here's the link to the mailing list discussion:

 http://thread.gmane.org/gmane.comp.handhelds.openembedded.core/68979

And the class itself (the other changes in the patchset have been merged).

 http://cgit.openembedded.org/openembedded-core-contrib/diff/meta/classes/packagefeed-stability.bbclass?h=paule/packagefeed-stability
Comment 2 Joshua Lock 2016-01-15 10:56:19 UTC 
I started taking a look at this work and added a couple of patches to:
1) Prevent do_package_compare being run for native packages (this was a hammer to prevent trying to run build-compare on the build-compare package).
2) Prevent do_package_compare being run for recipes where PACKAGES is empty (this was to prevent trying to run do_package_compare on gcc-source, which has no packages)
3) Ensure the -prediff directory is created (by setting the 'dirs' varflag on the do_package_compare function)
4) Update to the latest build-compare.

Having looked over some of the build-compare output I think we need to do something smarter with dev packages. For example packagegroup-core-boot is being copied with the following message:

packagegroup-core-boot: copying packagegroup-core-boot because packagegroup-core-boot-dev has a fixed version dependency on it and that package is going to be copied.

As indicated the only difference in the -dev file is the PR of the base package having increased:

./control differs (ASCII text)
--- old/./control       2016-01-14 23:00:19.303712892 +0000
+++ new/./control       2016-01-14 23:00:19.311712932 +0000
@@ -9,6 +9,6 @@
 License: MIT
 Architecture: qemux86
 OE: packagegroup-core-boot
-Depends: packagegroup-core-boot (= 1.0-r17.0)
+Depends: packagegroup-core-boot (= 1.0-r17.1)
 Recommends: base-passwd-dev, netbase-dev, udev-dev, base-files-dev, sysvinit-dev, busybox-hwclock-dev, modutils-initscripts-dev, v86d-dev, busybox-dev, update-alternatives-opkg-dev, init-ifupdown-dev, initscripts-dev

Can we just skip -dev and -dbg packages as suggested by Mark?

I think the conclusion of the thread where Paul notes his concern regarding patching a header is that only -dbg packages should be skipped.
Thus we perhaps need to improve the smarts around comparison of versioned Depends on the base package?

WIP tree is: 
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=joshuagl/packagefeed-stability
Comment 3 Joshua Lock 2016-01-15 11:35:59 UTC 
(In reply to comment #2)
>
> Having looked over some of the build-compare output I think we need to do
> something smarter with dev packages. For example packagegroup-core-boot is
> being copied with the following message:
> 
> packagegroup-core-boot: copying packagegroup-core-boot because
> packagegroup-core-boot-dev has a fixed version dependency on it and that
> package is going to be copied.
> 
> As indicated the only difference in the -dev file is the PR of the base
> package having increased:
> 
> ./control differs (ASCII text)
> --- old/./control       2016-01-14 23:00:19.303712892 +0000
> +++ new/./control       2016-01-14 23:00:19.311712932 +0000
> @@ -9,6 +9,6 @@
>  License: MIT
>  Architecture: qemux86
>  OE: packagegroup-core-boot
> -Depends: packagegroup-core-boot (= 1.0-r17.0)
> +Depends: packagegroup-core-boot (= 1.0-r17.1)
>  Recommends: base-passwd-dev, netbase-dev, udev-dev, base-files-dev,
> sysvinit-dev, busybox-hwclock-dev, modutils-initscripts-dev, v86d-dev,
> busybox-dev, update-alternatives-opkg-dev, init-ifupdown-dev, initscripts-dev
> 
> Can we just skip -dev and -dbg packages as suggested by Mark?

Having just taken a look at Paul's changes to build-compare I'm wondering whether the solution here is to update adjust_controlfile () in functions.sh to up Depends entries which match the base package?
Comment 4 Mark Hatle 2016-01-15 13:48:11 UTC 
I think we should definitely skip comparison of the '-dbg' files.  While possible to do a package upgrade due to '-dbg', this will be such a rare occurrence that I don't think it needs to be factored in.

Due to the potential for header changes I agree with Paul on the main and -dev packages should be compared.

As mentioned in the thread a simple optimization is as soon as the first package is found to have a difference stop comparing that series and copy/install them all (as a group).  Any one change, means they all changed.

As for the PR stuff, I though, at least on the RPM comparison, that the PR values of dependencies were filtered out of the comparison, since we do expect it to change -- but specific PR versions not be dependent between recipes.  (Versions yes, but not PR.)
Comment 5 Joshua Lock 2016-01-18 16:39:20 UTC 
(In reply to comment #4)
> I think we should definitely skip comparison of the '-dbg' files.  While
> possible to do a package upgrade due to '-dbg', this will be such a rare
> occurrence that I don't think it needs to be factored in.

Noted, I'll work on making this change.

> Due to the potential for header changes I agree with Paul on the main and
> -dev packages should be compared.

OK, great.

> As mentioned in the thread a simple optimization is as soon as the first
> package is found to have a difference stop comparing that series and
> copy/install them all (as a group).  Any one change, means they all changed.

Yep, this is on my todo too.

> As for the PR stuff, I though, at least on the RPM comparison, that the PR
> values of dependencies were filtered out of the comparison, since we do
> expect it to change -- but specific PR versions not be dependent between
> recipes.  (Versions yes, but not PR.)

Right, this is using stock OE-Core's default configuration so it's generating ipk that include the version of the main package in the Depends: of -dev and -dbg packages.

I hacked a nasty little change to build-compare to delete the version for the sake of the comparison:

http://git.openembedded.org/openembedded-core-contrib/commit/?h=joshuagl/packagefeed-stability&id=c929ba4661efe2c317261602c9044a6e96380569

After which I get fewer packages copied, but need to change the logic in the class as you've suggested so that when one generated package is copied for a recipe they all are (with that branch as it stands I get some of the python modules copied, but not all of them, which seems very undesirable).

I need to level up in Awk in order to make it cleaner, though.
Comment 6 Paul Eggleton 2016-07-12 22:48:48 UTC 
Robert, it looks like you've taken this one over based on your recently submitted patchset (thanks!) so I'll assign this one to you.
Comment 7 Robert Yang 2016-07-13 01:48:38 UTC 
Yes, currently, the rpm part have been sent to oe-core mailing list, I'm working on ipk and deb part.

git://git.openembedded.org/openembedded-core-contrib rbt/feed

I spent about 1 week on rpm, I guess I need about 20 days for all of them.