Bug 5407 - Create standard method for adding package feeds
Summary: Create standard method for adding package feeds
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 1.6
Hardware: All Multiple
: Medium enhancement
Target Milestone: 2.1 M1
Assignee: Leonardo Sandoval Gonzalez
QA Contact:
URL:
Whiteboard:
: 5435 (view as bug list)
Depends on:
Blocks:
 
Reported: 2013-10-29 11:38 UTC by Paul Eggleton
Modified: 2015-12-17 16:17 UTC (History)
10 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Eggleton 2013-10-29 11:38:44 UTC
Create a standard way of adding feeds to be written into the package manager configuration in the final image (when package-management is in IMAGE_FEATURES).  We have IPK_FEED_URIS, but that is specific to ipk - we should probably add counterparts for RPM and DEB.
Comment 1 Paul Eggleton 2013-10-29 11:40:25 UTC
Also, there is some ipk-specific code for a FEED_URIS variable in image.bbclass in addition to IPK_FEED_URIS handling in rootfs_ipk.bbclass; this should be tidied up.
Comment 2 Paul Eggleton 2013-10-29 11:47:15 UTC
Also worth referring to is distro-feed-configs.bb in meta-oe.
Comment 3 Ross Burton 2014-02-20 17:53:12 UTC
And poky-feed-config-opkg in oe-core.
Comment 4 Hongxu Jia 2014-03-04 12:42:12 UTC
*** Bug 5435 has been marked as a duplicate of this bug. ***
Comment 5 Hongxu Jia 2014-06-04 07:43:44 UTC
The deb/rpm/ipkg all supported by the following patch:

Author: David Nyström <david.c.nystrom@gmail.com>
Date:   Thu Feb 27 21:20:37 2014 +0100

    do_rootfs: Added PACKAGE_FEED_URIS functionality
    
    Adding a common interface to add predefined package manager
    channels to prebuilt rootfs:es.
    
    Adding PACKAGE_FEED_URIS = "http://myre.po/repo/, will
    assume repo directories named (rpm,ipk,deb) as subdirectories
    and statically add them to the rootfs, using the same PKG_ARCHs
    as the build which produced the images.
    
    Tested with RPM, IPK and DEB.
    
    deb feed functionality seem broken, is anyone using this ?
    
    (From OE-Core rev: 9b8811045546ad67b4695d980f09636d5506e50c)

We should clean up the duplicated configs: IPK_FEED_URIS/ PACKAGE_FEED_URIS/ BUILD_IMAGES_FROM_FEEDS/ FEED_URIS/ FEED_DEPLOYDIR_BASE_URI

//Hongxu
Comment 6 Hongxu Jia 2015-06-02 02:21:44 UTC
The deb/rpm/ipkg all supported by the following patch:

Author: David Nyström <david.c.nystrom@gmail.com>
Date:   Thu Feb 27 21:20:37 2014 +0100

    do_rootfs: Added PACKAGE_FEED_URIS functionality
    
    Adding a common interface to add predefined package manager
    channels to prebuilt rootfs:es.
    
    Adding PACKAGE_FEED_URIS = "http://myre.po/repo/, will
    assume repo directories named (rpm,ipk,deb) as subdirectories
    and statically add them to the rootfs, using the same PKG_ARCHs
    as the build which produced the images.
    
    Tested with RPM, IPK and DEB.
    
    deb feed functionality seem broken, is anyone using this ?
    
    (From OE-Core rev: 9b8811045546ad67b4695d980f09636d5506e50c)
Comment 7 Paul Eggleton 2015-06-16 11:02:03 UTC
It's taken me far too long to respond to this - sorry about that.

I think PACKAGE_FEED_URIS is definitely a start, however I think it has some limitations that make it difficult to use in the general case - for example, it hardcodes that the feed will be under /rpm/ or /ipk/ under the specified URI, and that the feed will be split by architecture and nothing else - that's all well and good if all you care about is testing directly pointing to the output of tmp/deploy, but otherwise you might not be prepared to live with those limitations.

Whatever mechanism we have, it ought to be able to replace the existing ones. We still have IPK_FEED_URIS and distro-feed-configs in meta-oe for example, I'd like to think we can have something in OE-Core so both of those can be removed.

Reopening and assigning to me for now - if someone else wants to take it then I'd be happy for that to happen.
Comment 8 Leonardo Sandoval Gonzalez 2015-08-10 23:05:43 UTC
(In reply to comment #7)
> It's taken me far too long to respond to this - sorry about that.
> 
> I think PACKAGE_FEED_URIS is definitely a start, however I think it has some
> limitations that make it difficult to use in the general case - for example,
> it hardcodes that the feed will be under /rpm/ or /ipk/ under the specified
> URI, and that the feed will be split by architecture and nothing else -
> that's all well and good if all you care about is testing directly pointing
> to the output of tmp/deploy, but otherwise you might not be prepared to live
> with those limitations.
> 
> Whatever mechanism we have, it ought to be able to replace the existing
> ones. We still have IPK_FEED_URIS and distro-feed-configs in meta-oe for
> example, I'd like to think we can have something in OE-Core so both of those
> can be removed.
> 
> Reopening and assigning to me for now - if someone else wants to take it
> then I'd be happy for that to happen.

I got a patch (locally, not sent yet to the ML) which uses PACKAGE_FEED_URIS AND a new variable called PACKAGE_FEED_PREFIX to create a full URI. Also, I removed all the hard-coded /rpm/, /ipk/ and /deb/ sub-strings on the links mentioned in your comment. 

Should I remove the IPK_FEED_URIS feature completely? I think we should, and just have one way to include package feeds no matter what the package manager is.
Comment 9 Leonardo Sandoval Gonzalez 2015-08-11 21:29:49 UTC
Patch sent to the ML.

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Subject: [PATCH] lib/oe/package_manager: Include PACKAGE_FEED_PREFIX instead of hardcode paths

Instead of hardcode paths (/rpm/, /ipk/, /deb/), use a user-defined prefix
when creating the URI feeds. URIs now will have the following syntax:

    PACKAGE_FEED_URIS_1/PACKAGE_FEED_PREFIX
    PACKAGE_FEED_URIS_2/PACKAGE_FEED_PREFIX
    .

where PACKAGE_FEED_URIS = "PACKAGE_FEED_URIS_1 PACKAGE_FEED_URIS_2 ...."

[YOCTO #5407]
Comment 10 Leonardo Sandoval Gonzalez 2015-09-15 14:32:47 UTC
Patch merged on master 338c325:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=338c3257f60e99c1fd101d5010daea4099eb1d0e
Comment 11 Alexander Kanevskiy 2015-09-15 14:36:35 UTC
Patch includes code changes. what about documentation changes to document that new behaviour in developer manual ?
Comment 12 Leonardo Sandoval Gonzalez 2015-09-15 15:16:54 UTC
(In reply to comment #11)
> Patch includes code changes. what about documentation changes to document
> that new behaviour in developer manual ?

You are completely right. I will file a bug about it.
Comment 13 Leonardo Sandoval Gonzalez 2015-09-15 15:51:43 UTC
(In reply to comment #11)
> Patch includes code changes. what about documentation changes to document
> that new behaviour in developer manual ?

There is now a bug about documenting the feature

https://bugzilla.yoctoproject.org/show_bug.cgi?id=8301
Comment 14 Leonardo Sandoval Gonzalez 2015-09-17 17:19:06 UTC
The patch did not solve the problem completely: there are two more points to consider. 1) instead of a single PREFIX variable, we should have something like
PACKAGE_FEED_PREFIXES = "/rpm/core2_64 /rpm/all ..." so there is a bit more flexibility to the user (with current implementation ALL URL share the same PREFIX) 2) PREFIX(S) name should be reconsider: SUFFIX(S) can be a better name, because it is actually a suffix inside the FEED.
Comment 15 Leonardo Sandoval Gonzalez 2015-10-05 21:24:31 UTC
New patch sent to the ML: 

http://lists.openembedded.org/pipermail/openembedded-core/2015-October/111297.html

In resume: FEED_PREFIX was rename to FEED_BASE_PATHS and FEED_ARCHS new variable created. More info on patch's description.
Comment 16 Leonardo Sandoval Gonzalez 2015-12-17 16:17:29 UTC
Merge into master branch (6650357)