Bug 3505

Summary: world-image as bitbake build target
Product: [Build System, Metadata & Runtime] BitBake Reporter: Martin Jansa <Martin.Jansa>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium CC: poky.bs.watcher, poky.watcher, sgw
Version: unspecified   
Target Milestone: 2.1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)
Attachments:
Description Flags
0001-world-image-add-extra-target.patch none

Description Martin Jansa 2012-11-28 15:23:22 UTC 
for my feed I'm using special image which installs all packagegroup-* to check that all packages can be installed by package manager

I think it would be useful for jenkins/autobuilder to build world-image instead of just world, because it tests more (file conflicts, preinst/postinst scripts, incompatible versions)

I've checked how bitbake generates list of build targets for "world"
and I haven't found way to add world-image.bb recipe which would IMAGE_INSTALL = "world"

I was thinking about saving that list in bitbake code to some variable and then reuse it from recipe
or better add another world-image set, which would gather not all PF, but sum of all PACKAGES unless some are excluded by 
EXCLUDED_FROM_WORLD_IMAGE = "runtime_package_foo" (we need to exclude more packages then from world as some packages are correctly RCONFLICTing with each other, but can be build in parallel).

I'm not sure if bitbake code (cooker + cache) is right place to do this..

Best way would be to have packagegroup-world and world-image recipe, so that we can test also upgrade path easily.
Comment 1 Martin Jansa 2012-12-12 05:02:18 UTC 
Created attachment 977 [details]
0001-world-image-add-extra-target.patch
Comment 2 Martin Jansa 2012-12-12 05:03:22 UTC 
(In reply to comment #1)
> Created attachment 977 [details]
> 0001-world-image-add-extra-target.patch

This is unfinished version of what I was trying to do to support world image.
Comment 4 Martin Jansa 2013-10-08 16:56:28 UTC 
This task is a bit stuck on weird OSError: [Errno 7] Argument list too long

Please check if you have any idea about issue described in this thread:
http://lists.openembedded.org/pipermail/openembedded-core/2013-January/074001.html
Comment 5 Martin Jansa 2014-04-02 17:56:44 UTC 
There are 2 implementations, preferred is "ad-hoc" image, because doesn't need bitbake change and is more flexible:
http://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/target-image

and world-image:
http://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/world-image
+
https://github.com/shr-distribution/bitbake/tree/jansa/world-image

The problem is the OSError from previous comment, but I haven't tried it many times since then.
Comment 7 Richard Purdie 2016-04-04 13:22:50 UTC 
With the patches from Paul, you can write a recipe like the following which 'works' against OE-Core assuming the bug it finds in cogl is fixed:

SUMMARY = "Image which includes world packages"

WORLD_IMAGE_EXCLUDE ?= "epiphany webkitgtk qt4 forktest nss meta-world-pkgdata"

def extend_image_install(d):
    if not d.getVar("BB_WORKERCONTEXT", False):
        return ""

    import oe.packagedata

    pkgdatadir = d.getVar("PKGDATA_DIR", True)

    packagelist = []

    files = os.listdir(pkgdatadir)
    for pn in filter(lambda f: not os.path.isdir(os.path.join(pkgdatadir, f)), files):
        try:
            pkgdata = oe.packagedata.read_pkgdatafile(os.path.join(pkgdatadir, pn))
        except OSError:
            continue

        packages = pkgdata.get("PACKAGES") or ""

        for pkg in packages.split():
            if pkg.endswith("-dbg"):
                continue
            if pkg.endswith("-dev"):
                continue
            if pkg != pn:
                continue
            if not os.path.exists(os.path.join(pkgdatadir, "runtime", pn + ".packaged")):
                continue
            if pkg.startswith("nativesdk-"):
                continue
            if pkg in ["kbd", "dropbear", "console-tools", "packagegroup-core-ssh-dropbear"]:
                continue
            packagelist.append(pkg)

    return " ".join(packagelist)

IMAGE_INSTALL_append = "${@extend_image_install(d)}"

python calculate_extra_depends() {
    exclude = '${WORLD_IMAGE_EXCLUDE}'.split()
    for p in world_target:
        if p == self_pn:
            continue

        if p in exclude:
            continue

        deps.append(p)
}
Comment 8 Richard Purdie 2016-04-13 15:00:11 UTC 
Resolving as the example given here appears to be able to do what is needed (or be extended so it does so).