I'm a bit reluctant to start adding more recXdeptask variants as people already struggle with the differences between the two we have.

Effectively what you're doing is creating two images inside one image recipe which isn't something we officially support. I understand the concern about duplication although there could be other ways to avoid that such as BBCLASSEXTEND of the recipe.

Looking at the code, I do have to wonder if we actually need those problematic self references. If we removed them, does that break the current image generation dependencies? 

In image recipes, we already do:

deltask do_package_qa
deltask do_packagedata
deltask do_package_write_ipk
deltask do_package_write_deb
deltask do_package_write_rpm

so I'm guessing the problem might just be the do_deploy dependencies?

I'd be interested to understand which dependencies in the existing code are causing problems and whether we actually need them.

Hi Richard,

Thank you for your feedback. I do understand you're not really keen on adding a new recursive task flag.

Anyway I would like to clarify how we build and use containers, and to give you a broad picture of our environment.
Maybe you will be able to advise me a better solution.

We mainly use containers for security reasons, to insulate critical code which handles secrets from code which needs to interact with the external environment.

Except for the init manager itself, all services run in containers, which means we have many of them.

Each service has a recipe which packages the daemon, its libraries, configuration files... This recipe has nothing special, it's just a regular one.

For each service, we have a container recipe. It uses the service's packages to build the container's rootfs.
In turn, it also generates packages, typically one containing the container's files (rootfs, config...), one with the init manager scripts.
This recipe is also almost a recipe except it inherits our container class which adds the do_rootfs() task and provides a default implementation of do_install().

When the container recipe is executed, the following tasks are executed:
- do_fetch / do_unpack / ... / do_compile: same sequence as a regular recipe
- do_rootfs: builds the rootfs
- do_install: copy the rootfs, create the LXC configuration and scripts, tune file ownerships and permissions 
- do_packagedata / do_package / do_package_qa / do_package_write_xxx: as usual

Here's a simplified version of our container class:

<<<
...
# Directory to generate the container's rootfs into.
CONTAINER_ROOTFS_BUILD ?= "${WORKDIR}/rootfs"

# Packages to install into the container's rootfs.
CONTAINER_INSTALL ?= ""

# Packages to forcibly remove from the container's rootfs after all the
# packages in ${CONTAINER_INSTALL} have been installed and post install scripts
# have been executed.
CONTAINER_UNNEEDED ?= "base-files base-passwd libcap-bin shadow \
    shadow-securetty update-rc.d ${VIRTUAL-RUNTIME_update-alternatives}"
...
inherit ... image-postinst-intercepts rootfs_${IMAGE_PKGTYPE}
...
# Disable conflicting packaging operations.
EXCLUDE_FROM_SHLIBS = "1"
INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
INHIBIT_PACKAGE_STRIP = "1"
PACKAGE_NO_LOCALE = "1"
...
fakeroot python do_rootfs() {
    from oe.rootfs import create_rootfs
    from oe.manifest import create_manifest

    localdata = d.createCopy()

    logger = localdata.getVar("BB_TASK_LOGGER", False)
    if logger:
        logcatcher = bb.utils.LogCatcher()
        logger.addHandler(logcatcher)
    else:
        logcatcher = None

    localdata.setVar("FORCE_RO_REMOVE", "1")
    localdata.setVar("IMAGE_GEN_DEBUGFS", "0")
    localdata.setVar("LINGUAS_INSTALL", "")
    localdata.setVar("NO_RECOMMENDATIONS", "1")
    localdata.setVar("PACKAGE_INSTALL", d.getVar("CONTAINER_INSTALL"))
    localdata.setVar("PACKAGE_INSTALL_ATTEMPTONLY", "")
    localdata.setVar("ROOTFS_RO_UNNEEDED", d.getVar("CONTAINER_UNNEEDED"))

    localdata.setVar("D", d.getVar("CONTAINER_ROOTFS_BUILD"))
    localdata.setVar("IMAGE_ROOTFS", d.getVar("CONTAINER_ROOTFS_BUILD"))

    localdata.setVar("DISTRO_FEATURES", "")

    localdata.setVar("IMAGE_FEATURES", "read-only-rootfs")

    localdata.setVar("ROOTFS_PREPROCESS_COMMAND", "")
    localdata.setVar("ROOTFS_POSTINSTALL_COMMAND", "")
    localdata.setVar("ROOTFS_POSTPROCESS_COMMAND", "")
    localdata.setVar("ROOTFS_POSTUNINSTALL_COMMAND",
                     "container_check_installed_packages")

    localdata.setVar("DEB_PREPROCESS_COMMANDS", "")
    localdata.setVar("DEB_POSTPROCESS_COMMANDS", "")

    localdata.setVar("OPKG_PREPROCESS_COMMANDS", "")
    localdata.setVar("OPKG_POSTPROCESS_COMMANDS", "")

    localdata.setVar("RPM_PREPROCESS_COMMANDS", "")
    localdata.setVar("RPM_POSTPROCESS_COMMANDS", "")

    localdata.setVar("IMAGE_DEVICE_TABLE", "")
    localdata.setVar("IMAGE_DEVICE_TABLES", "")
    localdata.setVar("USE_DEVFS", "0")

    localdata.setVar("USE_DEPMOD", "0")

    localdata.setVar("LDCONFIGDEPEND", "")

    create_manifest(localdata)

    create_rootfs(localdata, logcatcher=logcatcher)
}

do_rootfs[dirs] = "${TOPDIR}"
do_rootfs[cleandirs] = "${CONTAINER_ROOTFS_BUILD}"
do_rootfs[vardeps] += "container_check_installed_packages"
do_rootfs[file-checksums] += "${POSTINST_INTERCEPT_CHECKSUMS}"
do_rootfs[depends] += "\
    makedevs-native:do_populate_sysroot \
    virtual/fakeroot-native:do_populate_sysroot"
do_rootfs[rdepends] += "${@' '.join(['%s:do_package_write_${IMAGE_PKGTYPE}' \
    % name for name in (d.getVar('CONTAINER_INSTALL') or '').split()])}"
do_rootfs[recxdeptask] := "${@d.getVarFlag('do_rootfs', 'recrdeptask')}"
unset do_rootfs[recrdeptask]
>>>

In our image recipe, we install the container's packages in the usual way.

Let me know if my description is clear and if you can envisage other solutions to solve our container rootfs build issues.

Cheers

We split package generation and image generation into distinct separate recipes, primarily because the dependency needs are quite different. Here, you're trying to merge them back together and than hack bitbake to make it work. We specifically avoided doing this to try and keep things simpler as the complexity in bitbake is already bad enough.

As an example, think about what happens with licensing when you do this. It is far from clear which license code should be running, where, or what it should handle the licenses for. I appreciate this probably isn't a concern for you, but any code we accept would need to handle a lot more corner cases.

I suspect that you can do what you're looking at with BBCLASSEXTEND and making the single recipe have two different variants, one which generates the packages and the other which generates the image from the packages.

For an example of what you can do with BBCLASSEXTEND, have a look at the mcextend.bbclass in core. native.bbclass, nativesdk.bbclass and others are more complex versions of that, mcextend is simpler. In your case you could simply just have two classes, you don't need a lot of the templating or renaming those core classes do.

I am curious if we could tweak the behaviour of the existing rrecdeptask code without adverse effects. Looking at your patches, I think I can see a hint it might be possible but it is hard to say without some experiments and analysis. I suspect the problem with image.bbclass as it stands is the do_populate_lic task which is present in both images and recipes.

Hi Richard,

is there anything I can do to make this ticket move forward or should I just wait?

If you do think this change is not something your project wants to integrate, feel free to let me know. I understand our use case may not be something you want to focus on.

We are aware our implementation of container generation is just a first step to a well-integrated solution. We know we have issues we must deal with such as licenses as you pointed out. But we think it may be of interest to other users.

(In reply to Gauthier HADERER from comment #4)
> is there anything I can do to make this ticket move forward or should I just
> wait?

I think I was wondering if you had any thoughts on the things I mentioned in my reply. Could we change the existing behaviour without too much impact to work for both use cases?

My feeling is something may be possible but someone would need to do some analysis to work out if that is the case. If you are able that would help move it forward. It may be a while before I find any time to look at this myself.

> If you do think this change is not something your project wants to
> integrate, feel free to let me know. I understand our use case may not be
> something you want to focus on.
> 
> We are aware our implementation of container generation is just a first step
> to a well-integrated solution. We know we have issues we must deal with such
> as licenses as you pointed out. But we think it may be of interest to other
> users.

I'm reluctant to take something which only solves part of the problem, particularly when I don't think the license piece will be solvable with the approach you're taking. Once we add functionality like this, it is near impossible to remove so we need to be sure it will work and I'm not convinced of that as yet.

Created attachment 4992 [details]
log.do_rootfs

This file contains the error for do_rootfs

(In reply to Piotr Łobacz from comment #6)
> Created attachment 4992 [details]
> log.do_rootfs
> 
> This file contains the error for do_rootfs

I'm sorry this was not my intention to put this file in here but in other bug... Can someone delete it?