Bug 13053

Summary: oe-core OCI container compatibility / deployment and conversion
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Bruce Ashfield <bruce.ashfield>
Component: coreAssignee: Bruce Ashfield <bruce.ashfield>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium+ CC: kouheiaoki, meta.mr.watcher, meta.watcher, pokylinux, randy.macleod, richard.purdie
Version: 2.7   
Target Milestone: 6.0 M2   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)

Description Bruce Ashfield 2018-11-30 20:44:10 UTC
The prerequisite for this feature is bugzilla: 13051

An OCI container generated via oe-core should be capable of being copied (pushed) to a container registry, to be pull/executed on a different execution environment.

This means that the OCI container should support both the OCI imagespec v1 and the docker image format. Sufficient metadata from the build will be part of the image such 
that it can be executed on another host/container engine.

It should also be possible to install/nest the generated container "image" such that it can be included in a rootfs/image and run on target.

This forms the foundation of a CI/CD loop, but actual integration of the outputs into a CI environment is not part of this feature.
Comment 1 Bruce Ashfield 2019-08-27 15:30:33 UTC
This will be in meta-virt for 3.0, moving to 3.1 for the larger oe-core discussion about container support in core.
Comment 2 Robert Berger 2019-10-04 08:48:41 UTC
@Bruce: 
Do I understand this correctly? 
All the container stuff is being moved from 3.0 to 3.1? 

and it will show up here?
http://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization

In case you have something preliminary to test please let me know.

At the moment I create a rootfs suitable for a container and out of this a docker image. First preliminary experiments to use this with kubernetes are also being made.
Comment 3 Bruce Ashfield 2019-10-04 12:34:25 UTC
(In reply to comment #2)
> @Bruce: 
> Do I understand this correctly? 
> All the container stuff is being moved from 3.0 to 3.1? 
> 
> and it will show up here?
> http://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization
> 
> In case you have something preliminary to test please let me know.
> 
> At the moment I create a rootfs suitable for a container and out of this a
> docker image. First preliminary experiments to use this with kubernetes are
> also being made.

Correct. The prequisite support (but 13051) is already in meta-virt and has been used for some time to create directly deployable containers (and we now have docker, runc, podman and k8s available in meta-virt to run them on target).

The "on host" or runquemu-like work is nearly done and that will be part of the 3.1 effort. Patches will show up in meta-virt first for that.

That being said, the containers that are being generated now could easily be deployed and run with many different container frameworks, it just needs a bit of glue for whatever you are using (that assumes that you are running on a matching architecture, or you've arranged for qemu-system* to run things if the architecture isn't a match).
Comment 4 Robert Berger 2019-10-04 12:59:22 UTC
(In reply to comment #3)
> 
> Correct. The prequisite support (but 13051) is already in meta-virt and has
> been used for some time to create directly deployable containers (and we now
> have docker, runc, podman and k8s available in meta-virt to run them on
> target).

I am very interested in this. Will you be at ELCE?

> 
> The "on host" or runquemu-like work is nearly done and that will be part of
> the 3.1 effort. Patches will show up in meta-virt first for that.

Not quite sure what you mean there.

> 
> That being said, the containers that are being generated now could easily be
> deployed and run with many different container frameworks, it just needs a
> bit of glue for whatever you are using (that assumes that you are running on
> a matching architecture, or you've arranged for qemu-system* to run things
> if the architecture isn't a match).

At the moment I build a rootfs without kernel and so on for x86[1] and ARM and manually run some script, which uses this dockerfile[2] which just adds the rootfs which I generated before via bitbake (ADD app-container-image-mosquitto-container-x86-64.tar.bz2 /)

[1] https://gitlab.com/meta-layers/meta-resy/blob/master/conf/machine/container-x86-64.conf
[2] https://gitlab.com/app-container/app-container-mosquitto/blob/master/dockerfile/Dockerfile

Basically what I want is containers which can be deployed/managed via kubernetes where I can put pods on Embedded ARM boards while other server components run on x86 servers. So some containers will need to be x86 and ARM and the appropriate one should be picked per node.

We definitely need to talk ;)
Comment 5 Bruce Ashfield 2019-10-08 15:11:50 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > 
> > Correct. The prequisite support (but 13051) is already in meta-virt and has
> > been used for some time to create directly deployable containers (and we now
> > have docker, runc, podman and k8s available in meta-virt to run them on
> > target).
> 
> I am very interested in this. Will you be at ELCE?
> 
> > 
> > The "on host" or runquemu-like work is nearly done and that will be part of
> > the 3.1 effort. Patches will show up in meta-virt first for that.
> 
> Not quite sure what you mean there.

In a similar way that runqemu can test our images on your dev host, we can
build containers and execute them directly on the dev-host the same way. It
isn't overly complex, but the glue and integration is nearly done.

> 
> > 
> > That being said, the containers that are being generated now could easily be
> > deployed and run with many different container frameworks, it just needs a
> > bit of glue for whatever you are using (that assumes that you are running on
> > a matching architecture, or you've arranged for qemu-system* to run things
> > if the architecture isn't a match).
> 
> At the moment I build a rootfs without kernel and so on for x86[1] and ARM
> and manually run some script, which uses this dockerfile[2] which just adds
> the rootfs which I generated before via bitbake (ADD
> app-container-image-mosquitto-container-x86-64.tar.bz2 /)
> 

We have other people doing similar things with tools like skopeo, and this
is an area that I'll be looking to drive some convergence.

> [1]
> https://gitlab.com/meta-layers/meta-resy/blob/master/conf/machine/container-
> x86-64.conf
> [2]
> https://gitlab.com/app-container/app-container-mosquitto/blob/master/
> dockerfile/Dockerfile
> 
> Basically what I want is containers which can be deployed/managed via
> kubernetes where I can put pods on Embedded ARM boards while other server
> components run on x86 servers. So some containers will need to be x86 and
> ARM and the appropriate one should be picked per node.

All the components are in place for this, so we are generally ready to go.

I will be at ELCE!

> 
> We definitely need to talk ;)
Comment 6 Robert Berger 2019-10-08 19:41:39 UTC
(In reply to comment #5)
> > 
> > Basically what I want is containers which can be deployed/managed via
> > kubernetes where I can put pods on Embedded ARM boards while other server
> > components run on x86 servers. So some containers will need to be x86 and
> > ARM and the appropriate one should be picked per node.
> 
> All the components are in place for this, so we are generally ready to go.
> 
> I will be at ELCE!
> 
> > 
> > We definitely need to talk ;)

OK please reserve some time for me on ELCE. Dinner is on me ;)
Comment 7 Robert Berger 2020-03-20 17:07:20 UTC
Not sure this is the right bug, but I am wondering how I can do base containers with .oci images.

Think about 2 .oci containers created via BitBake. Both of them contain the Python runtime. One of them hello-world-1.py and another one hello-world-2.py.

At the moment I create 2 .oci containers upload them e.g. to docker hub and pull them on my target.

In order to save download bandwidth (and I guess space on my target as well) I would like to have something like "FROM ubuntu:18.04" in a Dockerfile, create a Python base container and add on top of this.

How can this be done with YP/OE/Bitbake and .oci images?
Comment 8 Bruce Ashfield 2020-03-23 16:56:16 UTC
(In reply to comment #7)
> Not sure this is the right bug, but I am wondering how I can do base
> containers with .oci images.
> 
> Think about 2 .oci containers created via BitBake. Both of them contain the
> Python runtime. One of them hello-world-1.py and another one
> hello-world-2.py.
> 
> At the moment I create 2 .oci containers upload them e.g. to docker hub and
> pull them on my target.
> 
> In order to save download bandwidth (and I guess space on my target as well)
> I would like to have something like "FROM ubuntu:18.04" in a Dockerfile,
> create a Python base container and add on top of this.
> 
> How can this be done with YP/OE/Bitbake and .oci images?

Can you clarify a couple of things ?

Do you want to create those base containers on the artifact repo (i.e. dockerhub) and then construct new containers using dockerfiles based on those uploads ?

I don't think it is that case, since there's nothing preventing you from doing that now.

Are you talking about doing this at build time, and potentially mixing yocto and non-yocto artifacts ? If so, There's no way to do this right now, since that fetching would have to be done at build time, and you'd be bringing in untested/unlicenced/unknown code and then layering your build on top of it. This has been talked about on the meta-virt list, but there's no viable way to do it now.

Or are you talking about creating a layered OCI image directly from a single bitbake build ? If so, that also isn't possible, but it is something that we can look into adding. Right now, you need two separate build and pushes to a registry to achieve this.
Comment 9 Robert Berger 2020-03-23 21:18:00 UTC
(In reply to comment #8)
> 
> Can you clarify a couple of things ?
> 
> Do you want to create those base containers on the artifact repo (i.e.
> dockerhub) and then construct new containers using dockerfiles based on
> those uploads ?
> 
> I don't think it is that case, since there's nothing preventing you from
> doing that now.
> 
> Are you talking about doing this at build time, and potentially mixing yocto
> and non-yocto artifacts ? If so, There's no way to do this right now, since
> that fetching would have to be done at build time, and you'd be bringing in
> untested/unlicenced/unknown code and then layering your build on top of it.
> This has been talked about on the meta-virt list, but there's no viable way
> to do it now.
> 
> Or are you talking about creating a layered OCI image directly from a single
> bitbake build ? If so, that also isn't possible, but it is something that we
> can look into adding. Right now, you need two separate build and pushes to a
> registry to achieve this.

I build with meta-virtualisation .oci images. I don't want to mix yocto and non-yocto artifacts.

I am also happy with 2 builds and pushes at the moment ;)

How will this work?

Let's assume I have this oci image:

https://gitlab.com/meta-layers/meta-desire/-/blob/master/dynamic-layers/virtualization-layer/recipes-core/images/app-container-image-python3-data-collector-oci.bb

which installs this:

https://gitlab.com/meta-layers/meta-desire/-/blob/master/recipes-core/images/app-container-image-python3-data-collector-common.inc

IMAGE_INSTALL += " \
	busybox \
        python3 \
        python3-localdatacollector \
"

and this one:

https://gitlab.com/meta-layers/meta-desire/-/blob/master/dynamic-layers/virtualization-layer/recipes-core/images/app-container-image-python3-mqttbrokerclient-oci.bb

which installs this:

https://gitlab.com/meta-layers/meta-desire/-/blob/master/recipes-core/images/app-container-image-python3-mqttbrokerclient-common.inc

IMAGE_INSTALL += " \
	busybox \
        python3 \
        python3-mqttbrokerclient \
	python3-mqttbrokerclient-executor \
"

I would be happy to find a way where busybox and python3 are as some "base image" in one build and the rest added on top of it in 2 more builds.

At the moment I build app-container-image-python3-data-collector-oci and push it (with skopeo) to docker hub and app-container-image-python3-mqttbrokerclient-oci as well and push it (with skopeo) to docker hub.

What's missing (in my brain I guess) is how to split it into base-python3-container + python3-localdatacollector and base-python3-container + python3-mqttbrokerclient and python3-mqttbrokerclient-executor.

Is this how it can be done?

Do I need to manually hack oci meta data to make it work?

What is the workflow you would suggest?
Comment 10 kouheiaoki 2021-05-13 20:42:08 UTC
Dear Robert Berger.

At last, it seems some errors on the documents, I list up them below.
And I sent you a merge request on gitlab.
Could you review and merge it?

> https://gitlab.com/meta-layers/meta-virtualization-tests/-/blob/master/01.00-build-docker/docker-build.md

*1) [12] [13] [14] is broken link
*2) Command after "In the container:" in "Build." section's needs "bitbake"
Comment 11 kouheiaoki 2021-05-13 23:45:33 UTC
(In reply to comment #10)
I'm sorry, This comment is my mistake...
Please delete or ignore this...
Comment 12 Bruce Ashfield 2022-09-29 15:22:44 UTC
Still valid, moving to 4.2 so it can be coordinated with binary distro efforts.
Comment 13 Randy MacLeod 2025-04-24 18:54:45 UTC
Bulk move as requesting during YP BB meeting.
Comment 14 Bruce Ashfield 2026-03-19 16:14:44 UTC
meta-virt has this implemented now.

mult-layer oci containers
vdkr/vpdm/vdcontainer

Details in the layer itself.