Bug 9937

Based on bug #9449 I would propose to increase the importance of this feature and to take it in consideration for implementation, instead of leaving it parked for "future" releases.

Here are some numbers that back the request.

Building an Ostro image with support for swupd took multiple hours, even if the entire content of the image was available already as part of the SSTATE.

This was caused mostly by pseudo and the way it handles concurrent accesse to its internal database.

After using a hacked version of pseudo that doesn't hijack ownership, the very same build process, run inside a container, took about 20 minutes.

The improvement is about 2 orders of magnitude, on a time scale that is very relevant for human beings.

Sorry, 1 order of magnitude.
Still quite dramatic, though.

Didn't this issue get fixed by some of the changes to pseudo internals?

AFAIK no, it's still open.
At least, we could reproduce it about 1 month ago with the latest Ostro.
Is there any specific patch/revision we should aim for?

As generic comment: as long as the issue persist (pseudo queuing parallel access to the filesystem because of the DB handling), it will not only affect swupd, but any attempt of generating updates for YP, because of the intrinsic nature of the problem.

Sadly, even if you allow bitbake to run as root you have several problems:

a) recipes need to create their own users which they do with adduser/deluser. In the root model, this would have to be reflected in the underlying user accounts, meaning that a given system had to be dedicated to a given build.

b) You can't really obtain a "clean" set of base user/group files since most distros do already have some underlying users which could conflict with those setup by the build.

c) You'd have to be very sure that the filesystem you were building on supported things like xattr to match the requirements of the build exactly.

Whilst many of these are surmountable by using a specifically developed VM image, it would require very precise setup from the user with a much greater potential for errors. It would go against the ethos of not requiring root access and it would introduce two different ways of running a build, meaning we'd have to double our QA test plan to test everything in both scenarios.

For all those reasons, I think we'll have to find a way to fix pseudo to perform better, or swupd to have some pseudo knowledge to help it avoid the performance issues (run different commands in separate pseudo instances?) and that we will not be supporting builds as root, if for no other reason that the impact on the testing matrix.

I won't reopen this, because we clearly have different perspective about what would be more user-friendly and the discussion would not be technical anymore.

But I personally think the container based solution would be preferable.

Example:

It might be by design, yet I find it confusing, that when I build -native packages, they are *not* built with a -native bootstrapped compiler.
Instead they are built with the host compiler, leading to any sort of different outcome when 2 people have different host distros.

The container approach would put everyone on the same ground and make much more effective use of the SSTATE for -native component.

Right now, if I happen to have a host distro which is different from the one used to populate a shared SSTATE, I'm left out in the cold and I have to rebuild all the -native components that are affected.

(In reply to comment #7)

> It might be by design, yet I find it confusing, that when I build -native
> packages, they are *not* built with a -native bootstrapped compiler.
> Instead they are built with the host compiler, leading to any sort of
> different outcome when 2 people have different host distros.
> 
> The container approach would put everyone on the same ground and make much
> more effective use of the SSTATE for -native component.
> 
> Right now, if I happen to have a host distro which is different from the one
> used to populate a shared SSTATE, I'm left out in the cold and I have to
> rebuild all the -native components that are affected.

We do have the concept of "uninative" which does in fact allow one sstate feed to feed native sstate objects to all users. This problem you describe of sstate reuse for native objects therefore doesn't exist any more.

> We do have the concept of "uninative" which does in fact allow one sstate feed to 
> feed native sstate objects to all users. This problem you describe of sstate 
> reuse for native objects therefore doesn't exist any more.

Yes, I'm aware of uninative, however it doesn't seem to work on my setup.
My host distro is SUSE 42.1 (yes, I do get the warning that it's not officially supported) and other people in my team use typically RedHat.

According to my understanding, even if the sstate claims that native files built on different hosts are compatible, because they use the same uninative libraries, in place of those coming with the host compiler, yet the compilers used are different (because different distros ship different compilers, usually) and they do produce different binary files.

A recent example of this, I believe, is how OTMPFILE is handled.

On my SUSE42.1, running code from ostree on the host was generating an error saying that OTMPFILE was not supported. Running basically the same command in a recipe was instead showing that OTMPFILE was supported, but it failed halfway.

On fedora, the detection succeeded consistently both in the host and in the pseudo environment.