Bug 5470 (tinfoil2)

This will apply to any tool that uses the tinfoil module. Unfortunately I suspect this means tinfoil will need to be reworked.

(In reply to comment #1)
> This will apply to any tool that uses the tinfoil module. Unfortunately I
> suspect this means tinfoil will need to be reworked.

Paul, do you have a design/plan to overcome this? devtool is also affected by tinfoil. One approach would be to consider the app name (bitbake, bitbake-layers, devtool, etc) and just make the bitbake.lock check if app == bitbake.

Actually I've got more than a design - I have an initial version in a contrib branch here:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=paule/tinfoil2

It still has a lot of rough edges; at the moment I'm mostly waiting for responses on an RFC regarding a proper API for tinfoil before proceeding:

http://lists.openembedded.org/pipermail/openembedded-core/2015-June/105804.html

When this first came up I did consider workarounds similar to what you are describing, but given the way bitbake works there's not currently a safe way for two servers to access the same set of cache files without a bunch more locking in place.

FYI work on this has stalled for now - it's turned out to be significantly more complicated than I'd anticipated, not only because there were more places that needed changing, but I also discovered a number of problems with memory resident mode that need fixing at the same time in order to make this of any use.

I have sent the patches I have up to this point that are independent of the tinfoil refactoring itself, and these have now been merged. I've rebased the paule/tinfoil2 branch on poky-contrib on top of latest master as well.

I've rebased the paule/tinfoil2 branch in poky-contrib on top of master and fixed it again so that at least the devtool, recipetool and bblayers oe-selftest tests pass. There's still quite a lot of cleanup and additional testing required though, and memres still needs work - e.g. I'm not actually witnessing any noticeable performance gain from using it at the moment.

I guess that the performance gains of using memres over regular setenv not only need to be noticeable but measurable. Leo, it would be nice to identify which performance parameters can be used to measure the improvements of using memres.

Waiting on Paul to provide a rebased-to-master branch of tinfoil2.

I have rebased the branch on top of master and fixed a bunch of issues today - the basics are at least working. I'll keep poking at it.

assigning back to Anibal since Paul has already provided a rebased branch. Leo is pending to free up Anibal's workload to see if this can be pulled in for 2.3 M1.

Note that I am still working on this in the mean time.

OK, I have just pushed another branch with a whole bunch of issues fixed. I've also written the following wiki page with some background and the current status:

https://wiki.yoctoproject.org/wiki/Tinfoil2

(In reply to comment #11)
> OK, I have just pushed another branch with a whole bunch of issues fixed.
> I've also written the following wiki page with some background and the
> current status:
> 
> https://wiki.yoctoproject.org/wiki/Tinfoil2

okay Paul, is this good now for taking over?

This bug was fixed by Paul with recently tinfoil2 changes hash [1],

We need to add more extensive tenting to memres, i'll setup an AB to make periodic runs of memres and discover more bugs.


[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=358befb76ede4b111a9ee913ae2586ea30dc23b7

Great Anibal, I guess it starts with a buildset that does oe-selftest from a memres environment.

BUILD: 2.3.m3.rc1: master:415b72ffcbd26e5f3664370d8b2a9b8105fb6342
ENVIRONMENT: genericx86


Steps to reproduce:
1. Build an image: core-image-sato
2. Wait until it finishes;
3. run "bitbake-layers show_layers"

I receive this:
scar@oscar-ThinkPad-T540p:~/layers/build$ bitbake-layers show-layers
layer                 path                                      priority
==========================================================================
meta                  /home/oscar/layers/meta                   5
meta-poky             /home/oscar/layers/meta-poky              5
meta-yocto-bsp        /home/oscar/layers/meta-yocto-bsp         5

(In reply to comment #15)
> BUILD: 2.3.m3.rc1: master:415b72ffcbd26e5f3664370d8b2a9b8105fb6342
> ENVIRONMENT: genericx86
> 
> 
> Steps to reproduce:
> 1. Build an image: core-image-sato
> 2. Wait until it finishes;
> 3. run "bitbake-layers show_layers"

Your steps didn't mention how you initialised your environment, did you use oe-init-build-env or oe-init-build-env-memres?

Initialize environment with oe-init-build-env.

(In reply to comment #17)
> Initialize environment with oe-init-build-env.

Hi Oscar,

You need to use oe-init-build-env-memres because without it memory resident mode isn't activated.

Please run again your tests using memres script.

Anibal

Created attachment 3677 [details]
bitbake-layers show-layers

BUILD: 2.3.m3.rc1: master:415b72ffcbd26e5f3664370d8b2a9b8105fb6342
ENVIRONMENT: genericx86


Steps to reproduce:
1. Initialize environment with oe-init-build-env-memres
2. Build an image: core-image-sato
3. Wait until it finishes;
4. I run "bitbake-layers show-layers"

(In reply to comment #19)
> Created attachment 3677 [details]
> bitbake-layers show-layers

Good, thanks!.