Bug 11138

Still worth doing when we have the people available. -- YP bug review.

Bulk move from 4.99 or 0.00 to 5.99

The mechanism comment #0 asked for does exist now. What is still missing
is anything that runs it after a user builds something inside the eSDK,
which was the first of the two options in comment #0.

THE MECHANISM

    bitbake build-sysroots -c build_native_sysroot
    bitbake build-sysroots -c build_target_sysroot

meta/recipes-core/meta/build-sysroots.bb re-assembles the old-style
shared sysroots, ${STAGING_DIR}/${BUILD_ARCH} and
${STAGING_DIR}/${MACHINE}, out of everything in the components directory
that matches the current MACHINE.

It is not gated by any task signature, which matters here. The recipe
deltasks populate_sysroot and inherits nopackages, so it has no sstate
task of its own, and both do_build_*_sysroot tasks are nostamp with
cleandirs. It can therefore be re-run at any point in the life of a
build directory, and it rebuilds the shared sysroot from scratch out of
whatever ${SSTATE_MANIFESTS}/manifest-*.populate_sysroot files happen to
be present. There is no "already done" state for it to short-circuit on.

It arrived in the same commit that created this bug's subject,
809746f56df4 ("Switch to Recipe Specific Sysroots", 2016-12-07):

    In order to work with eSDK we need a way to build something that
    looks like the old style sysroot. "bitbake build-sysroots" will
    construct such a sysroot based on everything in the components
    directory that matches the current MACHINE. [...] I'd suggest not
    relying on this for anything other than transitional purposes
    though. To see XXX in that sysroot, you'd have to have built that
    in a previous bitbake invocation.

Ten years on it is load-bearing rather than transitional:
meta/files/ext-sdk-prepare.py runs it at eSDK install time, devtool
sdk-install and devtool ide-sdk both call it, and
oeqa/selftest/cases/meta_ide.py and oeqa/selftest/cases/package.py
depend on it.

WHAT IS ACTUALLY BROKEN

Nothing re-runs it when the user builds something inside the eSDK, so
the shared sysroot the SDK's PATH points at silently goes stale.

Measured on a qemux86-64 eSDK built from master at dbeef84da24f. The
only patches on top of that touch meta/lib/oeqa/sdk/cases, nothing on
this path.

1. Install the eSDK. ext-sdk-prepare.py runs build-sysroots, and 142
   native components land in tmp/sysroots/x86_64.

2. Do an ordinary piece of eSDK work:

       devtool modify cpio
       devtool build cpio

3. Eight more native components now have a completed
   do_populate_sysroot - autoconf, automake, dwarfsrcfiles, file, lua,
   lzlib, popt, rpm - and not one of them is in the shared sysroot.
   Comparing the native and cross populate_sysroot manifests in
   tmp/sstate-control against the manifests build-sysroots hardlinks
   into the shared sysroot gives 150 built, 142 present.

   Taking lua as the example:

       tmp/sysroots-components/x86_64/lua-native/usr/bin/lua   present
       tmp/stamps/x86_64-linux/lua-native/
           5.5.0.do_populate_sysroot_setscene.<hash>           present
       tmp/sysroots/x86_64/usr/bin/lua                         absent

   tmp/sysroots/x86_64/usr/bin is on the eSDK's PATH, so as far as the
   user is concerned lua is simply not there.

DEVTOOL SDK-INSTALL DOES NOT RECOVER FROM THIS, IT CONFIRMS IT

scripts/lib/devtool/sdk.py does call build-sysroots, but both calls are
inside the "if install_recipes:" block (sdk.py:278 and 303-305 as of
a2425b3e36e0, the most recent commit to touch the file).
install_recipes only collects recipes for which checkstamp() returns
False, and checkstamp() is satisfied by a do_populate_sysroot or a
do_populate_sysroot_setscene stamp.

So for exactly the eight recipes above, "devtool sdk-install lua-native"
logs "lua-native is already installed", returns 0, and skips the
re-assembly, leaving lua as absent from PATH as it was before.

The stamp is being used as a proxy for "the file is in the shared
sysroot" when what it actually means is "the component is in
sysroots-components". Those two diverge the moment anything builds in
the eSDK without a build-sysroots run behind it.

HOW THIS LOOKS FROM oeqa, AND WHY IT HAS BEEN EASY TO MISS

The SDK test suite cannot currently see any of this.
ensure_host_package() in meta/lib/oeqa/sdk/case.py decides whether a
case may run by looking the package up in the SDK manifest. For an eSDK
that manifest is the sstate dependency closure, so it answers yes for
exactly the components described above: the host manifest of the eSDK
measured here lists lua-native, autoconf-native, file-native and
rpm-native, all four of which are absent from tmp/sysroots/x86_64.

The case then runs the command name, and the build host supplies it.
For perl and python3 the host always has one, because both are in
HOSTTOOLS and a build aborts before it starts if a HOSTTOOLS entry is
missing. So the case passes, on the host's interpreter, having
established nothing about the SDK.

That is already on record in public. Bug 16284 comment #0 is an
autobuilder do_testsdkext run on qemuriscv64 reporting

    RESULTS - perl.PerlTest.test_perl: PASSED (0.03s)

Nothing in that run established that the perl it exercised came out of
the eSDK rather than off the x86-64 build host.

Checking where the command resolves - requiring the path returned by
"command -v" to sit under the SDK install directory - turns that pass
into a skip that names the condition: "perl is /usr/bin/perl, outside
the SDK", or "the SDK has no lua" where the host has no copy either. I
have such a change queued for oeqa and will post it separately. It is a
detector rather than a fix and repopulates nothing; it is worth
mentioning here only because it gives this bug a reproducible signal,
instead of requiring someone to compare sstate manifests against the
shared sysroot by hand.

Not oversold: a resolution check says nothing about whether the command
works once found, so it does not cover 15854, and it does not help
15850, where sdk-install fails outright.

ONE THING TO KNOW BEFORE REPRODUCING

When ensure_host_package() does fall through to devtool sdk-install and
that install has real work to do, sdk-install reaches its build-sysroots
calls and re-assembles the whole shared sysroot rather than just the
piece that was asked for. Every component stale at that moment is
repaired as a side effect. A single case that trips the fallback can
therefore un-stale the tree for every case that runs after it, so a
testsdkext result can depend on test order. That follows from the code
rather than from a measurement.

TWO SHAPES OF FIX

a) Move the two build-sysroots calls out of the "if install_recipes:"
   block in sdk_install(). One level of indentation. It makes
   sdk-install honest, but it only helps people who reach for
   sdk-install in the first place.

b) Re-assemble after any eSDK build. That is comment #0's first option
   and the one that actually closes this bug; devtool build and devtool
   modify inside an eSDK would need the same hook.

Because the tasks are nostamp with cleandirs, running them is a full
re-hardlink of the shared sysroot rather than a cheap no-op, so (b) has
a cost worth measuring before it becomes unconditional.

RELATED

15850 (kmod test doesn't work in eSDK) and 15854 (Meson cannot be used
in minimal eSDK) are both "devtool sdk-install did not put the tool
where the eSDK can use it", from different causes: a recipe with no
do_populate_sysroot, and a recipe whose post-relocate step never runs.
This bug is the general case of those two.

THE QUESTION THIS RAISES

build-sysroots is now the answer to comment #0, but its author
explicitly said not to rely on it. Is it blessed as the supported
mechanism - documented for eSDK users, and re-run automatically after
eSDK builds - or is it replaced by something that keeps the shared
sysroot in step with the components directory as a matter of course?
Fixing sdk-install's indentation is worth doing either way, but it does
not answer that question.