Mailing list topic

https://lists.openembedded.org/g/bitbake-devel/message/13976?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2CAUTOREV%2C20%2C2%2C0%2C93674384

I've got another reproducer :
This recipe cmakehelloworld_git.bb:
----------------
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=be316b0d6cb4f744b08c9dc589a13eff"
SUMMARY = "XXX"
HOMEPAGE = "http://example.org"
SRC_URI = "git://github.com/jameskbride/cmake-hello-world.git;protocol=https;branch=master"
PV = "1.0+git"
SRCREV = "${AUTOREV}"
S = "${WORKDIR}/git"
inherit cmake
EXTRA_OECMAKE = ""
FILES:${PN}-staticdev:append = " /usr/bin/libHello.a"
----------------

And a symlink cmakehelloworld2_git.bb -> cmakehelloworld_git.bb

With these :
# clear workspace
rm -rf build/workspace/
sed -i '/workspace/d' build/conf/bblayers.conf
# devtool modify 2 AUTOREV recipes
devtool modify cmakehelloworld
devtool modify cmakehelloworld2


Result, this fail on kirkstone but not on master (Same log as reporter)

I've bisected the fix : commit 62afa02d01794376efab75623f42e7e08af08526 [0] does allow the second devtool modify to work correctly.

Sadly, this commit can't be trivially backported to kirkstone (event after paths fix, bitbake crashes on missing functions)

[0] https://git.yoctoproject.org/poky/commit/?id=62afa02d01794376efab75623f42e7e08af08526
commit 62afa02d01794376efab75623f42e7e08af08526
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Fri Aug 11 13:52:59 2023 +0100

    base/package: Move source revision information from PV to PKGV
    
    Source control information being present in PV used to be a hard requirement
    for bitbake to operate correctly. Now that hashes are a required part of task
    stamps, this requirement no longer exists.
    
    This means we can defer the hash pieces to PKGV and simplify PV.
    
    Use new bitbake fetcher API to inject the source revisions directly into the hash
    allowing removal of some horrible code from base.bbclass and avoiding any hardcoding
    about how SRCREV may or may not be used.
    
    Use that API to object the string to append to PKGV and append that directly.
    
    The user visible effect of this change is that PV will no longer have revision
    information in it and this will now be appended to PV through PKGV when the
    packages are written. Since PV is used in STAMP and WORKDIR, users will see
    small directory naming and stamp naming changes.
    
    This will mean that sstate reuse through hash equivalence where the source
    revision changes but the output does not will become possible as the sstate
    naming will become less specific and no longer contain the revision.
    
    The SRCPV variable will no longer be needed in PV and is effectively now just
    a null operation. Usage can be removed over time.
    
    (From OE-Core rev: a8e7b0f932b9ea69b3a218fca18041676c65aba0)
    
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Chris Wyse proposed a fix here : https://lists.yoctoproject.org/g/yocto/message/61639 :
> I ran into this problem as well.  It doesn't seem like SRCPV is being set
> properly before use in the externalsrc.bbclass.  I added the following line
> before bpn = d.getVar('BPN'), and now it works fine.  Unfortunately, I needed
> to create another layer that overrides the poky classes to include it in my
> build.  Here's the added line:
>
>   srcpv = d.getVar('SRCPV')

In patch format (in OE-core kirkstone branch) :

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 97d7379d9f..41f87b2ef8 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -36,6 +36,8 @@ python () {
     if externalsrcbuild and not externalsrcbuild.startswith("/"):
         bb.error("EXTERNALSRC_BUILD must be an absolute path")
 
+    srcpv = d.getVar('SRCPV')
+
     # If this is the base recipe and EXTERNALSRC is set for it or any of its
     # derivatives, then enable BB_DONT_CACHE to force the recipe to always be
     # re-parsed so that the file-checksums function for do_compile is run every

Chris Wyse's proposal does indeed fix the problem:
  srcpv = d.getVar('SRCPV')

Notes:
* srcpv is unused afterward so creating this variable is useless.
* The fix is in reading "SRCPV" : while expanding it, it does run bb.fetch2.get_srcrev(d) (setting up __BBSEENSRCREV)
* So, the problem is that on the second devtool modify, bb.fetch2.get_srcrev(d) is not called before accessing SRC_URI and trigger this FetchError:
File: '.../poky/bitbake/lib/bb/fetch2/git.py', lineno: 743, function: _latest_revision
     0739:        """
     0740:        Compute the HEAD revision for the url
     0741:        """
     0742:        if not d.getVar("__BBSEENSRCREV"):
 *** 0743:            raise bb.fetch2.FetchError("Recipe uses a floating tag/branch '%s' for repo '%s' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE)." % (ud.unresolvedrev[name], ud.host+ud.path))
     0744:
     0745:        # Ensure we mark as not cached
     0746:        bb.fetch2.get_autorev(d)
     0747:


https://git.yoctoproject.org/poky/commit/?id=62afa02d01794376efab75623f42e7e08af08526 does fix this by calling bb.fetch.get_hashvalue(d) before accessing SRC_URI.

As discussed in triage meeting:
* As the master fix is too invasive, a workaround for kirkstone is acceptable
* d.getVar('SRCPV') might have other side-effect (One side-effect being a work-around this bug)
* I'll clean up the current workaround and propose it for review.

Patch sent:
[kirkstone][PATCH] externalsrc: Ensure SRCREV is processed before accessing SRC_URI
https://lists.openembedded.org/g/openembedded-core/message/191978

Steve Sakoman wrote :
> This patch resulted in oe-seftest failures on the autobuilder:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6322
> 
> A representative log:
> https://errors.yoctoproject.org/Errors/Details/746003/

I can reproduce this problem locally with:
  oe-selftest -r devtool.DevtoolUpdateTests.test_devtool_update_recipe_local_files_subdir -T machine -T toolchain-user -T toolchain-system -j 15

The interesting part of the log:
Parsing recipes...ERROR: /home/yocon/Documents/projets/yocto/poky/build-kirkstone-st-2699726/meta-selftest/recipes-test/devtool/devtool-test-subdir.bb: Error executing a python function in <code>:

The stack trace of python calls that resulted in this exception/failure was:
File: '<code>', lineno: 10, function: <module>
     0006:__anon_725__tmp_devtoolqa8gqhqvbu_core_copy_meta_classes_package_rpm_bbclass(d)
     0007:__anon_29__tmp_devtoolqa8gqhqvbu_core_copy_meta_classes_debian_bbclass(d)
     0008:__anon_36__tmp_devtoolqa8gqhqvbu_core_copy_meta_classes_devshell_bbclass(d)
     0009:__anon_162__tmp_devtoolqa8gqhqvbu_core_copy_meta_classes_sstate_bbclass(d)
 *** 0010:__anon_149__tmp_devtoolqa8gqhqvbu_core_copy_meta_classes_externalsrc_bbclass(d)
File: '/tmp/devtoolqa8gqhqvbu/core-copy/meta/classes/externalsrc.bbclass', lineno: 66, function: __anon_149__tmp_devtoolqa8gqhqvbu_core_copy_meta_classes_externalsrc_bbclass
     0062:        else:
     0063:            d.setVar('B', '${WORKDIR}/${BPN}-${PV}')
     0064:
     0065:        # Ensure SRCREV has been processed before accessing SRC_URI
 *** 0066:        bb.fetch.get_srcrev(d)
     0067:
     0068:        local_srcuri = []
     0069:        fetch = bb.fetch2.Fetch((d.getVar('SRC_URI') or '').split(), d)
     0070:        for url in fetch.urls:
File: '/home/yocon/Documents/projets/yocto/poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 782, function: get_srcrev
     0778:        if urldata[u].method.supports_srcrev():
     0779:            scms.append(u)
     0780:
     0781:    if not scms:
 *** 0782:        raise FetchError("SRCREV was used yet no valid SCM was found in SRC_URI")
     0783:
     0784:    if len(scms) == 1 and len(urldata[scms[0]].names) == 1:
     0785:        autoinc, rev = getattr(urldata[scms[0]].method, method_name)(urldata[scms[0]], d, urldata[scms[0]].names[0])
     0786:        if len(rev) > 10:
Exception: bb.fetch2.FetchError: Fetcher failure: SRCREV was used yet no valid SCM was found in SRC_URI

Maybe I need to call get_srcrev() unconditionally, call it only if it supports srcrev()?

Bumping target milestone to 5.0 M2

Bulk move to Milestone 5.0 M3

Bulk move to 5.0 M4

Fix v2 merged https://git.openembedded.org/openembedded-core/commit/?h=kirkstone&id=f6563cca6c4bf627e904d81fbe5b0b0f2b16a107