10867 2017-01-01 21:38:39 +0000 bitbake with external layer fetcher fails to allow SRCPV to work 2024-06-13 15:07:57 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED OBSOLETE https://github.com/meta-rust/meta-rust/issues/136 Medium normal Future 1 cardoe unassigned markus.lehtonen poky.bs.watcher poky.watcher randy.macleod richard.purdie yoctobugzilla --- --- No (bug/feature does not impact docs) oldest_to_newest 69424 0 cardoe 2017-01-01 21:38:39 +0000 I've got a custom fetcher for Rust packages (which are called crates). It is very similar to the NPM fetcher. If I attempt to use SRCPV then bitbake fails when it hits one of my crate:// URLs. Which works just fine otherwise. It seems like bitbake is not fully parsing my recipe to load my fetcher before it resolves SRCPV. I am not sure what version of BitBake I'm using since I am using Yocto master and bitbake --version reports 1.32.0 which is older than any of the drop down options above. The fetcher code is here: https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py The bbclass that is inherited to load that is here: https://github.com/meta-rust/meta-rust/blob/master/classes/crate-fetch.bbclass The bbclass that is inherited by recipes and in turn inherits the above bbclass is here: https://github.com/meta-rust/meta-rust/blob/master/classes/cargo.bbclass The recipe that fails is here: https://github.com/cardoe/meta-rust/blob/rustfmt-git/recipes-example/rustfmt/rustfmt_git.bb The error is as follows: ERROR: ExpansionError during parsing /var/build/poky/meta-rust/recipes-example/rustfmt/rustfmt_git.bb Traceback (most recent call last): File "/var/build/poky/bitbake/lib/bb/data_smart.py", line 412, in DataSmart.expandWithRefs(s="${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}.${SRCPV}", varname='PV'): try: > s = __expand_var_regexp__.sub(varparse.var_sub, s) try: File "/var/build/poky/bitbake/lib/bb/data_smart.py", line 111, in VariableParse.var_sub(match=<_sre.SRE_Match object; span=(77, 85), match='${SRCPV}'>): else: > var = self.d.getVarFlag(key, "_content") self.references.add(key) File "/var/build/poky/bitbake/lib/bb/data_smart.py", line 773, in DataSmart.getVarFlag(var='SRCPV', flag='_content', expand=True, noweakdefault=False, parsing=False): cachename = var + "[" + flag + "]" > value = self.expand(value, cachename) File "/var/build/poky/bitbake/lib/bb/data_smart.py", line 436, in DataSmart.expand(s='${@bb.fetch2.get_srcrev(d)}', varname='SRCPV'): def expand(self, s, varname = None): > return self.expandWithRefs(s, varname).value File "/var/build/poky/bitbake/lib/bb/data_smart.py", line 426, in DataSmart.expandWithRefs(s='${@bb.fetch2.get_srcrev(d)}', varname='SRCPV'): except Exception as exc: > raise ExpansionError(varname, s, exc) from exc bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception NoMethodError: Could not find a fetcher which supports the URL: 'crate://crates.io/aho-corasick/0.5.3' 70897 1 markus.lehtonen 2017-02-23 12:05:32 +0000 I think what you're trying to do (i.e. implementing custom fetchers in layers) is currently not supported by bitbake. Your fetcher has not yet been added to bb.fetch2 at the time bitbake parses the recipe. So, I think we should turn this bug into an enhancement/feature request for supporting custom fetchers. Another possibility would be for you to just add your custom fetcher directly into bitbake and send it for review on the bitbake-devel mailing list. I don't know what is the general response (from RP) towards adding new fetchers, though. CC'ing RP - he might have some comments/insight. 73077 2 markus.lehtonen 2017-05-05 10:24:25 +0000 Have you tried using a global inherit (that is, e.g. in local.conf)? That might do the trick. Setting as needinfo and assigning back to reporter for comments. 80491 3 yoctobugzilla 2018-05-04 02:55:56 +0000 Adding crate-fetch to INHERIT in local.conf does not solve this issue. This issue also prevents the devtool from being useful for these recipes regardless of SRCPV parsing 91877 4 richard.purdie 2021-11-02 21:52:33 +0000 The challenge is that the extra module code is added too late in the parsing process for bitbake to see the fetcher module at the time SRCPV is parsed, which is very early in the parsing process. We'd hoped that INHERIT would solve that but it still isn't early enough to avoid issues with class being visible for SRCPV. I'm still not sure how to try and address this for external fetcher classes unfortunately. 93430 5 richard.purdie 2022-06-09 09:48:07 +0000 Crate support eventually moved into bitbake which resolved this. Before that, our best known method for handling this was in the code removed here: https://git.yoctoproject.org/poky/commit/?id=2a5dcb0662054c1d6bbe5ffe1c48fee10be30bd0 93431 6 richard.purdie 2022-06-09 09:50:15 +0000 If we do ever want to fix this we'd likely need to add some kind of special import mechanism for fetchers. Having the key fetchers in bitbake is probably desirable just to encourage standardisation so this isn't hugely pressing now crates are fixed and working so moving to a future bug. 99190 7 richard.purdie 2024-06-13 15:07:57 +0000 The crate code moved to the fetcher which resolved this specific issue. We're not sure we want the complexity added to the fetcher code to support autorev so for now we don't plan to do this and the original issue was resolved.