Bug 15104

I added zbus recipes in my openembedded-core fork if someone wants to try it: https://gitlab.com/fmartinsons/openembedded-core/-/tree/zbus_recipe_example/meta/recipes-extended/rust-example

Frederic, I'll try to take a look in 2 weeks after a short vacation.

Great, don't hesitate to ping me if I can help you to pin point the issue.

I'm still not caught up from after vacation but I was able to clone your oe-core clone, checkout the right branch and reproduce the problem.

I even straced bitbake:  strace -f -o bb-zbus.log bitbake zbus
and spent some fun time finding where cargo is being invoked, trying to see if I could quickly see what was going wrong. That's likely not a productive path so I've put it aside for now.

I'm starting to look at the log.do_fetch and log.do_compile and wondering if
there's any point in making the problem smaller by just writing a recipe for zvariant. Anyway more tomorrow or Friday hopefully.

Have you made any progress?

(In reply to Randy MacLeod from comment #4)
> I'm still not caught up from after vacation but I was able to clone your
> oe-core clone, checkout the right branch and reproduce the problem.
> 
> I even straced bitbake:  strace -f -o bb-zbus.log bitbake zbus
> and spent some fun time finding where cargo is being invoked, trying to see
> if I could quickly see what was going wrong. That's likely not a productive
> path so I've put it aside for now.

Maybe stracing cargo itself could be even more fun (joke aside, we may want to ping cargo maintainers if we cannot find the difference between yocto env and a regular env about this cargo patching process)

> 
> I'm starting to look at the log.do_fetch and log.do_compile and wondering if
> there's any point in making the problem smaller by just writing a recipe for
> zvariant. Anyway more tomorrow or Friday hopefully.

Don't think so since I already merged a zavariant recipe in oe (https://git.openembedded.org/openembedded-core/tree/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb) and patching path works perfectly there.

The problem is tied with cargo virtual workspace.

> 
> Have you made any progress?

Sadly not, I din't work on it since last month, I may have some time by the middle of June though.

Ok I find the issue, this is "simply" due to the absence of the Cargo.lock file in the git checkout of the zbus package (Cargo.lock is only ship when publishing on crates.io).

The patch procedure can only applied towards a Cargo.lock file.
Moreover, yocto downloads all the dependencies and do not want cargo to access the network, and that also, needs for Cargo.lock to be present.

We may want to use --frozen option to cargo (Require Cargo.lock and cache are up to date) instead of --offline during cargo build, the message generated may be more accurate e.g:

"""
Log data follows:
| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from /home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/zbus/3.11.0-r0/rust-targets/
| NOTE: cargo = /home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/zbus/3.11.0-r0/recipe-sysroot-native/usr/bin/cargo
| NOTE: cargo build -v --frozen --target x86_64-tapos-linux-gnu --release --manifest-path=/home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/zbus/3.11.0-r0/git//Cargo.toml
|     Updating git repository `https://github.com/gtk-rs/glib`
| error: failed to get `glib` as a dependency of package `zvariant v3.12.0 (/home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/zbus/3.11.0-r0/git/zvariant)`
| 
| Caused by:
|   failed to load source for dependency `glib`
| 
| Caused by:
|   Unable to update https://github.com/gtk-rs/glib?rev=c9ee583cea0
| 
| Caused by:
|   failed to fetch into: /home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/zbus/3.11.0-r0/cargo_home/git/db/glib-928cf7b282977403
| 
| Caused by:
|   attempting to update a git repository, but --frozen was specified
| WARNING: exit code 101 from a shell command.
| ERROR: Execution of '/home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/zbus/3.11.0-r0/temp/run.do_compile.227938' failed with exit code 101
"""

This is not very clear that the error is coming from the absence of Cargo.lock so I'll ask in the patch comment if we may handle that in bitbake class (with an explicit message stating that yocto requires Cargo.lock to be present).

Long story short, with the Cargo.lock file present , patching git source to be local checkout works perfectly.

I'm starting to write a patch to check for Cargo.lock present when we patch some git url (by the use  --frozen).

I came accross a drawback that I didn't foreseen, in case of patched
git url (those set up by cargo_common_do_patch_path), the Cargo.lock file must be modified
to remove source entry , e.g. something like in Cargo.lock

[[package]]
name = "crate-from-git"
version = "1.0.0"
source = "git+ssh://git@repo.com/mycrate.git?branch=main#1f9945fcab72499dcf17ce4c696c00b02e1a7365"
dependencies = [
 "chrono",
 "log",
 "mockall",
 "serde"
]

with that in "CARGO_HOME/config"

[patch."ssh://git@repo.com/mycrate.gitt"]
tap-zbus = { path = "/my/checkout/path/in/yocto" }

became the following in Cargo.lock

[[package]]
name = "crate-from-git"
version = "1.0.0"
dependencies = [
 "chrono",
 "log",
 "mockall",
 "serde"
]

Of course, using --frozen prevent that to make it work.

We may have used a post configure step (just after cargo_common_do_patch_path) which would have issue "cargo update --offline -p crate-from-git" but this 
doesn't work also since for this to work, the local cargo cache must have done
a git checkout by itself and this is not how we do it in yocto (we
fetch by ourself).

The only solution I found for the moment is to clean the Cargo.lock 
by myself, before frozen it for build steps...

There is also the question of rust-hello-world recipe which doesn't have 
a Cargo.lock so if we use --frozen flag, this recipe will not be buildable
anymore.

Patch series submitted to check Cargo.lock existence and remove rust-hello-world recipes: https://patchwork.yoctoproject.org/project/oe-core/list/?series=15271

Moving to 4.3-M3 rather than the "some time in the next year or two", 4.99.
Change if you really meant 4.99 Frederic.

No problem Randy, it has been merged in the master few days ago.
I'll close this ticket then.

Thanks Frédéric,

If you can, it's useful to include a link to the commit like this:

Fixed by:
https://git.openembedded.org/openembedded-core/commit/?id=b80f756dd480fc92f58d7e10105d3a2427a32795

and rust-hello-world removed by:
https://git.openembedded.org/openembedded-core/log/?qt=range&q=8baaf94d200f5355791ecd980727698b1ab0e539

Understood, thanks Randy, I'll think about it next time.