15162 2023-07-15 14:46:06 +0000 devtool build does not cope with multiple SRC_URI in a recipe 2024-08-08 14:59:07 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain unspecified x86 Multiple RESOLVED FIXED Medium+ normal 5.1 M2 1 worryelectric jstephan chris.laplante jstephan meta.mr.watcher meta.watcher randy.macleod richard.purdie tgamblin tom.isaacson --- --- No (bug/feature does not impact docs) oldest_to_newest 95924 0 worryelectric 2023-07-15 14:46:06 +0000 For a recipe with multiple SRC_URI devtool doesn't collect all of the required sources correctly. Take for example bzip2 recipe which has the following: SRC_URI = "https://sourceware.org/pub/${BPN}/${BPN}-${PV}.tar.gz \ git://sourceware.org/git/bzip2-tests.git;name=bzip2-tests;branch=master \ file://configure.ac;subdir=${BP} \ file://Makefile.am;subdir=${BP} \ file://run-ptest \ " On running `devtool modify bzip2`, the do_fetch, do_unpack, and do_patch tasks will be run into the devtool tmpdir. The temporary result of that is that the following files exist in tmp/work/cortexa53-poky-linux/bzip2/1.0.8-r0/devtooltmp-xxxxxxxx: initial_rev oe-local-files srcsubdir stamps temp workdir |-bzip2-1.0.8 |-git |-recipe-sysroot |-recipe-sysroot-native |-source-date-epoch The contents of srcsubdir is the absolute path to tmp/work/cortexa53-poky-linux/bzip2/1.0.8-r0/devtooltmp-8idreg9n/workdir/bzip2-1.0.8. Therefore it is the bzip2-1.0.8 directory that is copied to the workspace layer, and the git directory is missed. This results in `devtool build bzip2` failing at the do_install_ptest_base task when it expects files from the git directory to be present. Is this a known limitation in devtool? 95992 1 randy.macleod 2023-07-20 14:34:59 +0000 It is a bug, the tool should either handle the case or warn the user that they need to handle the recipe manually. 96578 2 jstephan 2023-09-27 16:54:03 +0000 Hi worryelectric and Randy, I just send a fix for this https://lists.openembedded.org/g/openembedded-core/message/188335 can you check it? Julien 96581 3 randy.macleod 2023-09-28 14:36:32 +0000 Thanks Julien. 96747 4 chris.laplante 2023-10-19 15:25:28 +0000 (In reply to Julien Stephan from comment #2) > Hi worryelectric and Randy, > > I just send a fix for this > https://lists.openembedded.org/g/openembedded-core/message/188335 can you > check it? > > Julien This patch seems a bit too invasive. It just moves everything in WORKDIR (except for source-date-epoch, recipe-sysroot, recipe-sysroot-native) but I think we need a more surgical approach that actually considers whether things are part of SRC_URI. I am going to try to take a look. 96749 5 jstephan 2023-10-19 21:21:57 +0000 Hi Chris, agree with you, and as explained in this thread, this workaround is clearly not suitable as a fix: https://lists.openembedded.org/g/openembedded-core/message/188765 I didn't have time to look at it more deeply. 97789 6 jstephan 2024-01-15 14:15:10 +0000 Hi All, I've been thinking about this bug and how to reliably fix it and I would like to discuss it. The fix I sent (https://lists.openembedded.org/g/openembedded-core/message/188335) can be partially improved by using a devtool unpack tracer (see https://bugzilla.yoctoproject.org/show_bug.cgi?id=15294) but the main issue is: where secondary sources extracted in WORKDIR should be placed in devtool context? I can think of three possible solutions: 1: move them in WORKDIR (as my previous fix does it but get the correct list from the unpack tracer) 2: create a devtool specific workdir to put them. The specific workdir can be placed in build/workspace/workdir/<PN> (near sources/appends/recipes/). 3: modify bitbake itself to extract all SRC_URI's element in a subdir of WORKDIR (let's call it "sources") But each solution has its drawback: 1: we will end up with sources in 2 different places: main source and sources extracted in S in an isolated place and other sources in WORKDIR (which is not isolated and can be wiped out). Moreover, devtool can not be used to create patches for secondary sources. And we need to decide the strategy to copy secondary sources i.e what about if the package was already built and we do a devtool modify on it: should we erase old files? copy only if file doesn't exists? 2: it fits well with devtool idea to have sources in an isolated place. This can be easily implemented by setting WORKDIR in externalsrc.bbclass, BUT we will have B, T, D .. in build/workspace/workdir/<PN>, which may not be something we want.. 3: it would be much easier for devtool to handle all sources at once BUT that would require a LOT of changes in all recipes that expect sources to be extracted in WORKDIR. We will require a way to not allow recipes to extract sources in WORKDIR in the future, probably modify a lot of selftest, and this will impact a lot of external layers.. Also we need to take into consideration that devtool should (or not?) be able to produce patches for secondary sources... Maybe I am missing another strategy? 97801 7 richard.purdie 2024-01-16 15:48:02 +0000 Ultimately we may well end up moving to 3). I have tried something like it and it was extremely painful and requires changes in many places. As such it probably isn't an option right now. I was thinking of something along the lines of your original patch but using the tracer to remove the need to make assumptions about files. I think the idea here is to incrementally improve on things and that would seem in keeping with that... Quite often the solution becomes clearer when you actually try and implement it, I wouldn't push really hard on something which appears to be becoming very hard to implement. 97952 8 jstephan 2024-01-23 14:10:06 +0000 Hi Richard, all, I tried to implement the unpack tracer mechanism, but found a more easiest fix for this bug. I sent a new version of the series here: https://lists.openembedded.org/g/openembedded-core/message/194237. Let me know if that works for you as a fix. Cheers Julien 99520 9 randy.macleod 2024-08-08 14:59:07 +0000 poky commit facab170b60e521d69262e62f7d60ceb064ff3a2