This error happened on the autobuilder: DEBUG: Executing shell function do_shared_workdir cp: cannot stat 'include/generated/utsrelease.h.tmp': No such file or directory WARNING: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x32/build/build/tmp/work/qemux86_64-poky-linux-gnux32/linux-yocto/3.14.29+gitAUTOINC+6eddbf4787_f65678ef48-r0/temp/run.do_shared_workdir.30276:1 exit 1 from cp -fR include/generated/asm-offsets.h include/generated/autoconf.h include/generated/bounds.h include/generated/compile.h include/generated/uapi include/generated/utsrelease.h $kerneldir/include/generated/ ERROR: Function failed: do_shared_workdir (log file is located at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x32/build/build/tmp/work/qemux86_64-poky-linux-gnux32/linux-yocto/3.14.29+gitAUTOINC+6eddbf4787_f65678ef48-r0/temp/log.do_shared_workdir.30276) It was in a MUT run but there wasn't anything that should have impacted the kernel in the branch over master.
We also have a race on kernel.release being reported on the mailing list.
I'm out of the office until the 17th, so if anyone has ideas, feel free to file them here (or take a crack at this). My suggestion on the mailing list was to save these files at the end of do_compile() and then finish the work in do_shared_workdir() .. but then we ask why we still need do_shared_workdir() at all. Removing it would mean that dependencies needed to be updated again, or we'd keep it around for a dummy dependency. Alternatively, can we drop a lock file down in do_compile, and have it as a dependency for other tasks ? We only then remove it in do_shared_workdir() ? I'm not sure on the bitbake internals to support a scheme like that.
Here is my attempt to fix this by getting rid of do_shared_workdir: https://lists.yoctoproject.org/pipermail/yocto/2015-March/024043.html I haven't noticed any performance degradation with this change. Here are results of 'time bitbake core-image-minimal' running on the same machine before and after the change: before: real 30m32.259s user 135m54.912s sys 24m48.292s after: real 30m23.198s user 134m52.312s sys 24m9.100s
As I stated on the list .. I do not want to juggle these task definitions again .. unless there's absolutely no other solution. They've been out there now, and have been adopted already. We are now forcing more churn. (In reply to comment #3) > Here is my attempt to fix this by getting rid of do_shared_workdir: > https://lists.yoctoproject.org/pipermail/yocto/2015-March/024043.html > > I haven't noticed any performance degradation with this change. Here are > results of 'time bitbake core-image-minimal' running on the same machine > before and after the change: > > before: > real 30m32.259s > user 135m54.912s > sys 24m48.292s > > after: > real 30m23.198s > user 134m52.312s > sys 24m9.100s
(In reply to comment #2) > My suggestion on the mailing list was to save these files at the end > of do_compile() and then finish the work in do_shared_workdir() .. but > then we ask why we still need do_shared_workdir() at all. Removing it > would mean that dependencies needed to be updated again, or we'd keep > it around for a dummy dependency. > Can you elaborate on this? Which dependencies need to be updated? do_shared_workdir runs in parallel with other tasks, which is the reason for race. My idea was that making this functionality part of do_compile would fix the race. Locking or copying files to yet another place doesn't look better than removing do_shared_workdir to me. Moreover, locking has the same effect as my change, just more complex and fragile.
(In reply to comment #5) > (In reply to comment #2) > > My suggestion on the mailing list was to save these files at the end > > of do_compile() and then finish the work in do_shared_workdir() .. but > > then we ask why we still need do_shared_workdir() at all. Removing it > > would mean that dependencies needed to be updated again, or we'd keep > > it around for a dummy dependency. > > > Can you elaborate on this? Which dependencies need to be updated? This has been in tree for over a month now. You can only see the recipes in oe-core, and hence you are breaking a whole set of recipes that have this dependency. So no. Whatever we come up with, should not remove that task definition again. I always get the email when we break other kernel workflows, and since there are some light weight ways to not subject more changes on those layers, I see no strong reason to do it. > > do_shared_workdir runs in parallel with other tasks, which is the reason for > race. My idea was that making this functionality part of do_compile would > fix the race. And the original idea of it was to make it more lightweight, and something that can run without a full compile happening. The granularity had some intentions. We are breaking those intentions be re-combining it. > > Locking or copying files to yet another place doesn't look better than > removing do_shared_workdir to me. Moreover, locking has the same effect as > my change, just more complex and fragile. Not really. Locking is common, and I wouldn't call it fragile .. unless you'd like to tag the entire code base this way :)
I forgot to add, the route that I was taking was either locking (and I was still looking at the proper semantics), or one way that did seem to work was an empty shared_workdir task, with the bulk of the code put back into do_compile. That kept the dependency the same, but avoided the race on the files. I was also looking to patching the kernel build, but that is more of a long term project, since I'll need to get those changes into the mainline kernel. So maybe to get this into the code base, break as few dependent layers as possible, having that dummy task (with a big flashing comment) is the right thing. It has the side benefit of being able to incremental move blocks of code back to it, or transfer more blobs to the artifacts dir, without needing to re-compile.
Thank you for the explanations! Now it's clear to me why I didn't see those dependencies :) (In reply to comment #7) > I forgot to add, the route that I was taking was either locking > (and I was still looking at the proper semantics), or one way that > did seem to work was an empty shared_workdir task, with the bulk > of the code put back into do_compile. > I like second approach better. It's almost what I did, but much simpler as doesn't require removing do_shared_workdir. > That kept the dependency the same, but avoided the race on the > files. > > I was also looking to patching the kernel build, but that is more > of a long term project, since I'll need to get those changes into > the mainline kernel. > > So maybe to get this into the code base, break as few dependent > layers as possible, having that dummy task (with a big flashing > comment) is the right thing. It has the side benefit of being > able to incremental move blocks of code back to it, or transfer > more blobs to the artifacts dir, without needing to re-compile. Well, we can move only copying of include/generated to do_compile if you want. This is a minimal change to fix this bug. Do you want me to update my patch and re-send it to openembedded-core ml?
(In reply to comment #8) > Thank you for the explanations! Now it's clear to me why I didn't see those > dependencies :) > > (In reply to comment #7) > > I forgot to add, the route that I was taking was either locking > > (and I was still looking at the proper semantics), or one way that > > did seem to work was an empty shared_workdir task, with the bulk > > of the code put back into do_compile. > > > > I like second approach better. It's almost > what I did, but much simpler as doesn't require removing do_shared_workdir. > > > That kept the dependency the same, but avoided the race on the > > files. > > > > I was also looking to patching the kernel build, but that is more > > of a long term project, since I'll need to get those changes into > > the mainline kernel. > > > > So maybe to get this into the code base, break as few dependent > > layers as possible, having that dummy task (with a big flashing > > comment) is the right thing. It has the side benefit of being > > able to incremental move blocks of code back to it, or transfer > > more blobs to the artifacts dir, without needing to re-compile. > Well, we can move only copying of include/generated to do_compile if you > want. This is a minimal change to fix this bug. > > Do you want me to update my patch and re-send it to openembedded-core ml? Absolutely. You are in the mode and familiar with the code, so lets strike while the iron is hot!
Great! Can you look at this thread please: https://lists.yoctoproject.org/pipermail/poky/2015-February/010015.html It looks like related issue to me. Should we also move copying of kernel.release to kernel_do_compile or it's better to do it in separate patch?
(In reply to comment #10) > Great! > > Can you look at this thread please: > https://lists.yoctoproject.org/pipermail/poky/2015-February/010015.html At the time that thread popped up, I did think that it was the same root cause. > > It looks like related issue to me. Should we also move copying of > kernel.release to kernel_do_compile or it's better to do it in separate > patch? I'd do it in the same patch. In the end, I'm ok with moving everything that is currently in the do_shared_workdir() into compile, as long as it is either directly called or the old task stays around .. we'll have our flexibility and close the race for now.
done: http://lists.openembedded.org/pipermail/openembedded-core/2015-March/102636.html Just out of curiosity. Bruce, can you point me out to oe-core recipes that use do_shared_workdir? I looked almost everywhere I could and didn't find anything except of what my first patch was fixing.
(In reply to comment #13) > done: > http://lists.openembedded.org/pipermail/openembedded-core/2015-March/102636. > html > > Just out of curiosity. Bruce, can you point me out to oe-core recipes that > use do_shared_workdir? I looked almost everywhere I could and didn't find > anything except of what my first patch was fixing. The layers are in some about to release products, and also from some queries that I received when the dependency was first changed (I can't find them now, but will look again). Admittedly, the breakage wouldn't be vast, I'm just trying to minimize any disruptions, and then revisit this early in the next cycle.
Thank you for the explanations. It definitely makes sense to be careful here.
Bruce, my patch has not been accepted yet. Anything wrong with it? Anything is still required from me here?
This may have slipped through the cracks. I thought it would be picked up by the normal master under test sweep, since you had replied to my comments on the first version. I'll pick it up and re-send it shortly.
Still don't see my patch merged. Bruce, let me know if I can do anything about it. Thanks.
I have the patch queued here (and have been testing with it), but with the feature freeze coming into play, I elected to not send that part of the queue and instead get it into the master when it opened again and get a backport to the stable branch. Unless we really want it sooner, I'll just fire it back out again and it can land where it lands!
I think it's better to have it merged because it's scheduled for 1.8 M4 and it's just a small bugfix.
I'm on a plane today, so can't easily repost the patch. Can you add my Signed-off-by along with yours and resend ?
I've just discussed it with Richard. He said we're late for 1.8 release, so no hurry.
I decided to recent my patch. Added your Signed-off-by as you've suggested: http://lists.openembedded.org/pipermail/openembedded-core/2015-April/103735.html
Fine with me! I'm about a day away from sending my latest series, and I'll just drop this from my queue. (In reply to comment #23) > I decided to recent my patch. Added your Signed-off-by as you've suggested: > http://lists.openembedded.org/pipermail/openembedded-core/2015-April/103735. > html
The fix failed testing on autobuilder: https://autobuilder.yoctoproject.org/main/builders/nightly-rpm/builds/256/steps/Running%20Sanity%20Tests/logs/stdio I'm trying to reproduce this failure locally. Bruce, do you have any idea what could be the reason?
(In reply to comment #25) > The fix failed testing on autobuilder: > https://autobuilder.yoctoproject.org/main/builders/nightly-rpm/builds/256/ > steps/Running%20Sanity%20Tests/logs/stdio > > I'm trying to reproduce this failure locally. > > Bruce, do you have any idea what could be the reason? argh. Nothing is jumping out. Have you looked at the qa test in question to see if the kernel dependency is correct ? I'll pull it up myself and check as well.
Another build failed. this time differently: https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc-lsb/builds/288/steps/BuildImages/logs/stdio Looks very strange as task order is good in the log: NOTE: recipe linux-qoriq-3.12-r0: task do_compile: Succeeded NOTE: recipe linux-qoriq-3.12-r0: task do_shared_workdir: Started NOTE: recipe linux-qoriq-3.12-r0: task do_compile_kernelmodules: Started NOTE: recipe linux-qoriq-3.12-r0: task do_shared_workdir: Succeeded NOTE: recipe linux-qoriq-3.12-r0: task do_compile_kernelmodules: Succeeded My guess is that shared dir should be created *after* compiling modules, not before it. Bruce, does it make sense for you?
Merged to master: http://cgit.openembedded.org/openembedded-core/commit/?id=b4ec689f71fe9504559c1921c893f5e096d6b744
I had a look on both local and public AB machines for any "ERROR: Function failed: do_shared_workdir" message and didn't found any.(+ discussion with Ed on how to approach) As of this, marking as verified