| Summary: | Possible race condition during do_rootfs | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] Meta-yocto | Reporter: | Beth Flanagan <elizabeth.flanagan> |
| Component: | meta-yocto | Assignee: | Mark Hatle <mark.hatle> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | major | ||
| Priority: | High | CC: | bluelightning, dongxiao.xu, jessica.zhang, josh, poky.bs.watcher, poky.watcher, richard.purdie, sgw |
| Version: | 1.0 | ||
| Target Milestone: | 1.1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | --- | |
|
Description
Beth Flanagan
2011-03-03 16:00:33 UTC
We need to add the rpm lock file to the package generation. The concern though is that we can generate all of the rpm packages we want, we just can't generate packages during a rootfs build.. Need some further information on how to do this. pull request for a fix has been sent. The fix uses shared locks vs exclusive locks. *** Bug 867 has been marked as a duplicate of this bug. *** The issue still exists in my build.
Commit 43a2d098008eee711399b8d64594d84ae034b9bf.
1) Run "bitbake poky-image-sato-live". Build the full sato-live image until it
is successful.
2) Bump connman's PR (or some other sato recipe's PR).
3) Rebuild the sato live image by "bitbake poky-image-sato-live".
Sometimes, I meet build error like the following, however it does not happen
every time.
-------------------------------------------------------
Generating solve db for
/distro/dongxiao/build-qemux86/tmp/deploy/rpm/qemux86...
total: 1 0.000000 MB 1.424841 secs
fingerprint: 1020 0.006796 MB 0.033057 secs
install: 340 0.000000 MB 0.371773 secs
dbadd: 340 0.000000 MB 0.362746 secs
dbget: 2196 0.000000 MB 0.004278 secs
dbput: 340 1.504908 MB 0.308950 secs
readhdr: 3401 2.961280 MB 0.005603 secs
hdrload: 1700 4.389932 MB 0.007001 secs
hdrget: 57535 0.000000 MB 0.043769 secs
Generating solve db for /distro/dongxiao/build-qemux86/tmp/deploy/rpm/i586...
error: open of
/distro/dongxiao/build-qemux86/tmp/deploy/rpm/i586/connman-plugin-ethernet-0.65-r4.i586.rpm
failed: No such file or directory
rpm.real: ./rpmio_internal.h:190: fdGetOPath: Assertion `fd != ((void *)0) &&
fd->magic == 0x04463138' failed.
/distro/dongxiao/build-qemux86/tmp/work/qemux86-poky-linux/poky-image-minimal-initramfs-1.0-r0/temp/run.do_rootfs.468:
line 375: 669 Aborted rpm --dbpath /var/lib/rpm
--define='_openall_before_chroot 1' -i --replacepkgs --replacefiles
--oldpackage -D "_dbpath $pkgdir/solvedb" --justdb --noaid --nodeps --noorder
--noscripts --notriggers --noparentdirs --nolinktos --stats --ignoresize
--nosignature --nodigest -D "__dbi_txn create nofsync" $pkgdir/solvedb/manifest
ERROR: Function 'do_rootfs' failed (see
/distro/dongxiao/build-qemux86/tmp/work/qemux86-poky-linux/poky-image-minimal-initramfs-1.0-r0/temp/log.do_rootfs.468
for further information)
-------------------------------------------------------
The root cause for this issue should be, poky-image-minimal-initramfs's
do_rootfs task doesn't have dependency on connman, thus their tasks will be run
simultaneously. Poky-image-minimal-initramfs's do_rootfs will call
"rootfs_rpm_do_rootfs" --> "package_update_index_rpm", where it will update all
the packages depsolver db in ${DEPLOY_DIR_RPM}.
When the package_update_index_rpm function is handling connman's rpm package,
and at the same time, connman is removing old rpm and trying to generate a new
one (e.x, from r4 to r5), then the build error will occur, saying that it could
not find r4 version of connman-plugin-ethernet...
This fix was applied: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=c180383885e32aed83c4f8691115859966fb53be but I think the lock is taken in the wrong place. The files are not placed into tmp/deploy/ at this point but later in the sstate code. Its that later point where we need to have the lock held (and also at the point any sstate package is uninstalled. Locks (rpm.lock) are already taken in python function sstate_clean and sstate_install. But the lock is not taken in "do_rootfs" --> "rootfs_rpm_do_rootfs" --> ""package_update_index_rpm", where it operates in "${DEPLOY_DIR_RPM}".
I am in confuse that the "do_rootfs" is written in shell, and how can we add lock for it or its sub-functions?
There is a global exclusive lock in the rootfs_rpm.bbclass:
# RPM doesn't work with multiple rootfs generation at once due to collisions in the use of files
# in ${DEPLOY_DIR_RPM}. This can be removed if package_update_index_rpm can be called concurrently
do_rootfs[lockfiles] += "${DEPLOY_DIR_RPM}/rpm.lock"
This is the same lock file that should be done whenever the tmp/deploy/rpm/* is being modified.
The intention is to use an exclusive lock in the do_rootfs, but use a shared lock during RPM copying into the deploy directory. We can manipulate multiple RPMs at the same time, but we must block when doing all of the rootfs operations.
I've pushed a possible fix to poky-contrib mhatle/bernard/fixes, commit id: 81c5ab86c5c0697cdee3958f69d4ad6251083a9d Please try that and see if it corrects the issue you were seeing. Sorry I still see the same error after applied your latest fix on mhatle/bernard/fixes. Please refresh from poky-contrib mhatle/bernard/fixes and try again. The top commit: 3182ada41f06e20c325d0f63ae9b211523490a60 takes the drastic approach of locking/serializing the whole system during the RPM package creation. I'm curious if this resolves the problem or not. Sorry, I just noticed I changed the wrong variable, the right commit has now been pushed: c0edbfdd704a8fcb6f07e3b5059bec6d3b7f22ab difference is affecting "sstate-lockfile" vs "lockfile".. ONE more time, the -right- commit for head is: 8824e8130cac060e9d06881aa98f8aba7bf4002b
Sorry for any confusion. Again the key line that should be there:
+do_package_write_rpm[sstate-lockfiles] += "${DEPLOY_DIR_RPM}/rpm.lock"
I am aware of that change and is now testing against the latest fix. Sorry this bug still exists after applying the fix of the following. But it seems that the rate to reproduce the issue is lower, now I can encounter this issue within about 5~10 trials.
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 4647116..56e8308 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -777,16 +777,10 @@ python do_package_rpm () {
cmd = cmd + " --define 'debug_package %{nil}'"
cmd = cmd + " -bb " + outspecfile
- # Take a shared lock, we can write multiple packages at the same time...
- # but we need to stop the rootfs/solver from running while we do...
- lf = bb.utils.lockfile(bb.data.expand("${DEPLOY_DIR_RPM}/rpm.lock", d), True)
-
# Build the rpm package!
bb.data.setVar('BUILDSPEC', cmd + "\n", d)
bb.data.setVarFlag('BUILDSPEC', 'func', '1', d)
bb.build.exec_func('BUILDSPEC', d)
-
- bb.utils.unlockfile(lf)
}
python () {
@@ -803,6 +797,7 @@ SSTATETASKS += "do_package_write_rpm"
do_package_write_rpm[sstate-name] = "deploy-rpm"
do_package_write_rpm[sstate-inputdirs] = "${PKGWRITEDIRRPM}"
do_package_write_rpm[sstate-outputdirs] = "${DEPLOY_DIR_RPM}"
+do_package_write_rpm[sstate-lockfiles] += "${DEPLOY_DIR_RPM}/rpm.lock"
python do_package_write_rpm_setscene () {
sstate_setscene(d)
I've noticed it should be "sstate-lockfile", not sstate-lockfiles. I'm guessing this is why the patch we expected to fix this didn't work... I'm going to add shared lockfile support to sstate, then we can use that... Fixed in master in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=68baef906513c1abef5372d92c769dc5d695e054 |