Autobuilder reports: ERROR: Function 'do_rootfs' failed (see /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-poky-linux/poky-image-lsb-1.0-r0/temp/log.do_rootfs.28453 for further information) ERROR: Logfile of failure stored in: /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-poky-linux/poky-image-lsb-1.0-r0/temp/log.do_rootfs.28453 Log data follows: | ERROR: Function 'do_rootfs' failed (see /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-poky-linux/poky-image-lsb-1.0-r0/temp/log.do_rootfs.28453 for further information) | Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/qemux86... | total: 1 0.000000 MB 2.016852 secs | fingerprint: 1146 0.008542 MB 0.062726 secs | install: 382 0.000000 MB 0.438539 secs | dbadd: 382 0.000000 MB 0.427742 secs | dbget: 2505 0.000000 MB 0.005212 secs | dbput: 382 1.720148 MB 0.353341 secs | readhdr: 3821 3.376208 MB 1.108379 secs | hdrload: 1910 5.010692 MB 0.009065 secs | hdrget: 65200 0.000000 MB 0.055676 secs | Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/i586... | error: open of /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/i586/task-poky-sdk-1.0-r7.i586.rpm failed: No such file or directory | rpm.real: ./rpmio_internal.h:190: fdGetOPath: Assertion `fd != ((void *)0) && fd->magic == 0x04463138' failed. | /builds2/pok Upon investigation we see that: eflanagan@autobuilder03:~> ls -al /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/i586/task-poky-sdk-1.0-r7.i586.rpm -rwxr--r-- 1 pokybuild users 3000 2011-03-03 13:47 /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/i586/task-poky-sdk-1.0-r7.i586.rpm This has also occurred elsewhere: ERROR: Function 'do_rootfs' failed (see /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-64-poky-linux/poky-image-minimal-1.0-r0/temp/log.do_rootfs.20074 for further information) ERROR: Logfile of failure stored in: /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-64-poky-linux/poky-image-minimal-1.0-r0/temp/log.do_rootfs.20074 Log data follows: | Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/qemux86-64... | total: 1 0.000000 MB 5.299116 secs | fingerprint: 1158 0.008484 MB 0.213973 secs | install: 386 0.000000 MB 0.790160 secs | dbadd: 386 0.000000 MB 0.771915 secs | dbget: 2469 0.000000 MB 0.007738 secs | dbput: 386 1.734856 MB 0.665613 secs | readhdr: 3861 3.402560 MB 3.726118 secs | hdrload: 1936 5.067812 MB 0.012818 secs | hdrget: 65936 0.000000 MB 0.072568 secs | Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/x86_64... | error: open of /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/x86_64/libtiffxx-dev-3.8.2-r2.x86_64.rpm failed: No such file or directory | total: 1 0.000000 MB 17.470722 secs | fingerprint: 3918 0.070208 MB 0.596252 secs | install: 1306 0.000000 MB 2.646070 secs | dbadd: 1306 0.000000 MB 2.610542 secs | dbget: 24728 0.000000 MB 0.028352 secs | dbput: 1306 7.334516 MB 2.321668 secs | readhdr: 13061 14.552984 MB 12.311993 secs | hdrload: 7526 27.525252 MB 0.044016 secs | hdrget: 235962 0.000000 MB 0.221356 secs | Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/all... | total: 1 0.000000 MB 1.034876 secs | fingerprint: 126 0.003376 MB 0.101518 secs | install: 42 0.000000 MB 0.063101 secs | dbadd: 42 0.000000 MB 0.059886 secs | dbget: 639 0.000000 MB 0.000695 secs | dbput: 42 0.268924 MB 0.035743 secs | readhdr: 421 0.534456 MB 0.806430 secs | hdrload: 210 0.793876 MB 0.001077 secs | hdrget: 6939 0.000000 MB 0.005857 secs | Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/x86_64-nativesdk... | total: 1 0.000000 MB 13.518695 secs | fingerprint: 1830 0.028198 MB 0.162148 secs | install: 610 0.000000 MB 1.221415 secs | dbadd: 610 0.000000 MB 1.205607 secs | dbget: 10548 0.000000 MB 0.012174 secs | dbput: 610 4.071680 MB 1.103758 secs | readhdr: 6101 8.095502 MB 11.510949 secs | hdrload: 3053 12.045458 MB 0.014814 secs | hdrget: 86330 0.000000 MB 0.076150 secs | Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/all... | Processing task-poky-boot... | error: Failed dependencies: | tinylogin is needed by task-poky-boot-1.0-r7.qemux86 | busybox is needed by task-poky-boot-1.0-r7.qemux86 | modutils-initscripts is needed by task-poky-boot-1.0-r7.qemux86 | initscripts is needed by task-poky-boot-1.0-r7.qemux86 | update-alternatives-cworth is needed by task-poky-boot-1.0-r7.qemux86 | udev is needed by task-poky-boot-1.0-r7.qemux86 | sysvinit is needed by task-poky-boot-1.0-r7.qemux86 | base-passwd is needed by task-poky-boot-1.0-r7.qemux86 | libc6 >= 2.12 is needed by v86d-0.1.9-r0.qemux86 | /bin/sh is needed by v86d-0.1.9-r0.qemux86 | /bin/sh is needed by netbase-4.44-r0.qemux86 | /bin/sh is needed by kernel-module-uvesafb-2.6.37+git1+e2737075b79e4fc682e41051cf1c0bc47a47d502_1+2b412826bbeb4a16abe2ea74f2456ab880c6e3c1-r16.qemux86 | module-init-tools-depmod is needed by update-modules-1.0-r8.all | /bin/sh is needed by update-modules-1.0-r8.all | /bin/sh is needed by kernel-module-cfbfillrect-2.6.37+git1+e2737075b79e4fc682e41051cf1c0bc47a47d502_1+2b412826bbeb4a16abe2ea74f2456ab880c6e3c1-r16.qemux86 | /bin/sh is needed by kernel-module-cfbimgblt-2.6.37+git1+e2737075b79e4fc682e41051cf1c0bc47a47d502_1+2b412826bbeb4a16abe2ea74f2456ab880c6eERROR: Function 'do_rootfs' failed (see /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-64-poky-linux/poky-image-minimal-1.0-r0/temp/log.do_rootfs.20074 for further information) | 3c1-r16.qemux86 | /bin/sh is needed by kernel-module-cfbcopyarea-2.6.37+git1+e2737075b79e4fc682e41051cf1c0bc47a47d502_1+2b412826bbeb4a16abe2ea74f2456ab880c6e3c1-r16.qemux86 eflanagan@autobuilder03:~> ls -al /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/x86_64/libtiffxx-dev-3.8.2-r2.x86_64.rpm -rwxr--r-- 1 pokybuild users 21049 2011-03-03 14:02 /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/x86_64/libtiffxx-dev-3.8.2-r2.x86_64.rpm
We need to add the rpm lock file to the package generation. The concern though is that we can generate all of the rpm packages we want, we just can't generate packages during a rootfs build.. Need some further information on how to do this.
pull request for a fix has been sent. The fix uses shared locks vs exclusive locks.
http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=c180383885e32aed83c4f8691115859966fb53be
*** Bug 867 has been marked as a duplicate of this bug. ***
The issue still exists in my build. Commit 43a2d098008eee711399b8d64594d84ae034b9bf. 1) Run "bitbake poky-image-sato-live". Build the full sato-live image until it is successful. 2) Bump connman's PR (or some other sato recipe's PR). 3) Rebuild the sato live image by "bitbake poky-image-sato-live". Sometimes, I meet build error like the following, however it does not happen every time. ------------------------------------------------------- Generating solve db for /distro/dongxiao/build-qemux86/tmp/deploy/rpm/qemux86... total: 1 0.000000 MB 1.424841 secs fingerprint: 1020 0.006796 MB 0.033057 secs install: 340 0.000000 MB 0.371773 secs dbadd: 340 0.000000 MB 0.362746 secs dbget: 2196 0.000000 MB 0.004278 secs dbput: 340 1.504908 MB 0.308950 secs readhdr: 3401 2.961280 MB 0.005603 secs hdrload: 1700 4.389932 MB 0.007001 secs hdrget: 57535 0.000000 MB 0.043769 secs Generating solve db for /distro/dongxiao/build-qemux86/tmp/deploy/rpm/i586... error: open of /distro/dongxiao/build-qemux86/tmp/deploy/rpm/i586/connman-plugin-ethernet-0.65-r4.i586.rpm failed: No such file or directory rpm.real: ./rpmio_internal.h:190: fdGetOPath: Assertion `fd != ((void *)0) && fd->magic == 0x04463138' failed. /distro/dongxiao/build-qemux86/tmp/work/qemux86-poky-linux/poky-image-minimal-initramfs-1.0-r0/temp/run.do_rootfs.468: line 375: 669 Aborted rpm --dbpath /var/lib/rpm --define='_openall_before_chroot 1' -i --replacepkgs --replacefiles --oldpackage -D "_dbpath $pkgdir/solvedb" --justdb --noaid --nodeps --noorder --noscripts --notriggers --noparentdirs --nolinktos --stats --ignoresize --nosignature --nodigest -D "__dbi_txn create nofsync" $pkgdir/solvedb/manifest ERROR: Function 'do_rootfs' failed (see /distro/dongxiao/build-qemux86/tmp/work/qemux86-poky-linux/poky-image-minimal-initramfs-1.0-r0/temp/log.do_rootfs.468 for further information) ------------------------------------------------------- The root cause for this issue should be, poky-image-minimal-initramfs's do_rootfs task doesn't have dependency on connman, thus their tasks will be run simultaneously. Poky-image-minimal-initramfs's do_rootfs will call "rootfs_rpm_do_rootfs" --> "package_update_index_rpm", where it will update all the packages depsolver db in ${DEPLOY_DIR_RPM}. When the package_update_index_rpm function is handling connman's rpm package, and at the same time, connman is removing old rpm and trying to generate a new one (e.x, from r4 to r5), then the build error will occur, saying that it could not find r4 version of connman-plugin-ethernet...
This fix was applied: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=c180383885e32aed83c4f8691115859966fb53be but I think the lock is taken in the wrong place. The files are not placed into tmp/deploy/ at this point but later in the sstate code. Its that later point where we need to have the lock held (and also at the point any sstate package is uninstalled.
Locks (rpm.lock) are already taken in python function sstate_clean and sstate_install. But the lock is not taken in "do_rootfs" --> "rootfs_rpm_do_rootfs" --> ""package_update_index_rpm", where it operates in "${DEPLOY_DIR_RPM}". I am in confuse that the "do_rootfs" is written in shell, and how can we add lock for it or its sub-functions?
There is a global exclusive lock in the rootfs_rpm.bbclass: # RPM doesn't work with multiple rootfs generation at once due to collisions in the use of files # in ${DEPLOY_DIR_RPM}. This can be removed if package_update_index_rpm can be called concurrently do_rootfs[lockfiles] += "${DEPLOY_DIR_RPM}/rpm.lock" This is the same lock file that should be done whenever the tmp/deploy/rpm/* is being modified. The intention is to use an exclusive lock in the do_rootfs, but use a shared lock during RPM copying into the deploy directory. We can manipulate multiple RPMs at the same time, but we must block when doing all of the rootfs operations.
I've pushed a possible fix to poky-contrib mhatle/bernard/fixes, commit id: 81c5ab86c5c0697cdee3958f69d4ad6251083a9d Please try that and see if it corrects the issue you were seeing.
Sorry I still see the same error after applied your latest fix on mhatle/bernard/fixes.
Please refresh from poky-contrib mhatle/bernard/fixes and try again. The top commit: 3182ada41f06e20c325d0f63ae9b211523490a60 takes the drastic approach of locking/serializing the whole system during the RPM package creation. I'm curious if this resolves the problem or not.
Sorry, I just noticed I changed the wrong variable, the right commit has now been pushed: c0edbfdd704a8fcb6f07e3b5059bec6d3b7f22ab difference is affecting "sstate-lockfile" vs "lockfile"..
ONE more time, the -right- commit for head is: 8824e8130cac060e9d06881aa98f8aba7bf4002b Sorry for any confusion. Again the key line that should be there: +do_package_write_rpm[sstate-lockfiles] += "${DEPLOY_DIR_RPM}/rpm.lock"
I am aware of that change and is now testing against the latest fix.
Sorry this bug still exists after applying the fix of the following. But it seems that the rate to reproduce the issue is lower, now I can encounter this issue within about 5~10 trials. diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass index 4647116..56e8308 100644 --- a/meta/classes/package_rpm.bbclass +++ b/meta/classes/package_rpm.bbclass @@ -777,16 +777,10 @@ python do_package_rpm () { cmd = cmd + " --define 'debug_package %{nil}'" cmd = cmd + " -bb " + outspecfile - # Take a shared lock, we can write multiple packages at the same time... - # but we need to stop the rootfs/solver from running while we do... - lf = bb.utils.lockfile(bb.data.expand("${DEPLOY_DIR_RPM}/rpm.lock", d), True) - # Build the rpm package! bb.data.setVar('BUILDSPEC', cmd + "\n", d) bb.data.setVarFlag('BUILDSPEC', 'func', '1', d) bb.build.exec_func('BUILDSPEC', d) - - bb.utils.unlockfile(lf) } python () { @@ -803,6 +797,7 @@ SSTATETASKS += "do_package_write_rpm" do_package_write_rpm[sstate-name] = "deploy-rpm" do_package_write_rpm[sstate-inputdirs] = "${PKGWRITEDIRRPM}" do_package_write_rpm[sstate-outputdirs] = "${DEPLOY_DIR_RPM}" +do_package_write_rpm[sstate-lockfiles] += "${DEPLOY_DIR_RPM}/rpm.lock" python do_package_write_rpm_setscene () { sstate_setscene(d)
I've noticed it should be "sstate-lockfile", not sstate-lockfiles. I'm guessing this is why the patch we expected to fix this didn't work... I'm going to add shared lockfile support to sstate, then we can use that...
Fixed in master in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=68baef906513c1abef5372d92c769dc5d695e054
Bernard http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=3d2c481ab0047a0432b61ec226aa7da2e64bdde1