Excerpt from the autobuilder log https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-arm/builds/177/steps/BuildImages_2/logs/stdio --------------- snip ----------------- ERROR: Index creation command '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/build/tmp/sysroots/x86_64-linux/usr/bin/createrepo --update -q /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/build/tmp/deploy/rpm/all' failed with return code -11: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery ==> rpmdbe_event_notify(0x206ac30, PANIC(0), 0x7fffac6ce22c) app_private (nil) error: db_init:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-rpm/build/build/tmp/work/x86_64-linux/rpm-native/5.4.14-r0/rpm-5.4.14/rpmdb/db3.c:1155: dbenv->failchk(-30973): BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: BDB1581 File handles still open at environment close rpmdb: BDB1582 Open file handle: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/build/tmp/sysroots/x86_64-linux/var/lib/rpm/__db.001 rpmdb: BDB1582 Open file handle: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/build/tmp/sysroots/x86_64-linux/var/lib/rpm/__db.002 rpmdb: BDB1582 Open file handle: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/build/tmp/sysroots/x86_64-linux/var/lib/rpm/__db.003 ERROR: Function failed: do_rootfs --------------- snip ----------------- This is repeated in other builds, and we have seen this frequently since the new rpm & db recipes were merged in.
Another failure on a master build: https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc/builds/181/steps/Building%20Toolchain%20Images/logs/stdio
This happens very often in our build server as well.
Tracked at https://bugs.launchpad.net/rpm/+bug/1354240
createrepo is (likely) unprepared to run with a transactional ACID Berkeley DB database. The likeliest problem is that createrepo is just exiting without closing the rpmdb, possibly on a (createrepo) error exit. This can be confirmed by running with the equivalent of "rpm -vv" in createrepo. You should see the database indices closed on exit. Note that rpmlib will try to correct an error from a previous abnormal exit on the next invocation, so look for errors on the previous command. DB_RUNRECOVERY is just the symptom of an automated repair, not the problem that needs fixing. What python bindings are in use? Presumably from rpm5 ... What createrepo? Note that createrepo developers categorically refuse to accept any patches from @rpm5.org, YMMV.
We're using createrepo version 0.4.11 with some patches. Download is from: http://createrepo.baseurl.org/download/createrepo-0.4.11.tar.gz The code is fairly old (2007) so it wouldn't surprise me at all if there are issues relating to the closing of the DB. In addition we apply the patches located at: http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/createrepo/createrepo
Another failure: https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-arm/builds/208/steps/BuildImages_2/logs/stdio (someone had wondered about gcc 4.9 issues, this is after our recent gcc updates)
Same failure happened in our internal builder on 08/18/2014.
For me the RPM is very broken. Our autobuilder fails every two weeks with some sort of corruption. Yesterday I got: ----- snip ----- wayland fsl-image-machine-test@imx6qsabresd (1/5) ERROR: Unable to install packages. Command '.../build-wayland/tmp/sysroots/x86_64-linux/usr/bin/smart --quiet --data-dir=.../build-wayland/tmp/work/imx6qsabresd-poky-linux-gnueabi/fsl-image-machine-test/1.0-r0/rootfs/var/lib/smart install -y packagegroup-fsl-tools-gpu@imx6qsabresd packagegroup-core-tools-testapps@imx6qsabresd packagegroup-base-extended@imx6qsabresd run-postinsts@all packagegroup-fsl-tools-benchmark@imx6qsabresd packagegroup-core-tools-profile@imx6qsabresd packagegroup-fsl-gstreamer-full@all packagegroup-fsl-tools-gpu-external@all weston-init@all packagegroup-fsl-gstreamer@imx6qsabresd packagegroup-fsl-tools-testapps@all packagegroup-core-boot@imx6qsabresd locale-base-en-us@cortexa9hf_vfp_neon locale-base-en-gb@cortexa9hf_vfp_neon' returned 1: ------ snip ----- In case we cannot find the root cause of it, we ought to revert to previous RPM version which were solid.
Please note that DB_RUNRECOVERY fixups are fully automated when rpm runs next: an rpmdb in rpm5 is fully ACID. The root cause of this issue (and the fix) is that yum is exiting without closing an rpmdb properly. The warnings are being displayed to diagnose the problem and are not "corruption". If you want a fix, I need some assistance identifying a reproducer. I can't fix what I cannot see, and I am not seeing this issue running under RHEL6 yum.
(In reply to comment #9) > Please note that DB_RUNRECOVERY fixups are fully automated > when rpm runs next: an rpmdb in rpm5 is fully ACID. > > The root cause of this issue (and the fix) is that yum is > exiting without closing an rpmdb properly. The warnings > are being displayed to diagnose the problem and are not > "corruption". > > If you want a fix, I need some assistance identifying > a reproducer. > > I can't fix what I cannot see, and I am not seeing > this issue running under RHEL6 yum. I am seeing build failures from time to time on Debian Wheezy. So this is a regression for me. Also the host shouldn't matter here as RPM is a native package built during the build process so I am surprise you are not seeing failures from time to time. In the builder it happens every two weeks in incremental builds. Daisy works fine and does not has this errors so it is clearly a regression in Dizzy.
Otavio what do you mean by incremental builds? Do you have "INC_RPM_IMAGE_GEN" enabled? Or do you mean that you are just building by: bitbake target git pull bitbake target git pull bitbake target ...etc...? If the INC_RPM_IMAGE_GEN is enabled, then an existing database should be there, and has somehow been corrupted. Without that set, each invocation is a new root filesystem image with a clean RPM database. I am also confused as to the logs. The first message from Paul (2014-07-24) indicate a failure with the -host- system RPM database: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/build/tmp/sysroots/x86_64-linux/var/lib/rpm/__db.001 This is indicating that the system is running createrepo, and in the process is generated a new (presumably empty RPM database) in the target sysroot. I'm not sure why this is happening, but even if it is -- the database should be empty and thus not causing any issues. Otavio, are you getting the same error -- or are you getting an error running 'smart'? A problem with 'smart' will be very different from the original problem that was reported, as we'll need more information as to the environment to determine a potential cause.
(In reply to comment #11) > Otavio what do you mean by incremental builds? > > Do you have "INC_RPM_IMAGE_GEN" enabled? No. > Or do you mean that you are just building by: > > bitbake target > git pull > bitbake target > git pull > bitbake target > ...etc...? Yes. > If the INC_RPM_IMAGE_GEN is enabled, then an existing database should be > there, and has somehow been corrupted. > > Without that set, each invocation is a new root filesystem image with a > clean RPM database. Right but when the build fails it just happen to succeed again if I remove the tmp. > I am also confused as to the logs. The first message from Paul (2014-07-24) > indicate a failure with the -host- system RPM database: > > /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/build/ > tmp/sysroots/x86_64-linux/var/lib/rpm/__db.001 > > This is indicating that the system is running createrepo, and in the process > is generated a new (presumably empty RPM database) in the target sysroot. > I'm not sure why this is happening, but even if it is -- the database should > be empty and thus not causing any issues. Shouldn't it use the native utilities built by Poky? > Otavio, are you getting the same error -- or are you getting an error > running 'smart'? A problem with 'smart' will be very different from the > original problem that was reported, as we'll need more information as to the > environment to determine a potential cause. I receive, from time to time two errors: /.../tmp/sysroots/x86_64-linux/var/lib/rpm/__db.001 and also: ERROR: Unable to install packages. Command '.../build-wayland/tmp/sysroots/x86_64-linux/usr/bin/smart --quiet --data-dir=.../build-wayland/tmp/work/imx6qsabresd-poky-linux-gnueabi/fsl-image-machine-test/1.0-r0/rootfs/var/lib/smart install -y packagegroup-fsl-tools-gpu@imx6qsabresd packagegroup-core-tools-testapps@imx6qsabresd packagegroup-base-extended@imx6qsabresd run-postinsts@all packagegroup-fsl-tools-benchmark@imx6qsabresd packagegroup-core-tools-profile@imx6qsabresd packagegroup-fsl-gstreamer-full@all packagegroup-fsl-tools-gpu-external@all weston-init@all packagegroup-fsl-gstreamer@imx6qsabresd packagegroup-fsl-tools-testapps@all packagegroup-core-boot@imx6qsabresd locale-base-en-us@cortexa9hf_vfp_neon locale-base-en-gb@cortexa9hf_vfp_neon' returned 1: The last from the smart.
Reproducer: > > bitbake target > > git pull > > bitbake target > > git pull > > bitbake target > > ...etc...? > > Yes. > > > If the INC_RPM_IMAGE_GEN is enabled, then an existing database should be > > there, and has somehow been corrupted. > > > > Without that set, each invocation is a new root filesystem image with a > > clean RPM database. > > Right but when the build fails it just happen to succeed again if I remove > the tmp. Do you HAVE to remove tmp? or does it keep failing until you do. Have you tried simply bitbake -c clean <image>? We need to understand where the fault condition is (based on the description it looks like there may be two). > > I am also confused as to the logs. The first message from Paul (2014-07-24) > > indicate a failure with the -host- system RPM database: > > > > /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/build/ > > tmp/sysroots/x86_64-linux/var/lib/rpm/__db.001 > > > > This is indicating that the system is running createrepo, and in the process > > is generated a new (presumably empty RPM database) in the target sysroot. > > I'm not sure why this is happening, but even if it is -- the database should > > be empty and thus not causing any issues. > > Shouldn't it use the native utilities built by Poky? Yes, the createrepo is generated as part of the native packages (as is RPM). The database path listed above in in the native sysroot. There really is no reason we need to create a database -- other then it's probably generated because RPM is being used. > > Otavio, are you getting the same error -- or are you getting an error > > running 'smart'? A problem with 'smart' will be very different from the > > original problem that was reported, as we'll need more information as to the > > environment to determine a potential cause. > > I receive, from time to time two errors: > > /.../tmp/sysroots/x86_64-linux/var/lib/rpm/__db.001 The one would be one error.. > and also: > > ERROR: Unable to install packages. Command > '.../build-wayland/tmp/sysroots/x86_64-linux/usr/bin/smart --quiet > --data-dir=.../build-wayland/tmp/work/imx6qsabresd-poky-linux-gnueabi/fsl- > image-machine-test/1.0-r0/rootfs/var/lib/smart install -y > packagegroup-fsl-tools-gpu@imx6qsabresd > packagegroup-core-tools-testapps@imx6qsabresd > packagegroup-base-extended@imx6qsabresd run-postinsts@all > packagegroup-fsl-tools-benchmark@imx6qsabresd > packagegroup-core-tools-profile@imx6qsabresd > packagegroup-fsl-gstreamer-full@all packagegroup-fsl-tools-gpu-external@all > weston-init@all packagegroup-fsl-gstreamer@imx6qsabresd > packagegroup-fsl-tools-testapps@all packagegroup-core-boot@imx6qsabresd > locale-base-en-us@cortexa9hf_vfp_neon locale-base-en-gb@cortexa9hf_vfp_neon' > returned 1: > > The last from the smart. This is a different issue. We need to split this into two defects. I'll go ahead and do that.
The bugs have been split. This one should focus on the failure during or around crearepo due to the: dbenv->failchk(-30973): BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
(In reply to comment #13) > Reproducer: > > > > bitbake target > > > git pull > > > bitbake target > > > git pull > > > bitbake target > > > ...etc...? > > > > Yes. > > > > > If the INC_RPM_IMAGE_GEN is enabled, then an existing database should be > > > there, and has somehow been corrupted. > > > > > > Without that set, each invocation is a new root filesystem image with a > > > clean RPM database. > > > > Right but when the build fails it just happen to succeed again if I remove > > the tmp. > > Do you HAVE to remove tmp? or does it keep failing until you do. It keeps failing. > Have you tried simply bitbake -c clean <image>? We need to understand > where the fault condition is (based on the description it looks like there > may be two). No; never. > > > I am also confused as to the logs. The first message from Paul (2014-07-24) > > > indicate a failure with the -host- system RPM database: > > > > > > /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-fsl-arm/build/build/ > > > tmp/sysroots/x86_64-linux/var/lib/rpm/__db.001 > > > > > > This is indicating that the system is running createrepo, and in the process > > > is generated a new (presumably empty RPM database) in the target sysroot. > > > I'm not sure why this is happening, but even if it is -- the database should > > > be empty and thus not causing any issues. > > > > Shouldn't it use the native utilities built by Poky? > > Yes, the createrepo is generated as part of the native packages (as is RPM). > The database path listed above in in the native sysroot. There really is no > reason we need to create a database -- other then it's probably generated > because RPM is being used. So I am concerned why only me sees it. > > > Otavio, are you getting the same error -- or are you getting an error > > > running 'smart'? A problem with 'smart' will be very different from the > > > original problem that was reported, as we'll need more information as to the > > > environment to determine a potential cause. > > > > I receive, from time to time two errors: > > > > /.../tmp/sysroots/x86_64-linux/var/lib/rpm/__db.001 > > The one would be one error.. > > > and also: > > > > ERROR: Unable to install packages. Command > > '.../build-wayland/tmp/sysroots/x86_64-linux/usr/bin/smart --quiet > > --data-dir=.../build-wayland/tmp/work/imx6qsabresd-poky-linux-gnueabi/fsl- > > image-machine-test/1.0-r0/rootfs/var/lib/smart install -y > > packagegroup-fsl-tools-gpu@imx6qsabresd > > packagegroup-core-tools-testapps@imx6qsabresd > > packagegroup-base-extended@imx6qsabresd run-postinsts@all > > packagegroup-fsl-tools-benchmark@imx6qsabresd > > packagegroup-core-tools-profile@imx6qsabresd > > packagegroup-fsl-gstreamer-full@all packagegroup-fsl-tools-gpu-external@all > > weston-init@all packagegroup-fsl-gstreamer@imx6qsabresd > > packagegroup-fsl-tools-testapps@all packagegroup-core-boot@imx6qsabresd > > locale-base-en-us@cortexa9hf_vfp_neon locale-base-en-gb@cortexa9hf_vfp_neon' > > returned 1: > > > > The last from the smart. > > This is a different issue. We need to split this into two defects. I'll go > ahead and do that. Ok.
The next time this occurs do the equivalent of: rm -f tmp/sysroots/x86_64-linux/var/lib/rpm/* If after that the issue goes away, it could be a race condition where multiple things were trying to open or control the BerkleyBD log files at the same time. One thing suggested to me by the RPM5 maintainer was that some systems attempt to "cleanup" the __db.* files after they run (for no good reason!). If createrepo [or another part of our scripting is doing this] while RPM is running, then invalid information could be written causing these issues. I don't have the time today, but if there is someone that can scan the bbclass, lib/or/rootfs.py and createrepo for an unlink/rm of the -host- __db files. That might point to the actual problem.
A clarification to If after that the issue goes away, it could be a race condition where multiple things were trying to open or control the BerkleyBD log files at the same time. Berkeley DB can easily handle multiple accesses and -- if rw access -- will also handle multiple readers/writers. The synchronization is through the __db* files. If the __db* files are _REMOVED__, then the __db* files will be recreated by another process, but there will be two non-interacting sets of locks, each in different __db* files, a condition that is essentially the same as no locking.
This bug is hardly to replicate. I guess this may be caused by create rpm db in parallel. It generated rarely when we built a image and it will pass if we run same comand to build again. Currently, I don't have a good method to debug this issue.
Is this still happening? BTW, I don't see the error in the provided logs. Have they been rewritten somehow?
Paul, can you give an update on the status of this bug? Is it still happening? How often? Where can I see the logs?
I haven't been monitoring the autobuilder for it, but I am unaware of anything that could have fixed it. Otavio do you still see this issue?
(In reply to comment #21) > I haven't been monitoring the autobuilder for it, but I am unaware of > anything that could have fixed it. Otavio do you still see this issue? Yes. Less often but it does.
The reason of this from my point of view is that rpm db gets broken by multiple instances of createrepo running in parallel. The simplest way to fix(work around?) this is to remove rpm database from sysroot before running createrepo. It's also possible to check if rpmdb is broken and rebuild it. Third possible solution is to upgrade createrepo or even switch to createrepo_c. I'd propose to start from checking/rebuilding and see if it fixes this bug. Opinions?
(In reply to comment #23) > The reason of this from my point of view is that rpm db gets broken by > multiple instances of createrepo running in parallel. > > The simplest way to fix(work around?) this is to remove rpm database from > sysroot before running createrepo. > > It's also possible to check if rpmdb is broken and rebuild it. > > Third possible solution is to upgrade createrepo or even switch to > createrepo_c. > > I'd propose to start from checking/rebuilding and see if it fixes this bug. > > Opinions? Upgrading to a later version of createrepo (or createrepo_c) is a good idea. The bug in need of fixing is in createrepo, which is not properly closing an rp[mdb when exiting, likely exiting abnormally. rpm already automates DB_RUNRECOVERY when/where needed. Running "rpm -qa" as root should suffice. There is no need to "rpm --rebuilddb" with rpm5: Berkeley DB is transactionally protected, all operations are logged, any/uncommitted transaction will be discarded with existing DB_RUNRECOVERY automation.
I've briefly looked at the lates createrepo and didn't see any calls of closeDB or similar. However, it uses rpmUtils.transaction.initReadOnlyTransaction() call. Old createrepo uses rpm.TransactionSet() Bringing new createrepo would be quite hard as it depends on yumdata, rpmUtils and may be other python APIs which Yocto doesn't have. createrepo_c looks better in this sense and its faster. I'm still thinking about possible workaround. Removing /var/lib/rpm/__db.* would most probably fix the issue. At least it's harmless and easy to do. This rpm database is created and used only by createrepo to create metadata for rpm repositories, so I don't see why we can't remove it before running createrepo. It looks like a clean approach to me to remove old rpm database before re-creating rpm repositories.
At least remove the __db* files using dbXY_recover without a -e flag.
What's dbXY_recover and why should I use it? I was going to do something similar to rm -rf /var/lib/rpm/*
One more idea is to run createrepo sequentially. running createrepo in parallel doesn't influence the performance as rpm database is locked when createrepo calls rpm.TransactionSet(). However, it may be the reason for the race when multiple createrepos try to create the same database. I tested this by running createrepo sequentially and in parallel and measuring the time. time bitbake core-image-minimal -c rootfs -f shows the same results.
Here are potential fixes for this bug: http://lists.openembedded.org/pipermail/openembedded-core/2015-March/103219.html http://lists.openembedded.org/pipermail/openembedded-core/2015-March/103222.html
Adding a --dbpath option, and adding a unique path to each createrepo invocation, will likely avoid (but not solve) the error message. createrepo is trying to lookup external pubkeys (i.e. not self-signed pubkeys from packages). rpmlib will attempt to retrieve pubkeys through HKP if enabled. There are hints in this report that multiple createrepo processes are running when the error occurs. You likely should attempt to serialize createrepo execution, keeping track of the pid, and perhaps the exit status, of createrepo if you wish to verify the cause of the error.
> Adding a --dbpath option, and adding a unique path > to each createrepo invocation, will likely avoid > (but not solve) the error message. It depends on when the failure is happening. As rpm database is deleted before running createrepo it might also solve the issue as potentially broken db will never exist at start. Anyway I consider this as a good first step to solve the issue. If it will continue to happening we know that we have to run createrepo sequentially(see below). > There are hints in this report that multiple createrepo > processes are running when the error occurs. You likely > should attempt to serialize createrepo execution, keeping > track of the pid, and perhaps the exit status, of createrepo > if you wish to verify the cause of the error. I tried this approach and it was rejected due to possible negative impact on performance: http://lists.openembedded.org/pipermail/openembedded-core/2015-March/103171.html
Yes createrepo (from 2007) was/is performance challenged. Modern versions of createrepo (and createrepo_c) are multi-threaded and can reuse previously calculated digests if/when used incrementally. So serializing createrepo need not be a performance bottleneck.
I was lead to believe that more modern versions of createrepo changed the data format. So smartpm would need to be updated to understand the new format and the new createrepo to understand the dependency hinting. Something to consider for the next version of YP.
Latest update of the fix uses different rpm db locations for *every* run of createrepo: http://lists.openembedded.org/pipermail/openembedded-core/2015-April/103277.html This is much better than serializing as it doesn't impact performance.
Later versions of createrepo have changed the file naming but not the XML markup (from memory: last I looked closely was ~2y ago). (aside) There is also rpmrepo.c, which was ~20% faster than createrepo when originally written, and could be made faster still if something other than the full *.rpm file digest (which is an expensive operation) were substituted into the package id field. afb@rpm5.org probably knows all the cost/benefit details of createrepo from his smart development experience.
Both fixes have been accepted to master: b544b4b23658580ed3454310852fa8af35f46e78 673d0224cd391fe6d99535c4eb060d56a239c3d8