Bug 8578

Summary: opkg-make-index skips some packages so the package index is not up to date during rootfs generation
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Jin <jin>
Component: coreAssignee: Robert Yang <liezhi.yang>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium+ CC: alejandro.delcastillo, meta.mr.watcher, meta.watcher, richard.purdie
Version: 2.0   
Target Milestone: 2.4 M1   
Hardware: Other   
OS: arm   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Jin 2015-10-26 14:50:56 UTC
We have a nightly build of our poky based distros and after updating to poky 2.0 rc we started to get failures upon rootfs generation.

The build is a multimachine build for armv5 and imx6, the failure happens during the do_rootfs step:

 * opkg_install_pkg: Package ds-support-log-collector md5sum mismatch. Either the opkg or the package index are corrupt. Try 'opkg update'.

I traced it down to a problem in the opkg-make-index utility, it seems that
for some reason the package info, particularly the md5sum is not recalculated for some packages. This happens when the if clause in opkg-make-index:110 matches:

if filename in pkgsStamps and int(fnameStat.st_mtime) == pkgsStamps[filename]:

Then the package information is taken from old_pkg_hash[filename] and keeps old and invalid checksums. So as a result of this the md5sum in the resulting Packages file is wrong. If I hack this "if" out and force it to update, then the problem disappears and the Packages file is written with the correct md5sums.


Unfortunately I could not continue debugging, because once I tried the hack everything worked and I could not reproduce the problem again, however I am sure it will happen in the coming few days just as it always did and I will then continue my efforts.

I should also not that so far it happend only to allarch packages which had an RDEPENDS_${PN} setting which forced them to be rebuilt each time even though there were no changes, I will address this issue separately on the ML as discussed on IRC, still I am mentioning it here since it is probably part of something that triggers the problem described above.
Comment 1 Jin 2015-10-28 11:47:28 UTC
Had the problem again today and verified that mtime of Packages.stamps and mtime of the file on disk are indeed the same.

So, from what I see, somehow the Packages.stamps get updated with the current mtime while the md5sum is not being recalculated. Will try to get more debug logs from that "first time" when this happens.
Comment 2 Richard Purdie 2016-03-16 17:53:13 UTC
Oddly enough there is a separate bug where we saw something similar to this happen with createrepo and rpms. In that case we were also unable to figure out what triggered this rather strange problem. Would certainly be nice to get to the bottom of it! I'm suspecting reuse of sstate artefacts somehow...
Comment 3 Robert Yang 2016-10-12 08:56:55 UTC
I can't reproduce the problem, need more investigations.
Comment 4 Robert Yang 2017-03-31 02:21:30 UTC
I suggest move it as "Future", I'm no idea about why it happened.
Comment 5 Robert Yang 2017-05-19 10:25:48 UTC
Can't reproduce, so close it, feel free to reopen it if you reproduce it again.