| Summary: | Cached archives in downloads (${DL_DIR}) are not uniquely identified | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Alex Lennon <ajlennon> |
| Component: | bitbake | Assignee: | Richard Purdie <richard.purdie> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | normal | ||
| Priority: | Undecided | CC: | bluelightning, poky.bs.watcher, poky.watcher, ross.burton |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
I would note that we require checksums for non local archives so the chances of unpacking something incorrectly are low. There is a problem with the full paths not being stored in DL_DIR but this has been that way for many years. You can also work around this by specifying downloadfilename= as a URL parameter. It's worth pointing out that the tarballs generated by github are not guaranteed to be the same for all time, and in the future the tarball will be regenerated and so the checksum will change. Instead of using a github-generated tarball we recommend that you either use the git fetcher and checkout the hash of the tag, or use a proper persistent release tarball. "It's worth pointing out that the tarballs generated by github are not guaranteed to be the same for all time" Thanks Ross. That is interesting. Something of a deficiency in the way Github manages release tarballs then. Brief Googling seems to indicate this is likely to be a timestamp changing in the regenerated archive when a GitHub staging area is cleared out? I shall use an alternative mechanism for referencing code in Github in future. I do believe the wider issue of guaranteeing the uniqueness of previously download ed archives is still relevant though. "I would note that we require checksums for non local archives so the chances of unpacking something incorrectly are low." Thanks Richard. Yes, in retrospect the incorrect archive was unpacked because as I was creating the new recipe I took the checksums generated for me by bitbake as those for my new recipe. I agree that what would happen in the general case would be that the recipe would fail to unpack with a checksum error. Presumably this failure could occur at any point that a user pulls in a recipe from any layer where there is a file naming conflict with any recipe that has already been built. Would it not be a more comprehensive solution to avoid potential conflicts between source archives by maintaining the full URI from which the file was downloaded? In the meantime I shall make use of downloadfilename in the recipe, thanks. So I think given how rarely this occurs I'd have to say I don't think we should change anything here. The only way to fix this would be to either change the layout of DL_DIR, or to rename the files as they are downloaded, neither of which seem particularly desirable. Upstreams *should* use proper file names for their downloads; if they do not, we have the means to work around that using downloadfilename=. Given all of that, I'm marking this as WONTFIX. NB. Feedback from GitHub on the reason for the archive check-summing issue, "We do our best to keep the files from changing. However, they are dynamically generated, and are subject to change if the underlying server libraries change. We make every attempt to fix this when we detect it. If you must rely on the files absolutely never changing, I'd recommend uploading the binaries to Releases manually or through the API. These are stored in permanent storage and will not change." |
Overview: Cached archives in ${DL_DIR} are stored by name only and do not include the source URI from which they were retrieved. This means that during a build, if two recipes are set to retrieve an archive with the same file-name but from different sources then a previously cached and potentially incorrect archive can be unpacked. Detail / Steps to reproduce: I have some code committed to github here https://github.com/DynamicDevices/bbexample/ It's tagged v1.0 so github generates a source tarball for me here https://github.com/DynamicDevices/bbexample/archive/v1.0.tar.gz I use a SRC_URI in my recipe like this SRC_URI = "https://github.com/DynamicDevices/bbexample/archive/v${PV}.tar.gz" This should result in the file being downloaded, or pulled from cache, unpacked and so forth. When I build this recipe I get the wrong source code unpacked (from a mono-helloworld project I created a while ago) The mono-helloworld project is also at github and also tagged v1.0, which results in an archive of the same name at GitHub https://github.com/DynamicDevices/mono-helloworld/archive/v1.0.tar.gz If I look in my downloads folder I see there is a v1.0.tar.gz file and a corresponding .done file. This contains the mono-helloworld sources, as I have previously built that recipe. Thus I believe that bitbake is incorrectly assuming that the mono-helloworld tarball is the cached bbexample tarball as there's no URI information there for bitbake to distinguish the difference. I was able to show this to be the case by removing the v1.0.tar.gz file containing the mono-helloworld sources in which case a rebuild of my bbexample recipe results in the correct sources being pulled down, although of course then the mono-helloworld recipe is broken