| Summary: | package.bbclass does not treat directory symlinks properly, leading to build failures | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] Meta-yocto | Reporter: | Heinz Wrobel <Heinz.Wrobel> | ||||
| Component: | meta-yocto | Assignee: | Markus Lehtonen <markus.lehtonen> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | Medium | CC: | poky.bs.watcher, poky.watcher, sgw | ||||
| Version: | 2.0 | ||||||
| Target Milestone: | 2.2 | ||||||
| Hardware: | Other | ||||||
| OS: | Multiple | ||||||
| Whiteboard: | |||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||
| Attachments: |
|
||||||
Thank you for the easily reproducible test cases. I tested it quickly and it seems that populate_packages() is not handling directory symlinks coherently. Test cases #1 and #2 are valid and they fail with ipkg, too. However, we do not really support rpm4 anymore so testcase #3 won't probably get fixed. Hmm, taking a closer look I'm not entirely sure about this. Suppose you have a directory structure like this:
var
├── foo
│ └── bar
│ └── file
└── symlink -> foo/bar/
Looking at your test cases
#1: FILES_{PN} = "var/ var/foo var/foo/bar var/symlink var/symlink/file"
#2: FILES_{PN} = "var/ var/symlink var/symlink/file"
all the files after "var/" are redundant because they're already included by it. Basically, I think it shouldn't fail just because you're referencing files twice. But, I think the problem here is listing files under symlinked directories in the first place. The ordering of FILES_{PN} becomes significant.
What if you defined
FILES_{PN} = "var/symlink/file var/"
or just
FILES_{PN} = "var/symlink/file"
Looking at package.bbclass it looks like it would probably require significant changes to make all the symlink corner cases work perfectly. To be honest, I'm not sure it is worth jumping through hoops and risking breaking something else for this.
What we could do, for example, would be to detect files under symlinked directories that were listed in FILES_{PN}, "truncate" it's path to the symlink directory and print a warning that the user should use real paths instead of symlinks. For example, "var/symlink/file" would be truncated to "var/symlink".
What do you think?
Key thing is to eliminate the obscure, seemingly non-deterministic behavior.
Yes, if populate_packages() would check all file specs to see if any mid-path components are dir symlinks, I would write up your approach as follows:
File specs with mid-path dir symlinks could either be truncated with clear warning that the recipe X needs to be fixed, or simple rejected with an error. In essence, if a dir symlink is used, it must be the last path component only, and any other use must be clearly flagged as being a bad idea, explaining that dir symlinks may only be at the end of any path.
Net result would be that any recipe would have to specify real paths only.
A concern that I have is what this means for existing recipes. If a recipe uses "{localstatedir}log", how should it know that a distro doesn't really provide that path but "{localstatedir}volatile/log" in reality?
In other words, how would a recipe deal with symlinks beyond its control or provided by means outside the .bb file?
Would sorting the paths to ensure all dir symlinks resolve properly help?
Again, I am concerned about side effects, but I don't know enough to understand all relationships here.
I quickly wrote a patch that should actually handle all cases. It is not very beautiful but shouldn't break anything. Please see: http://git.openembedded.org/openembedded-core-contrib/commit/?h=marquiz/fixes&id=39fe51eaead3d5b12e9c2539c02e0b086a8ee0c7 As you mentioned, a warning about dir symlinks might be a good addition. Any comments? Patchset sent for review: https://patchwork.openembedded.org/series/1127/ |
Created attachment 3252 [details] Simple recipe with package.bbclass symlink bug test cases [This is reported based on QorIQ-SDK-V2.0-20160527-yocto, which is built upon poky 2.0 jethro per poky.conf, running on CentOS 6.8 x64] Turns out that poky/meta/package.bbclass is not handling directory symlinks properly in populate_packages() when the package files are split into sub-packages. Depending on how the user specifies FILES variable content, building the recipe can fail, also if the preferred rpm provider is set to be an rpm.org version 4. It should be noted that rpm.org and rpm5 are separate developments and unfortunately not quite compatible. Unfortunately, the recipe names in Yocto are "rpm" as if they were just different versions of the same tree. Software requiring rpm.org version 4 can't be added easily without fixing up other recipes like consolekit 0.4.6 because of the packaging bug(s) described below. The attached mini recipe provides four test cases to illustrate how directory symlinks are improperly handled. Two test cases show generic failures, one test case show a failure provoked if rpm 4 is used, and one test case shows a work around that is possible, but totally non-obvious.