9827 2016-06-27 06:44:34 +0000 package.bbclass does not treat directory symlinks properly, leading to build failures 2016-08-11 06:27:17 +0000 1 1 7 Build System, Metadata & Runtime Meta-yocto meta-yocto 2.0 Other Multiple RESOLVED FIXED Medium normal 2.2 1 Heinz.Wrobel markus.lehtonen poky.bs.watcher poky.watcher sgw --- --- No (bug/feature does not impact docs) oldest_to_newest 63333 0 3252 Heinz.Wrobel 2016-06-27 06:44:34 +0000 Created attachment 3252 Simple recipe with package.bbclass symlink bug test cases [This is reported based on QorIQ-SDK-V2.0-20160527-yocto, which is built upon poky 2.0 jethro per poky.conf, running on CentOS 6.8 x64] Turns out that poky/meta/package.bbclass is not handling directory symlinks properly in populate_packages() when the package files are split into sub-packages. Depending on how the user specifies FILES variable content, building the recipe can fail, also if the preferred rpm provider is set to be an rpm.org version 4. It should be noted that rpm.org and rpm5 are separate developments and unfortunately not quite compatible. Unfortunately, the recipe names in Yocto are "rpm" as if they were just different versions of the same tree. Software requiring rpm.org version 4 can't be added easily without fixing up other recipes like consolekit 0.4.6 because of the packaging bug(s) described below. The attached mini recipe provides four test cases to illustrate how directory symlinks are improperly handled. Two test cases show generic failures, one test case show a failure provoked if rpm 4 is used, and one test case shows a work around that is possible, but totally non-obvious. 63545 1 markus.lehtonen 2016-06-30 19:45:32 +0000 Thank you for the easily reproducible test cases. I tested it quickly and it seems that populate_packages() is not handling directory symlinks coherently. Test cases #1 and #2 are valid and they fail with ipkg, too. However, we do not really support rpm4 anymore so testcase #3 won't probably get fixed. 63564 2 markus.lehtonen 2016-07-01 07:35:07 +0000 Hmm, taking a closer look I'm not entirely sure about this. Suppose you have a directory structure like this: var ├── foo │   └── bar │   └── file └── symlink -> foo/bar/ Looking at your test cases #1: FILES_{PN} = "var/ var/foo var/foo/bar var/symlink var/symlink/file" #2: FILES_{PN} = "var/ var/symlink var/symlink/file" all the files after "var/" are redundant because they're already included by it. Basically, I think it shouldn't fail just because you're referencing files twice. But, I think the problem here is listing files under symlinked directories in the first place. The ordering of FILES_{PN} becomes significant. What if you defined FILES_{PN} = "var/symlink/file var/" or just FILES_{PN} = "var/symlink/file" Looking at package.bbclass it looks like it would probably require significant changes to make all the symlink corner cases work perfectly. To be honest, I'm not sure it is worth jumping through hoops and risking breaking something else for this. What we could do, for example, would be to detect files under symlinked directories that were listed in FILES_{PN}, "truncate" it's path to the symlink directory and print a warning that the user should use real paths instead of symlinks. For example, "var/symlink/file" would be truncated to "var/symlink". What do you think? 63565 3 Heinz.Wrobel 2016-07-01 07:58:22 +0000 Key thing is to eliminate the obscure, seemingly non-deterministic behavior. Yes, if populate_packages() would check all file specs to see if any mid-path components are dir symlinks, I would write up your approach as follows: File specs with mid-path dir symlinks could either be truncated with clear warning that the recipe X needs to be fixed, or simple rejected with an error. In essence, if a dir symlink is used, it must be the last path component only, and any other use must be clearly flagged as being a bad idea, explaining that dir symlinks may only be at the end of any path. Net result would be that any recipe would have to specify real paths only. A concern that I have is what this means for existing recipes. If a recipe uses "{localstatedir}log", how should it know that a distro doesn't really provide that path but "{localstatedir}volatile/log" in reality? In other words, how would a recipe deal with symlinks beyond its control or provided by means outside the .bb file? Would sorting the paths to ensure all dir symlinks resolve properly help? Again, I am concerned about side effects, but I don't know enough to understand all relationships here. 63570 4 markus.lehtonen 2016-07-01 10:54:30 +0000 I quickly wrote a patch that should actually handle all cases. It is not very beautiful but shouldn't break anything. Please see: http://git.openembedded.org/openembedded-core-contrib/commit/?h=marquiz/fixes&id=39fe51eaead3d5b12e9c2539c02e0b086a8ee0c7 As you mentioned, a warning about dir symlinks might be a good addition. Any comments? 64867 5 markus.lehtonen 2016-08-04 10:25:17 +0000 Patchset sent for review: https://patchwork.openembedded.org/series/1127/ 65054 6 markus.lehtonen 2016-08-11 06:27:17 +0000 Fixed in master: http://git.openembedded.org/openembedded-core/commit/?id=29d1738329ddf4e63844a9ad1158a1d41e2ee343 http://git.openembedded.org/openembedded-core/commit/?id=27b285bd641d62f65154e6deec5146c0c8bb1458 3252 2016-06-27 06:44:34 +0000 2016-06-27 06:44:34 +0000 Simple recipe with package.bbclass symlink bug test cases bugtestrpmsymlink.tgz application/x-gzip-compressed 1759 Heinz.Wrobel H4sIAL3JcFcAA+1Y224bNxD1s76CVQI0QWXdJTsp0iJwklaInQS2UyNPBrXLlVivlluSa0Uw/O89 w73q4hhpjaZFd2BA9i7nkHPmzHDkaTKzwlgdL8xqEcroqrP34NaFHRyM0s/x+mdme73+wWAw6o/6 g/Fetzfodwd7bPTwR9m2xFiuGdvTg2fY/e51973/j9p0K//058PuQQkej4dfyP9oI//DHiTBug97 jN32P88/ZbvxrQ9R2zezrfpvT6cPvcc99X8w6h64+h/2BsPhYEj9f9Qb1/X/T9jZx5OTl6ef2AvW PIcOWKA0s3PBIIi20jOWqYLJyFOLmFs5laG0q2bjeHL0+t3Za3L85cPxfr/d/cE9vHwzOX59dnn0 61tAs59/xvtAhuJ5p/P45uj9ycn7d5eZ6+WryeltJ3P+ceGPXhx2e8Fh99lh1+8d9Hx/POyPxr3u 4WDA+31/KsbNRuNR4xE7T3RkmEosTsoti9XVqrMQlndi7l3xmYCEvZAbw6RhkbJsziMfQcyYL7Xw rNIrgGSBGRZrFQsdrhAioOIk5FZcZkjmyVO2nIvIUZI9YxSOYVwLQonBBjytYiaZ7udubfZKxCLy aVMVsblaOoTECA0X4clAAsIxBZBrriWfhoJ5KrIisi02TWTonF0qsD7GSx6xgMuwxXhoFJMBPN2x tAiE1sKnnFE019LHNojdCDCk2FQwuOYJvRbaSJxp2Ib/xDIzV0no0yJQBRBHab4YxNHvI4oWcDHX IIf54lqEKl7gqAYgtCiJIBybRHgNJon0PxJpKaJUNKFos4/VNa1KaMCI+AKEIAOfFBLktmti4ybj eBrQ2hVbCjz9HfWKPAYIGdvn0UANOR0GSMxqgQ3PVGCXBKUFTqOJzy0WiNbvrePI94UPEMGNRAxL aecksUB+JsckZgrwOjuyYaG8cgkzKhRX0EC3PWyPgeNxZBkwKqiIhhDQaZ+YpyDPeFpOBVEeqiWy 0GicnR5dfjydsLJWqCdD7Wcvmo9vLt6fvqVSIfUzV6QeN4L1njP8ndr5HOl+83JyfJafgHWgqs61 gpqB2AnVzGGiFiDdUAvur7AUyvbwu3WBp1bIPZARD1OYwlu6cvGEMcInflnTHTRHLEDEZ2msccog KE/5Ii9Gby68q1TaEJp0mCoSxVrKX4GTwrdJpy7xJKeQS+SbT9PkoGqyzahz5ccBlo8KA7lpN3p8 8+HdLdH7+CZUHurHUsxS37LNBwVld78hOrbf3vUwT2U1df3nX8qcYyFvu4a58haRB8XEWlJ7Vlnt 8LCAKTqby0r+unzs8k70V3Oec16g4CacoWdk0s3PUFRIoJIIaT+vvEu5LwDQ6NAbdMsho7ysXrlU q2zLrCTsnDXXhNUsECAIanLGeWNxsa4JP20LkrBuqehc6HU5QQVI0dLpGCGUaegIfE0uaJng0omG 0P+KVL4m5YNdKX/y3dO7sk65ym4KwFNPRV341A55AUThtNPjk4urEEcSErjBb5tdULnNeYxriS7G Muf0HGRndU3p2vBtlVlzWaS+6SthwP424668s5ad+eBScyVLVQ66F0oXkRqSa1nteSSuT+QaRcSV KSTlyd262WVLt6XFsl21MEnXC6dzAE/RcQ31cgCtVIL8c0+kLS+h0AuMyoWJz9ZalvDA3c9MJxFE qK8KsSotZ9Q1S1qykaGsQ7pWshiElwoyjzTQalGGtyuc8zW/CFNFmPbZ6gZzuiYLysrCMq4lYHZr s5fMS7S7P2lkGLZouguTYuAYtnu9dh+TgYxjelbhNptF3KiUTV0IKQypSzl+Nio9nccK90lAtDvi XD4cG3SGEeV5bZJpuRInaKJ4J4RbkAEM1wHyMcihuForEX4ToCmZhat2456S/8JlcEeZDzfKPJXH mnqoL7vCoD6eXaq5BKYioOqo9uRqT6AW4Bq8Qrfb7BjogOjPFtn3W26DAoJK05WtSQk1ieeJsv1X QSpVtyWgHbUHKWEWLctjCakrV7501uI6nqH+Kw1Hq2Q2r+7cIkc31fqup9MldEe33hoNLua0zBZk t/KZYnotVWLQkqyyaQdV0X72tDxMellGSi+QFjef07BellPxNWHOr0U2TdM+lYmJa3cv/o1Zoxyw dukMEvPVJToohXHpOriP7yU3jWx/9ijVGn7csJX36Owa95PFYpV9acH5IQDcE26OrwDQQmlMgoxe iDRWvjlCuFrL0kLSKdtyBShW+DKU3fnuaqm0r0axLguG7fuI+NXtncR8tQPxVTgJb15MqD/d6+YS UPiGEds3AduY+XZA0Ia39X/Qaqutttpqq6222mqrrbbaaquttn+v/Ql7bccYACgAAA==