8585 2015-10-27 11:13:03 +0000 Improve detection of missing host dependencies 2016-04-01 17:16:11 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 2.0 All Multiple RESOLVED FIXED Medium+ enhancement 2.1 M4 1 bluelightning bill.randle alexandru.c.georgescu bill.randle meta.mr.watcher meta.watcher richard.purdie ross.burton daniel.alexandrux.istrate --- --- No (bug/feature does not impact docs) oldest_to_newest 55810 0 bluelightning 2015-10-27 11:13:03 +0000 We have several checks in sanity.bbclass to verify that certain tools are installed on the host. A while back we decided to make these checks run only once and then we stamp that they are done and effectively never run them again unless TMPDIR gets deleted or we bump SANITY_VERSION. As far as I can tell we have never bumped SANITY_VERSION, despite having adjusted the dependencies and checks numerous times. Additionally, users have occasionally removed packages or upgraded their distributions and then find slightly obscure errors that are traced to these tools being missing; had the checks been run the issue would have been spotted immediately. I'd like to re-evaluate how long these checks actually take and if it's still too expensive to run them every time, it would be nice if we could at least somehow detect when a distro upgrade has taken place and trigger re-running all of the sanity checks if it has. Additionally, we might consider using some sort of signature in conf/sanity_info so that if e.g. SANITY_REQUIRED_UTILITIES changed then the corresponding check would be re-run automatically rather than us having to remember to bump SANITY_VERSION. 55811 1 ross.burton 2015-10-27 11:18:00 +0000 Surely checking $PATH for ten binaries can't take that long, even in the context of just the metadata parse, and memory-resident bitbake will only need to perform the check at startup anyway. Marking medium+ 2.1M2. 58231 2 ross.burton 2016-01-26 13:26:06 +0000 check_sanity_version_changed() takes my machine 2.8 seconds, 2.7 seconds of which is the "am I online" test as that hits both the main Yocto download site and bugzilla. Removing bugzilla from CONNECTIVITY_URLS so it just pings the main yocto download site reduces the time taken to 1 second. Changing the URL to just google.com reduces the time for the connectivity test to 0.2s. 58232 3 ross.burton 2016-01-26 14:18:21 +0000 Some real numbers: NOTE: connectivity took 0.156 seconds NOTE: sanity took 0.316 seconds The connectivity ping to google.com took 0.156 seconds, and check_sanity_version_changed() (including the ping) took 0.316 seconds. 58247 4 ross.burton 2016-01-26 17:26:11 +0000 Adding Richard to the CC as he has strong opinions that bitbake startup shouldn't be slower. Fight! 58452 5 ross.burton 2016-02-02 14:45:13 +0000 Connectivity check change was merged in meta-yocto b253c6073be44090a19d1743deb58ef566853056. 58458 6 richard.purdie 2016-02-02 15:44:40 +0000 I believe we already have far too much junk/busy work going on with bitbake startup already. Memory resident bitbake might change that but we're not there yet. Even then, I don't think we want to check all these things every time, just in case. We should however get more diligent about bumping SANITY_VERSION, or use the checksums mechanism we now have to test when to rerun the sanity tests. We could save NATIVELSBSTRING to detect distro upgrades? 58471 7 bluelightning 2016-02-02 19:43:49 +0000 Yes - I think we should definitely use NATIVELSBSTRING to force the checks to re-execute, and we should be bumping SANITY_VERSION pretty much every time we change the checks. 60256 8 3054 bill.randle 2016-03-22 00:47:33 +0000 Created attachment 3054 Proposed fix using NATIVELSBSTRING Here's a proposed fix for your consideration. It uses a change in NATIVELSBSTRING to flag a recheck. The true value of NATIVELSBSTRING is saved in the sanity_info file. 60300 9 bill.randle 2016-03-22 17:27:06 +0000 Patch submitted to OE mailing list. 60696 10 bill.randle 2016-04-01 17:16:11 +0000 Merged to master and verified by Richard Purdie. http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/classes/sanity.bbclass?id=e33124f828d0cd2dd600fff3f6ad3dc98d2723a1 3054 2016-03-22 00:47:33 +0000 2016-03-22 00:47:33 +0000 Proposed fix using NATIVELSBSTRING bug_8585_proposed_fix text/plain 2358 bill.randle ZGlmZiAtLWdpdCBhL21ldGEvY2xhc3Nlcy9zYW5pdHkuYmJjbGFzcyBiL21ldGEvY2xhc3Nlcy9z YW5pdHkuYmJjbGFzcwppbmRleCA1OTdmM2RlLi5mYWRhZDFiIDEwMDY0NAotLS0gYS9tZXRhL2Ns YXNzZXMvc2FuaXR5LmJiY2xhc3MKKysrIGIvbWV0YS9jbGFzc2VzL3Nhbml0eS5iYmNsYXNzCkBA IC02NDEsOSArNjQxLDkgQEAgZGVmIGNoZWNrX3Nhbml0eV9zc3RhdGVfZGlyX2NoYW5nZShzc3Rh dGVfZGlyLCBkYXRhKToKICAgICByZXR1cm4gdGVzdG1zZwogICAgICAgIAogZGVmIGNoZWNrX3Nh bml0eV92ZXJzaW9uX2NoYW5nZShzdGF0dXMsIGQpOgotICAgICMgU2FuaXR5IGNoZWNrcyB0byBi ZSBkb25lIHdoZW4gU0FOSVRZX1ZFUlNJT04gY2hhbmdlcworICAgICMgU2FuaXR5IGNoZWNrcyB0 byBiZSBkb25lIHdoZW4gU0FOSVRZX1ZFUlNJT04gb3IgTkFUSVZFTFNCU1RSSU5HIGNoYW5nZXMK ICAgICAjIEluIG90aGVyIHdvcmRzLCB0aGVzZSB0ZXN0cyBydW4gb25jZSBpbiBhIGdpdmVuIGJ1 aWxkIGRpcmVjdG9yeSBhbmQgdGhlbiAKLSAgICAjIG5ldmVyIGFnYWluIHVudGlsIHRoZSBzYW5p dHkgdmVyc2lvbiBjaGFuZ2VzLgorICAgICMgbmV2ZXIgYWdhaW4gdW50aWwgdGhlIHNhbml0eSB2 ZXJzaW9uIG9yIGhvc3QgZGlzdHJ1YnV0aW9uIGlkL3ZlcnNpb24gY2hhbmdlcy4KIAogICAgICMg Q2hlY2sgdGhlIHB5dGhvbiBpbnN0YWxsIGlzIGNvbXBsZXRlLiBnbGliLTIuMC1uYXRpdmVzIHJl cXVyaWVzCiAgICAgIyB4bWwucGFyc2Vycy5leHBhdApAQCAtOTM3LDYgKzkzNyw3IEBAIGRlZiBj aGVja19zYW5pdHkoc2FuaXR5X2RhdGEpOgogICAgIGxhc3Rfc2FuaXR5X3ZlcnNpb24gPSAwCiAg ICAgbGFzdF90bXBkaXIgPSAiIgogICAgIGxhc3Rfc3N0YXRlX2RpciA9ICIiCisgICAgbGFzdF9u YXRpdmVsc2JzdHIgPSAiIgogICAgIHNhbml0eXZlcmZpbGUgPSBzYW5pdHlfZGF0YS5leHBhbmQo IiR7VE9QRElSfS9jb25mL3Nhbml0eV9pbmZvIikKICAgICBpZiBvcy5wYXRoLmV4aXN0cyhzYW5p dHl2ZXJmaWxlKToKICAgICAgICAgd2l0aCBvcGVuKHNhbml0eXZlcmZpbGUsICdyJykgYXMgZjoK QEAgLTk0NywxMiArOTQ4LDE3IEBAIGRlZiBjaGVja19zYW5pdHkoc2FuaXR5X2RhdGEpOgogICAg ICAgICAgICAgICAgICAgICBsYXN0X3RtcGRpciA9IGxpbmUuc3BsaXQoKVsxXQogICAgICAgICAg ICAgICAgIGlmIGxpbmUuc3RhcnRzd2l0aCgnU1NUQVRFX0RJUicpOgogICAgICAgICAgICAgICAg ICAgICBsYXN0X3NzdGF0ZV9kaXIgPSBsaW5lLnNwbGl0KClbMV0KKyAgICAgICAgICAgICAgICBp ZiBsaW5lLnN0YXJ0c3dpdGgoJ05BVElWRUxTQlNUUklORycpOgorICAgICAgICAgICAgICAgICAg ICBsYXN0X25hdGl2ZWxzYnN0ciA9IGxpbmUuc3BsaXQoKVsxXQogCiAgICAgY2hlY2tfc2FuaXR5 X2V2ZXJ5YnVpbGQoc3RhdHVzLCBzYW5pdHlfZGF0YSkKICAgICAKICAgICBzYW5pdHlfdmVyc2lv biA9IGludChzYW5pdHlfZGF0YS5nZXRWYXIoJ1NBTklUWV9WRVJTSU9OJywgVHJ1ZSkgb3IgMSkK ICAgICBuZXR3b3JrX2Vycm9yID0gRmFsc2UKLSAgICBpZiBsYXN0X3Nhbml0eV92ZXJzaW9uIDwg c2FuaXR5X3ZlcnNpb246IAorICAgICMgTkFUSVZFTFNCU1RSSU5HIHZhciBtYXkgaGF2ZSBiZWVu IG92ZXJyaWRkZW4gd2l0aCAidW5pdmVyc2FsIiwgc28KKyAgICAjIGdldCBhY3R1YWwgaG9zdCBk aXN0cmlidXRpb24gaWQgYW5kIHZlcnNpb24KKyAgICBuYXRpdmVsc2JzdHIgPSBsc2JfZGlzdHJv X2lkZW50aWZpZXIoc2FuaXR5X2RhdGEpCisgICAgaWYgbGFzdF9zYW5pdHlfdmVyc2lvbiA8IHNh bml0eV92ZXJzaW9uIG9yIGxhc3RfbmF0aXZlbHNic3RyICE9IG5hdGl2ZWxzYnN0cjogCiAgICAg ICAgIGNoZWNrX3Nhbml0eV92ZXJzaW9uX2NoYW5nZShzdGF0dXMsIHNhbml0eV9kYXRhKQogICAg ICAgICBzdGF0dXMuYWRkcmVzdWx0KGNoZWNrX3Nhbml0eV9zc3RhdGVfZGlyX2NoYW5nZShzc3Rh dGVfZGlyLCBzYW5pdHlfZGF0YSkpCiAgICAgZWxzZTogCkBAIC05NjQsNiArOTcwLDcgQEAgZGVm IGNoZWNrX3Nhbml0eShzYW5pdHlfZGF0YSk6CiAgICAgICAgICAgICBmLndyaXRlKCJTQU5JVFlf VkVSU0lPTiAlc1xuIiAlIHNhbml0eV92ZXJzaW9uKSAKICAgICAgICAgICAgIGYud3JpdGUoIlRN UERJUiAlc1xuIiAlIHRtcGRpcikgCiAgICAgICAgICAgICBmLndyaXRlKCJTU1RBVEVfRElSICVz XG4iICUgc3N0YXRlX2RpcikgCisgICAgICAgICAgICBmLndyaXRlKCJOQVRJVkVMU0JTVFJJTkcg JXNcbiIgJSBuYXRpdmVsc2JzdHIpIAogCiAgICAgc2FuaXR5X2hhbmRsZV9hYmljaGFuZ2VzKHN0 YXR1cywgc2FuaXR5X2RhdGEpCiAK