10961 2017-01-24 22:25:06 +0000 Toolchain environment variables cleaned from environment during devtool usage 2017-02-16 21:14:41 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 2.3 x86 Multiple RESOLVED FIXED Medium+ major 2.3 M3 1 joshuagloe bluelightning poky.bs.watcher poky.watcher richard.purdie --- Regression (Used to work) Don't know oldest_to_newest 70102 0 3604 joshuagloe 2017-01-24 22:25:06 +0000 Created attachment 3604 Patch to demonstrate cleaned environment Environment variables (such as the`OE_` and `OECORE_` namespaced environment variables) are cleaned from the environment during devtool usage within an eSDK environment since the tinfoil2 changes. --- The easiest way to observe the issue is, on a machine with ccache installed (I'm using Fedora 24): * build an eSDK toolchain: $ bitbake core-image-minimal -c populate_sdk_ext * extract the generated toolchain somewhere: $ sh /srv/builds/poky/tmp/deploy/sdk/poky-glibc-x86_64-core-image- minimal-core2-64-toolchain-ext-2.2+snapshot.sh * modify enable_uninative() to dump values from the toolchain environment (as a demonstration of significant variables being cleaned): $ cd ~/poky_sdk/layers/poky $ patch -p1 ~/env-debug.patch <<attached to this bug>> * source the environment script and trigger enable_uninative call $ cd ~/poky_sdk $ source environment-setup-core2-64-poky-linux $ devtool add ~/Projects/swupd-client $ devtool build swupd-client Observe several "WARNING: OECORE_SDK_VERSION not in environment". --- The change was introduced after 209a19ce1666d55bf300a4bee80ed4f5eedc3691 and seems to have been introduced before a92d27acd8f42b61a2fc575311838b45d171f0af (the tinfoil2 series). The shortlog between the two refs is: The shortlog between the two branches is: Mariano Lopez (1): oeqa/utils/commands.py: Fix get_bb_vars() when called without arguments Ola x Nilsson (1): oe-pkgdata-util: Make read-value handle override variables Paul Eggleton (33): bitbake: data_smart: fix resetting of reference on variablehistory bitbake: knotty: make quiet option a level option bitbake: knotty: fix --observe-only option bitbake: server/xmlrpc: send back 503 response with correct encoding bitbake: data_smart: implement remote datastore functionality bitbake: command: provide a means to shut down from the client in memres mode bitbake: tinfoil: rewrite as a wrapper around the UI bitbake: remotedata: enable transporting datastore from the client to the server bitbake: tinfoil: implement server-side recipe parsing bitbake: tinfoil: pass datastore to server when expanding python references bitbake: cooker: allow buildFile warning to be hidden programmatically bitbake: data_smart: support serialisation bitbake: runqueue: enable setVariable command to affect task execution bitbake: siggen: add means of ignoring basehash mismatch bitbake: server/process: don't change UI process signal handler on terminate bitbake: lib/bb/build: enable access to logger within tasks lib/oe/rootfs: fix log_check warnings being printed twice with RPM packaging classes/image: suppress log_check mechanism for warnings/errors logged through BitBake devtool: package: don't try to initialise tinfoil twice lib/oe/recipeutils: use cooker function instead of bb.providers oe-selftest: make tinfoil quiet when using to start QEMU oe-selftest: use tinfoil.parse_recipe() devtool / recipetool: use tinfoil parsing API classes/base: fix license file checksumming when source not under TMPDIR classes/patch: move in logic to commit for additional tasks classes/patch: move several functions to oe.patch recipetool: add OE lib path devtool: fix extraction of source to work in memres mode lib/oe/recipeutils: drop parse_recipe_simple() devtool: extract: disable basehash mismatch errors devtool: prevent BBHandledException from showing traceback oe-selftest: devtool: improve test_devtool_modify slightly oe-selftest: add basic tinfoil tests 70213 1 3608 bluelightning 2017-01-26 21:58:16 +0000 Created attachment 3608 Patch to fix issue (for morty only) I've dug into this and the answer is these variables should always be cleaned at this point during execution - they get put back later. In morty and earlier we were taking a lot of shortcuts to start up parts of bitbake, and one of those shortcuts was not cleaning the environment like bitbake normally does. Tinfoil2 didn't mask the issue, it fixed it - because it now uses most of the same code paths that bitbake normally does. The reason we never had a problem with this before is that prior to http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=morty&id=8108c477405071ae40710d94a1148be8dacff063 , as far as I'm aware we didn't have anything where the behaviour was affected by the uncleaned environment. The actual problem for morty isn't described above - here's what you see when trying the instructions there on a system where ccache is installed (Fedora 25 in my case): ---------- snip ---------- ccache: error: Could not find compiler "gcc" in PATH ERROR: Error running gcc --version: Traceback (most recent call last): File "/home/paul/poky_sdk_morty/layers/poky/meta/lib/oe/utils.py", line 241, in host_gcc_version output = subprocess.check_output("%s --version" % compiler, shell=True, env=env).decode("utf-8") File "/home/paul/poky_sdk_morty/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 626, in check_output **kwargs).stdout File "/home/paul/poky_sdk_morty/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 708, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command 'gcc --version' returned non-zero exit status 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/cookerdata.py", line 271, in parseBaseConfiguration bb.event.fire(bb.event.ConfigParsed(), self.data) File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line 201, in fire fire_class_handlers(event, d) File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line 124, in fire_class_handlers execute_handler(name, handler, event, d) File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line 96, in execute_handler ret = handler(event) File "/home/paul/poky_sdk_morty/layers/poky/meta/classes/uninative.bbclass", line 85, in uninative_event_enable enable_uninative(d) File "/home/paul/poky_sdk_morty/layers/poky/meta/classes/uninative.bbclass", line 97, in enable_uninative d.setVar("NATIVELSBSTRING", "universal%s" % oe.utils.host_gcc_version(d)) File "/home/paul/poky_sdk_morty/layers/poky/meta/lib/oe/utils.py", line 243, in host_gcc_version bb.fatal("Error running %s --version: %s" % (compiler, e.output.decode("utf-8"))) File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/__init__.py", line 104, in fatal raise BBHandledException() bb.BBHandledException ---------- snip ---------- I've attached a morty-specific fix for this which I'll send out soon assuming nobody objects. 70224 2 joshuagloe 2017-01-27 09:18:50 +0000 (In reply to comment #1) > Created attachment 3608 [details] > Patch to fix issue (for morty only) > > I've dug into this and the answer is these variables should always be > cleaned at this point during execution - they get put back later. In morty > and earlier we were taking a lot of shortcuts to start up parts of bitbake, > and one of those shortcuts was not cleaning the environment like bitbake > normally does. Tinfoil2 didn't mask the issue, it fixed it - because it now > uses most of the same code paths that bitbake normally does. The reason we > never had a problem with this before is that prior to > http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/ > ?h=morty&id=8108c477405071ae40710d94a1148be8dacff063 , as far as I'm aware > we didn't have anything where the behaviour was affected by the uncleaned > environment. Thanks for digging in and explaining! > The actual problem for morty isn't described above - here's what you see > when trying the instructions there on a system where ccache is installed > (Fedora 25 in my case): > > ---------- snip ---------- > ccache: error: Could not find compiler "gcc" in PATH > ERROR: Error running gcc --version: > Traceback (most recent call last): > File "/home/paul/poky_sdk_morty/layers/poky/meta/lib/oe/utils.py", line > 241, in host_gcc_version > output = subprocess.check_output("%s --version" % compiler, shell=True, > env=env).decode("utf-8") > File > "/home/paul/poky_sdk_morty/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/ > python3.5/subprocess.py", line 626, in check_output > **kwargs).stdout > File > "/home/paul/poky_sdk_morty/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/ > python3.5/subprocess.py", line 708, in run > output=stdout, stderr=stderr) > subprocess.CalledProcessError: Command 'gcc --version' returned non-zero > exit status 1 > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/cookerdata.py", > line 271, in parseBaseConfiguration > bb.event.fire(bb.event.ConfigParsed(), self.data) > File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line > 201, in fire > fire_class_handlers(event, d) > File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line > 124, in fire_class_handlers > execute_handler(name, handler, event, d) > File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line > 96, in execute_handler > ret = handler(event) > File > "/home/paul/poky_sdk_morty/layers/poky/meta/classes/uninative.bbclass", line > 85, in uninative_event_enable > enable_uninative(d) > File > "/home/paul/poky_sdk_morty/layers/poky/meta/classes/uninative.bbclass", line > 97, in enable_uninative > d.setVar("NATIVELSBSTRING", "universal%s" % oe.utils.host_gcc_version(d)) > File "/home/paul/poky_sdk_morty/layers/poky/meta/lib/oe/utils.py", line > 243, in host_gcc_version > bb.fatal("Error running %s --version: %s" % (compiler, > e.output.decode("utf-8"))) > File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/__init__.py", > line 104, in fatal > raise BBHandledException() > bb.BBHandledException > ---------- snip ---------- > > I've attached a morty-specific fix for this which I'll send out soon > assuming nobody objects. No objections from me. I'd sent a workaround to master to remove CCACHE_PATH from toolchain-scripts[1] and was going to propose backporting that to morty. Any thoughts on whether that change is a) desirable in general (it's not clear from the commit message *why* CCACHE_PATH was added to toolchain-scripts) and b) if so, whether it should still be backported? 1. https://patchwork.openembedded.org/patch/136304/ 70777 3 richard.purdie 2017-02-16 21:14:41 +0000 A patch for the is morty was merged a per the patch from Paul. The toolchain-scripts.bbclass change from Joshua was also merged: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=0eafcc05d83da3a0add0d116fe239b528c144f15. I believe the various issues have therefore been resolved. 3604 2017-01-24 22:25:06 +0000 2017-01-24 22:25:06 +0000 Patch to demonstrate cleaned environment env-debug.patch text/plain 572 joshuagloe ZGlmZiAtLWdpdCBhL21ldGEvY2xhc3Nlcy91bmluYXRpdmUuYmJjbGFzcyBiL21ldGEvY2xhc3Nl cy91bmluYXRpdmUuYmJjbGFzcwppbmRleCA0MTBmYjcyLi5hMmU4ZDhkIDEwMDY0NAotLS0gYS9t ZXRhL2NsYXNzZXMvdW5pbmF0aXZlLmJiY2xhc3MKKysrIGIvbWV0YS9jbGFzc2VzL3VuaW5hdGl2 ZS5iYmNsYXNzCkBAIC04Niw2ICs4NiwxMSBAQCBweXRob24gdW5pbmF0aXZlX2V2ZW50X2VuYWJs ZSgpIHsKIH0KIAogZGVmIGVuYWJsZV91bmluYXRpdmUoZCk6CisgICAgdmVyID0gb3MuZW52aXJv bi5nZXQoIk9FQ09SRV9TREtfVkVSU0lPTiIsICIiKQorICAgIGlmIG5vdCB2ZXI6CisgICAgICAg IGJiLndhcm4oIk9FQ09SRV9TREtfVkVSU0lPTiBub3QgaW4gZW52aXJvbm1lbnQiKQorICAgIGVs c2U6CisgICAgICAgIGJiLm5vdGUoIk9FQ09SRV9TREtfVkVSU0lPTiBpcyAlcyIgJSB2ZXIpCiAg ICAgbG9hZGVyID0gZC5nZXRWYXIoIlVOSU5BVElWRV9MT0FERVIiKQogICAgIGlmIG9zLnBhdGgu ZXhpc3RzKGxvYWRlcik6CiAgICAgICAgIGJiLmRlYnVnKDIsICJFbmFibGluZyB1bmluYXRpdmUi KQo= 3608 2017-01-26 21:58:16 +0000 2017-01-26 21:58:16 +0000 Patch to fix issue (for morty only) 0001-tinfoil-clean-environment-when-starting-up-cooker.patch text/plain 2282 bluelightning RnJvbSA1NWI0ODdhMDc1YjZhNzBjNmViOWJkYzAxMTlmYjkyYmVhZjY0ODAzIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBQYXVsIEVnZ2xldG9uIDxwYXVsLmVnZ2xldG9uQGxpbnV4Lmlu dGVsLmNvbT4KRGF0ZTogRnJpLCAyNyBKYW4gMjAxNyAxMDozOTozMiArMTMwMApTdWJqZWN0OiBb UEFUQ0hdIHRpbmZvaWw6IGNsZWFuIGVudmlyb25tZW50IHdoZW4gc3RhcnRpbmcgdXAgY29va2Vy CgpEdXJpbmcgbm9ybWFsIGJpdGJha2UgZXhlY3V0aW9uLCB0aGUgZW52aXJvbm1lbnQgaXMgY2xl YW5lZCBvZiB2YXJpYWJsZXMKbm90IG9uIGEgd2hpdGVsaXN0IHdoaWxlIHN0YXJ0aW5nIHVwIGNv b2tlciwgYW5kIHRoZW4gcmVzdG9yZWQKYWZ0ZXJ3YXJkcy4gUHJpb3IgdG8gdGhlIHRpbmZvaWwy IHJld29yayBpbiBtYXN0ZXIgd2Ugd2VyZSB0YWtpbmcgYQpudW1iZXIgb2Ygc2hvcnRjdXRzIHdp dGhpbiB0aW5mb2lsIGFuZCBvbmUgb2YgdGhvc2Ugd2FzIG5vdCBkb2luZyB0aGlzCmVudmlyb25t ZW50IGNsZWFuaW5nLiBIb3dldmVyLCBwcmlvciB0byBPRS1Db3JlIHJldgozZDM5Y2E1YzkxZGJi NjJmYjQzMTk5ZjkxNmJkMzkwY2Q2MjEyZTNkIHdlIGRpZG4ndCBoYXZlCmFueSBjb2RlIChhcyBm YXIgYXMgSSdtIGF3YXJlKSB0aGF0IHdhcyBhZmZlY3RlZCBieSB0aGlzIHNob3J0Y3V0LCBoZW5j ZQp3aHkgdGhpcyB3YXNuJ3QgYW4gaXNzdWUgdXAgdG8gbm93LgoKVGhlIHJlc3VsdCBpcyB0aGUg Zm9sbG93aW5nIGVycm9yIHdoZW4gYXR0ZW1wdGluZyB0byBydW4gImRldnRvb2wgYnVpbGQiCmlu IHRoZSBlU0RLLCBhcyBDQ0FDSEVfUEFUSCBpcyBhbGxvd2VkIHRocm91Z2ggZnJvbSB0aGUgZVNE SydzCmVudmlyb25tZW50IHNldHVwIHNjcmlwdDoKCi0tLS0tLS0tLS0tIHNuaXAgLS0tLS0tLS0t LS0KY2NhY2hlOiBlcnJvcjogQ291bGQgbm90IGZpbmQgY29tcGlsZXIgImdjYyIgaW4gUEFUSAou Li4Kc3VicHJvY2Vzcy5DYWxsZWRQcm9jZXNzRXJyb3I6IENvbW1hbmQgJ2djYyAgLS12ZXJzaW9u JyByZXR1cm5lZApub24temVybyBleGl0IHN0YXR1cyAxCi0tLS0tLS0tLS0tIHNuaXAgLS0tLS0t LS0tLS0KCldlIGNhbiBmaXggdGhpcyBieSBzaW1wbHkgZG9pbmcgdGhlIGVudmlyb25tZW50IGZp bHRlcmluZyB3aGlsZSB3ZSBhcmUKc3RhcnRpbmcgdXAgY29va2VyLCB0aHVzIHRoZSBlbnZpcm9u bWVudCB3aGVuIHVuaW5hdGl2ZS5iYmNsYXNzIGNvbWVzIHRvCmRvIHRoZSBnY2MgdmVyc2lvbiBj aGVjayBpbiBmYjk0NWMwZmQyZTY2ZDcwNDYxZTZjZjJlNjAyMDIwZWVhYmUzMmY3Cml0IGlzIG5v dCBhZmZlY3RlZCBieSBDQ0FDSEVfUEFUSCBvciBvdGhlciB2YXJpYWJsZXMgaW4gdGhlIGV4dGVy bmFsCmVudmlyb25tZW50IHRoYXQgc2hvdWxkIGhhdmUgYmVlbiBmaWx0ZXJlZCBvdXQuCgpGaXhl cyBbWU9DVE8gIzEwOTYxXS4KClNpZ25lZC1vZmYtYnk6IFBhdWwgRWdnbGV0b24gPHBhdWwuZWdn bGV0b25AbGludXguaW50ZWwuY29tPgotLS0KIGJpdGJha2UvbGliL2JiL3RpbmZvaWwucHkgfCAz ICsrKwogMSBmaWxlIGNoYW5nZWQsIDMgaW5zZXJ0aW9ucygrKQoKZGlmZiAtLWdpdCBhL2JpdGJh a2UvbGliL2JiL3RpbmZvaWwucHkgYi9iaXRiYWtlL2xpYi9iYi90aW5mb2lsLnB5CmluZGV4IDg4 OTllODYuLjlmYTViNWIgMTAwNjQ0Ci0tLSBhL2JpdGJha2UvbGliL2JiL3RpbmZvaWwucHkKKysr IGIvYml0YmFrZS9saWIvYmIvdGluZm9pbC5weQpAQCAtNTEsMTAgKzUxLDEzIEBAIGNsYXNzIFRp bmZvaWw6CiAgICAgICAgIGZlYXR1cmVzID0gW10KICAgICAgICAgaWYgdHJhY2tpbmc6CiAgICAg ICAgICAgICBmZWF0dXJlcy5hcHBlbmQoQ29va2VyRmVhdHVyZXMuQkFTRURBVEFTVE9SRV9UUkFD S0lORykKKyAgICAgICAgY2xlYW5lZHZhcnMgPSBiYi51dGlscy5jbGVhbl9lbnZpcm9ubWVudCgp CiAgICAgICAgIHNlbGYuY29va2VyID0gQkJDb29rZXIoc2VsZi5jb25maWcsIGZlYXR1cmVzKQog ICAgICAgICBzZWxmLmNvbmZpZ19kYXRhID0gc2VsZi5jb29rZXIuZGF0YQogICAgICAgICBiYi5w cm92aWRlcnMubG9nZ2VyLnNldExldmVsKGxvZ2dpbmcuRVJST1IpCiAgICAgICAgIHNlbGYuY29v a2VyX2RhdGEgPSBOb25lCisgICAgICAgIGZvciBrIGluIGNsZWFuZWR2YXJzOgorICAgICAgICAg ICAgb3MuZW52aXJvbltrXSA9IGNsZWFuZWR2YXJzW2tdCiAKICAgICBkZWYgcmVnaXN0ZXJfaWRs ZV9mdW5jdGlvbihzZWxmLCBmdW5jdGlvbiwgZGF0YSk6CiAgICAgICAgIHBhc3MKLS0gCjIuOS4z Cgo=