| Summary: | Toolchain environment variables cleaned from environment during devtool usage | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Joshua Lock <joshuagloe> | ||||||
| Component: | bitbake | Assignee: | Paul Eggleton <bluelightning> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | major | ||||||||
| Priority: | Medium+ | CC: | poky.bs.watcher, poky.watcher, richard.purdie | ||||||
| Version: | 2.3 | ||||||||
| Target Milestone: | 2.3 M3 | ||||||||
| Hardware: | x86 | ||||||||
| OS: | Multiple | ||||||||
| Whiteboard: | |||||||||
| OS type for building Yocto: | --- | Type of Regression: | Regression (Used to work) | ||||||
| Verified: | Documentation change: | Don't know | |||||||
| Attachments: |
|
||||||||
Created attachment 3608 [details] Patch to fix issue (for morty only) I've dug into this and the answer is these variables should always be cleaned at this point during execution - they get put back later. In morty and earlier we were taking a lot of shortcuts to start up parts of bitbake, and one of those shortcuts was not cleaning the environment like bitbake normally does. Tinfoil2 didn't mask the issue, it fixed it - because it now uses most of the same code paths that bitbake normally does. The reason we never had a problem with this before is that prior to http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=morty&id=8108c477405071ae40710d94a1148be8dacff063 , as far as I'm aware we didn't have anything where the behaviour was affected by the uncleaned environment. The actual problem for morty isn't described above - here's what you see when trying the instructions there on a system where ccache is installed (Fedora 25 in my case): ---------- snip ---------- ccache: error: Could not find compiler "gcc" in PATH ERROR: Error running gcc --version: Traceback (most recent call last): File "/home/paul/poky_sdk_morty/layers/poky/meta/lib/oe/utils.py", line 241, in host_gcc_version output = subprocess.check_output("%s --version" % compiler, shell=True, env=env).decode("utf-8") File "/home/paul/poky_sdk_morty/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 626, in check_output **kwargs).stdout File "/home/paul/poky_sdk_morty/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 708, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command 'gcc --version' returned non-zero exit status 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/cookerdata.py", line 271, in parseBaseConfiguration bb.event.fire(bb.event.ConfigParsed(), self.data) File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line 201, in fire fire_class_handlers(event, d) File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line 124, in fire_class_handlers execute_handler(name, handler, event, d) File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line 96, in execute_handler ret = handler(event) File "/home/paul/poky_sdk_morty/layers/poky/meta/classes/uninative.bbclass", line 85, in uninative_event_enable enable_uninative(d) File "/home/paul/poky_sdk_morty/layers/poky/meta/classes/uninative.bbclass", line 97, in enable_uninative d.setVar("NATIVELSBSTRING", "universal%s" % oe.utils.host_gcc_version(d)) File "/home/paul/poky_sdk_morty/layers/poky/meta/lib/oe/utils.py", line 243, in host_gcc_version bb.fatal("Error running %s --version: %s" % (compiler, e.output.decode("utf-8"))) File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/__init__.py", line 104, in fatal raise BBHandledException() bb.BBHandledException ---------- snip ---------- I've attached a morty-specific fix for this which I'll send out soon assuming nobody objects. (In reply to comment #1) > Created attachment 3608 [details] > Patch to fix issue (for morty only) > > I've dug into this and the answer is these variables should always be > cleaned at this point during execution - they get put back later. In morty > and earlier we were taking a lot of shortcuts to start up parts of bitbake, > and one of those shortcuts was not cleaning the environment like bitbake > normally does. Tinfoil2 didn't mask the issue, it fixed it - because it now > uses most of the same code paths that bitbake normally does. The reason we > never had a problem with this before is that prior to > http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/ > ?h=morty&id=8108c477405071ae40710d94a1148be8dacff063 , as far as I'm aware > we didn't have anything where the behaviour was affected by the uncleaned > environment. Thanks for digging in and explaining! > The actual problem for morty isn't described above - here's what you see > when trying the instructions there on a system where ccache is installed > (Fedora 25 in my case): > > ---------- snip ---------- > ccache: error: Could not find compiler "gcc" in PATH > ERROR: Error running gcc --version: > Traceback (most recent call last): > File "/home/paul/poky_sdk_morty/layers/poky/meta/lib/oe/utils.py", line > 241, in host_gcc_version > output = subprocess.check_output("%s --version" % compiler, shell=True, > env=env).decode("utf-8") > File > "/home/paul/poky_sdk_morty/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/ > python3.5/subprocess.py", line 626, in check_output > **kwargs).stdout > File > "/home/paul/poky_sdk_morty/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/ > python3.5/subprocess.py", line 708, in run > output=stdout, stderr=stderr) > subprocess.CalledProcessError: Command 'gcc --version' returned non-zero > exit status 1 > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/cookerdata.py", > line 271, in parseBaseConfiguration > bb.event.fire(bb.event.ConfigParsed(), self.data) > File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line > 201, in fire > fire_class_handlers(event, d) > File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line > 124, in fire_class_handlers > execute_handler(name, handler, event, d) > File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/event.py", line > 96, in execute_handler > ret = handler(event) > File > "/home/paul/poky_sdk_morty/layers/poky/meta/classes/uninative.bbclass", line > 85, in uninative_event_enable > enable_uninative(d) > File > "/home/paul/poky_sdk_morty/layers/poky/meta/classes/uninative.bbclass", line > 97, in enable_uninative > d.setVar("NATIVELSBSTRING", "universal%s" % oe.utils.host_gcc_version(d)) > File "/home/paul/poky_sdk_morty/layers/poky/meta/lib/oe/utils.py", line > 243, in host_gcc_version > bb.fatal("Error running %s --version: %s" % (compiler, > e.output.decode("utf-8"))) > File "/home/paul/poky_sdk_morty/layers/poky/bitbake/lib/bb/__init__.py", > line 104, in fatal > raise BBHandledException() > bb.BBHandledException > ---------- snip ---------- > > I've attached a morty-specific fix for this which I'll send out soon > assuming nobody objects. No objections from me. I'd sent a workaround to master to remove CCACHE_PATH from toolchain-scripts[1] and was going to propose backporting that to morty. Any thoughts on whether that change is a) desirable in general (it's not clear from the commit message *why* CCACHE_PATH was added to toolchain-scripts) and b) if so, whether it should still be backported? 1. https://patchwork.openembedded.org/patch/136304/ A patch for the is morty was merged a per the patch from Paul. The toolchain-scripts.bbclass change from Joshua was also merged: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=0eafcc05d83da3a0add0d116fe239b528c144f15. I believe the various issues have therefore been resolved. |
Created attachment 3604 [details] Patch to demonstrate cleaned environment Environment variables (such as the`OE_` and `OECORE_` namespaced environment variables) are cleaned from the environment during devtool usage within an eSDK environment since the tinfoil2 changes. --- The easiest way to observe the issue is, on a machine with ccache installed (I'm using Fedora 24): * build an eSDK toolchain: $ bitbake core-image-minimal -c populate_sdk_ext * extract the generated toolchain somewhere: $ sh /srv/builds/poky/tmp/deploy/sdk/poky-glibc-x86_64-core-image- minimal-core2-64-toolchain-ext-2.2+snapshot.sh * modify enable_uninative() to dump values from the toolchain environment (as a demonstration of significant variables being cleaned): $ cd ~/poky_sdk/layers/poky $ patch -p1 ~/env-debug.patch <<attached to this bug>> * source the environment script and trigger enable_uninative call $ cd ~/poky_sdk $ source environment-setup-core2-64-poky-linux $ devtool add ~/Projects/swupd-client $ devtool build swupd-client Observe several "WARNING: OECORE_SDK_VERSION not in environment". --- The change was introduced after 209a19ce1666d55bf300a4bee80ed4f5eedc3691 and seems to have been introduced before a92d27acd8f42b61a2fc575311838b45d171f0af (the tinfoil2 series). The shortlog between the two refs is: The shortlog between the two branches is: Mariano Lopez (1): oeqa/utils/commands.py: Fix get_bb_vars() when called without arguments Ola x Nilsson (1): oe-pkgdata-util: Make read-value handle override variables Paul Eggleton (33): bitbake: data_smart: fix resetting of reference on variablehistory bitbake: knotty: make quiet option a level option bitbake: knotty: fix --observe-only option bitbake: server/xmlrpc: send back 503 response with correct encoding bitbake: data_smart: implement remote datastore functionality bitbake: command: provide a means to shut down from the client in memres mode bitbake: tinfoil: rewrite as a wrapper around the UI bitbake: remotedata: enable transporting datastore from the client to the server bitbake: tinfoil: implement server-side recipe parsing bitbake: tinfoil: pass datastore to server when expanding python references bitbake: cooker: allow buildFile warning to be hidden programmatically bitbake: data_smart: support serialisation bitbake: runqueue: enable setVariable command to affect task execution bitbake: siggen: add means of ignoring basehash mismatch bitbake: server/process: don't change UI process signal handler on terminate bitbake: lib/bb/build: enable access to logger within tasks lib/oe/rootfs: fix log_check warnings being printed twice with RPM packaging classes/image: suppress log_check mechanism for warnings/errors logged through BitBake devtool: package: don't try to initialise tinfoil twice lib/oe/recipeutils: use cooker function instead of bb.providers oe-selftest: make tinfoil quiet when using to start QEMU oe-selftest: use tinfoil.parse_recipe() devtool / recipetool: use tinfoil parsing API classes/base: fix license file checksumming when source not under TMPDIR classes/patch: move in logic to commit for additional tasks classes/patch: move several functions to oe.patch recipetool: add OE lib path devtool: fix extraction of source to work in memres mode lib/oe/recipeutils: drop parse_recipe_simple() devtool: extract: disable basehash mismatch errors devtool: prevent BBHandledException from showing traceback oe-selftest: devtool: improve test_devtool_modify slightly oe-selftest: add basic tinfoil tests