Bug 10748

Summary: Means of querying history for a single variable
Product: [Build System, Metadata & Runtime] BitBake Reporter: Paul Eggleton <bluelightning>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium+ CC: kergoth, mariano.lopez, poky.bs.watcher, poky.watcher, tvgamblin
Version: 2.2   
Target Milestone: 3.4 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)

Description Paul Eggleton 2016-11-30 20:01:41 UTC
bitbake -e is very useful, but it has a couple of drawbacks:

1) It lists absolutely everything, when most of the time what you want is the history for a single variable. People are used to piping the output through grep but that is only useful for a single variable, and sometimes not even that (see below). Usually I tell people to do what I do which is to pipe it through less and search for the variable, but then many people aren't even aware that less has search capabilities.

2) The final values of some commonly used variables such as DISTRO and MACHINE are excluded. I understand why that is - bitbake is showing you what the environment will look like and these variables are marked "unexport" so that is expected. However, that's not what most people want to see and it also makes it harder to search for the variable in the output.

It would be nice if we had a dedicated means (perhaps a separate tool) for querying the history / value of a variable. This would be trivial to implement with tinfoil.
Comment 1 Paul Eggleton 2016-11-30 20:02:31 UTC
s/only useful for a single variable/only useful in order to see the final value/
Comment 2 Paul Eggleton 2016-12-22 02:39:00 UTC
*** Bug 10830 has been marked as a duplicate of this bug. ***
Comment 3 Paul Eggleton 2016-12-22 02:40:14 UTC
A specific additional requirement derived from bug 10830 - have a mode that just prints the value of the variable or "" if there is no value.
Comment 4 Richard Purdie 2017-07-07 15:04:49 UTC
https://gist.github.com/kergoth/4133101

(15:02:57) RP: kergoth: I'm tempted to add that to bitbake itself, maybe as "bitbake-var" as users probably see this as variables, not an environment. Any thoughts on that?

kergoth: zero_note: bitbake-env went away in favor of a subcommand (show) of the 'bb' experimental sub-command based utility, which is currently broken and not yet fixed due to changes in tinfoil (bitbake's library used for external utilities). also neither of them supported the new variable tracking mechanism. but yes, i think
(16:10:59) kergoth: RP: it's not a bad idea. I'd still like to prototype integration of a sub-command based cli ui, but that can always be a future step, it's clear bitbake-env/bb-show provides value on its own, bitbake -e | grep is common and quite ugly
(16:17:57) RP: kergoth: agreed on a cli, I still have shell.py around as a partial inspiration for that and this wouldn't preclude that
(16:19:24) kergoth: zero_note, RP: I was never 100% happy with the interface of that tool, for what it's worth. for scripting, you really also want the ability to see *just the value* of the variable, rather than having to sed out the quotes and whatnot, but then what's the behavior if you set that and pass multiple variables? could just error, but that doesn't seem super clean.. also it would need variable tracking support added, and desperately needs the ability to
(16:19:24) kergoth:  accept FOO[bar] syntax to see specific flags rather than having to look at all flags for a specified variable..
(16:20:30) zero_note: imho, bitbake -e recipe | grep become usable only after several times of usage, when one knows how to customize the grep reg expression
(16:20:31) RP: kergoth: some kind of stripped output or value only shouldn't be too difficult and yes, those other things are valid issues
(16:21:06) kergoth: zero_note: yeah. you can write a shell function to semi-automate it, but only for the common case, it doesn't scale well, and is rather brittle
(16:25:34) kergoth: would also be nice to be able to get the unexpanded value without having to grep out the commented section, and a better way to extract the varaible tracking data for a particular variable in a more script-friendly format..
(16:25:54) kergoth: being limited to the shell-like output bitbake -e provides isn't ideal — there's a reason we did it that way, but..
(16:26:13) kergoth: at any rate, i do think we should integrate such a script, but i'd like to see a discussion about the details
(16:27:46) kergoth: the upside to the current behavior is the output is nearly in bitbake metadata format. that's a possibility, too, for the variable tracking — it could optionally emit the multiple bitbake format lines that led to the final value, rather than the final assignment itself
Comment 5 Richard Purdie 2021-04-16 15:03:07 UTC
A quick hacky implementation (which needs a fix to the tinfoil variable history emit function):

#! /usr/bin/env python3

import argparse
import io
import os
import sys

scripts_path = os.path.dirname(os.path.realpath(__file__))
lib_path = scripts_path + "/lib"
sys.path.insert(0, lib_path)
import scriptpath
scriptpath.add_bitbake_lib_path()
scriptpath.add_oe_lib_path()

import bb.tinfoil

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Bitbake Get Variable")
    parser.add_argument("variable", help="variable name to query")
    parser.add_argument("-r", "--recipe", help="recipe name", default=None, required=False)
    args = parser.parse_args()

    with bb.tinfoil.Tinfoil(tracking=True) as tinfoil:
        if not args.recipe:
            tinfoil.prepare(quiet=2, config_only=True)
            bb.data.emit_var(args.variable, d=tinfoil.config_data, all=True)
        else:
            tinfoil.prepare(quiet=2)
            d = tinfoil.parse_recipe(args.recipe)
            bb.data.emit_var(args.variable, d=d, all=True)
Comment 6 Richard Purdie 2021-04-16 17:12:01 UTC
I've sent out patches for this with a bit more cleanup