Bug 10680 - multiconfig globbing support
Summary: multiconfig globbing support
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 2.3
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 2.2.2
Assignee: Juro Bystricky
QA Contact: Jair Gonzalez
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2016-11-16 13:04 UTC by Richard Purdie
Modified: 2017-03-19 20:22 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Purdie 2016-11-16 13:04:15 UTC
Currently you can't build a target for all the configured multiconfigs without specifying a list. This enhancement is to support globbing so that you can do this, e.g. instead of:

bitbake multiconfig:A:bash multiconfig:B:bash bash

bitbake multiconfig:*:bash
Comment 1 Juro Bystricky 2016-12-21 01:23:12 UTC
Currently testing this patch:
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 30131fb..a6841e7 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -666,7 +666,23 @@ class BBCooker:
         if not task.startswith("do_"):
             task = "do_%s" % task

-        fulltargetlist = self.checkPackages(pkgs_to_build, task)
+        targetlist = self.checkPackages(pkgs_to_build, task)
+        fulltargetlist = []
+
+        # Wild card expansion:
+        # Replace string such as "multiconfig:*:bash"
+        # into "multiconfig:A:bash multiconfig:B:bash bash"
+        for k in targetlist:
+            if k.startswith("multiconfig:") and k.split(":")[1] == "*":
+                for mc in self.multiconfigs:
+                    if mc:
+                        fulltargetlist.append(k.replace('*', mc))
+                    # default task
+                    else:
+                        fulltargetlist.append(k.split(":")[2])
+            else:
+                fulltargetlist.append(k)
+
         taskdata = {}
         localdata = {}
Comment 2 Richard Purdie 2016-12-22 14:47:48 UTC
Patch looks good to me, please send to the bitbake list and we can get this closed.
Comment 3 Juro Bystricky 2016-12-22 16:02:02 UTC
Before I send this to the mailing list, I have one suggestion:

We do this:

BB_MULTICONFIG = "A B"

$ bitbake multiconfig:*:bash 

expands to:

$ bitbake multiconfig:A:bash multiconfig:B:bash bash

But what to do in this case, something like this:

$ bitbake multiconfig:*:bash core-image-minimal

I suggest we expand it like this:


$ bitbake multiconfig:A:bash multiconfig:B:bash core-image-minimal

In other words, if "default" is explicitly specified, we don't override it.
This would make the syntax more flexible as well.
Comment 4 Juro Bystricky 2016-12-22 16:45:51 UTC
In particular, I would find syntax support for this very useful:

$ bitbake multiconfig:*:meta-toolchain hosttols

(building a set of baremetal toolchains + hostools)
Comment 5 Juro Bystricky 2017-01-13 00:05:16 UTC
I ran into a few real world use cases where it is desirable to use multiconfig with two different tasks (recipes). For example, I frequently build SDKs with several toolchains + additional host tools. Another use is to build one image for main CPU(s) and a different co-image for a companion CPU.
The additional logic is simple: if the user specified the "default" task explicitly, use it, otherwise assume the "default" task to be the same as the multiconfig task(s).


Currently supported multiconfig sytax:
BBMULTICONFIG = "A B C ..."

$ bitbake multiconfig:A:task1 multiconfig:B:task2 multiconfig:C:task3 ... taskN

(where task1, task2,...,taskN,... may or may not be the same.)

Suported is also any subset of tasks, i.e.:

$ bitbake multiconfig:A:task1
$ bitbake multiconfig:A:task1 taskM
$ bitbake multiconfig:A:task1 multiconfig:B:task2
$ bitbake taskN


Proposed new syntax addition implemented in this patch (two formats):

1. Default task implicit:

$ bitbake multiconfig:*:task

expands as:

$ bitbake multiconfig:A:task multiconfig:B:task multiconfig:C:task ... task

2. Default task explicit

$ bitbake multiconfig:*:task1 task2

expands as:

$ bitbake multiconfig:A:task1 multiconfig:B:task1 multiconfig:C:task1 ... task2

This patch should allow all of the above:

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index f6bad36..bcd4dca 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -666,7 +666,37 @@ class BBCooker:
         if not task.startswith("do_"):
             task = "do_%s" % task
 
-        fulltargetlist = self.checkPackages(pkgs_to_build, task)
+        targetlist = self.checkPackages(pkgs_to_build, task)
+        fulltargetlist = []
+        defaulttask_implicit = ''
+        defaulttask_explicit = False
+        wildcard = False
+
+        # Wild card expansion:
+        # Replace string such as "multiconfig:*:bash"
+        # into "multiconfig:A:bash multiconfig:B:bash bash"
+        for k in targetlist:
+            if k.startswith("multiconfig:"):
+                if wildcard:
+                    bb.fatal('multiconfig conflict')
+                if k.split(":")[1] == "*":
+                    wildcard = True
+                    for mc in self.multiconfigs:
+                        if mc:
+                            fulltargetlist.append(k.replace('*', mc))
+                        # implicit default task
+                        else:
+                            defaulttask_implicit = k.split(":")[2]
+                else:
+                    fulltargetlist.append(k)
+            else:
+                defaulttask_explicit = True
+                fulltargetlist.append(k)
+
+        if not defaulttask_explicit and defaulttask_implicit != '':
+            fulltargetlist.append(defaulttask_implicit)
+
+        bb.debug(1,"fulltargetlist: %s" % (str(fulltargetlist)))
         taskdata = {}
         localdata = {}
Comment 6 Juro Bystricky 2017-01-13 00:09:22 UTC
It takes forever to test this properly....
Comment 7 Juro Bystricky 2017-01-15 20:42:44 UTC
sent patch to the mailing list:
http://lists.openembedded.org/pipermail/bitbake-devel/2017-January/008395.html
Comment 9 Henry Bruce 2017-03-15 22:13:41 UTC
multiconfig is a powerful new feature whose usability (and thus rate of adoption) is greatly enhanced by the globbing feature. Requesting backport for morty so that those not ready to jump on pyro can benefit form this enhancement.
Comment 10 Stephen K Jolley 2017-03-16 15:33:00 UTC
Triage team agreed to backport to 2.2.2.