Bug 10945

Summary: data_smart.py: _remove causes extra work
Product: [Build System, Metadata & Runtime] BitBake Reporter: Patrick Ohly <patrick.ohly>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium CC: poky.bs.watcher, poky.watcher
Version: unspecified   
Target Milestone: 4.99   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Patrick Ohly 2017-01-20 10:39:56 UTC
getVarFlag() does some work related to _remove and the entire _remove multiple times. This is not wrong, but might have a performance impact.

There are two reasons for this:
- applying the split/join when there is nothing to remove
- removing when parsing=True

This can be reproduced with "bitbake -e qemu-native" and an entry in local.conf:
SRC_URI_remove_pn-foobar = "removeme"

It happens to break qemu-native because of an invalid _prepend without trailing space - that's how I noticed.

Here's a patch which might fix the issues. According to Richard, this needs good testing (bitbake selftest, parsing OE) because he has a feeling that getVarFlags() behaves this way for a reason.


diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 79d591a23..16840069b 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -772,7 +772,7 @@ class DataSmart(MutableMapping):
                 cachename = var + "[" + flag + "]"
             value = self.expand(value, cachename)
 
-        if value and flag == "_content" and local_var is not None and "_remove" in local_var:
+        if not parsing and value and flag == "_content" and local_var is not None and "_remove" in local_var:
             removes = []
             self.need_overrides()
             for (r, o) in local_var["_remove"]:
@@ -784,9 +784,10 @@ class DataSmart(MutableMapping):
                 if match:
                     removes.extend(self.expand(r).split())
 
-            filtered = filter(lambda v: v not in removes,
-                              value.split())
-            value = " ".join(filtered)
+            if removes:
+                filtered = filter(lambda v: v not in removes,
+                                  value.split())
+                value = " ".join(filtered)
             if expand and var in self.expand_cache:
                  # We need to ensure the expand cache has the correct value
                  # flag == "_content" here
Comment 1 Richard Purdie 2017-01-23 14:48:21 UTC
I did take part of the fix:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=45df694a9f472ac2f684aadac4d864c3dfdc48a7

since that part was obviously correct. The other piece (parsing=True) needs further investigation.
Comment 2 Richard Purdie 2018-10-18 12:42:17 UTC
Remaining optimisation added in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=58e3f45d4b7d69d663ce7c94933d4da7b5f071e3