Bug 10945 - data_smart.py: _remove causes extra work
Summary: data_smart.py: _remove causes extra work
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium enhancement
Target Milestone: 4.99
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-01-20 10:39 UTC by Patrick Ohly
Modified: 2018-10-18 12:42 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Patrick Ohly 2017-01-20 10:39:56 UTC
getVarFlag() does some work related to _remove and the entire _remove multiple times. This is not wrong, but might have a performance impact.

There are two reasons for this:
- applying the split/join when there is nothing to remove
- removing when parsing=True

This can be reproduced with "bitbake -e qemu-native" and an entry in local.conf:
SRC_URI_remove_pn-foobar = "removeme"

It happens to break qemu-native because of an invalid _prepend without trailing space - that's how I noticed.

Here's a patch which might fix the issues. According to Richard, this needs good testing (bitbake selftest, parsing OE) because he has a feeling that getVarFlags() behaves this way for a reason.


diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 79d591a23..16840069b 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -772,7 +772,7 @@ class DataSmart(MutableMapping):
                 cachename = var + "[" + flag + "]"
             value = self.expand(value, cachename)
 
-        if value and flag == "_content" and local_var is not None and "_remove" in local_var:
+        if not parsing and value and flag == "_content" and local_var is not None and "_remove" in local_var:
             removes = []
             self.need_overrides()
             for (r, o) in local_var["_remove"]:
@@ -784,9 +784,10 @@ class DataSmart(MutableMapping):
                 if match:
                     removes.extend(self.expand(r).split())
 
-            filtered = filter(lambda v: v not in removes,
-                              value.split())
-            value = " ".join(filtered)
+            if removes:
+                filtered = filter(lambda v: v not in removes,
+                                  value.split())
+                value = " ".join(filtered)
             if expand and var in self.expand_cache:
                  # We need to ensure the expand cache has the correct value
                  # flag == "_content" here
Comment 1 Richard Purdie 2017-01-23 14:48:21 UTC
I did take part of the fix:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=45df694a9f472ac2f684aadac4d864c3dfdc48a7

since that part was obviously correct. The other piece (parsing=True) needs further investigation.
Comment 2 Richard Purdie 2018-10-18 12:42:17 UTC
Remaining optimisation added in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=58e3f45d4b7d69d663ce7c94933d4da7b5f071e3