Bug 12473 - Recipe cache not working for nested DataSmart objects (BB_ORIGENV)
Summary: Recipe cache not working for nested DataSmart objects (BB_ORIGENV)
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: All Multiple
: Medium major
Target Milestone: 4.99
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2018-01-11 14:26 UTC by dj
Modified: 2020-04-28 03:13 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description dj 2018-01-11 14:26:51 UTC
The recipe cache (activated by CACHE option) is not working if there are nested DataSmart objects. For example BB_ORIGENV is itself a DataSmart and also an entry in the main DataSmart object. The get_hash function calculates a hash based on the values of the DataSmart dict. For nested objects the value may not be persistent. In the case of the BB_ORIGENV the value contains the memory address of the object which is different each bitbake invocation. This causes the hash to change in each invocation. The hash is used in the file names of the cache files. Therefore the cache can never be used.

This problem exists in this project: https://github.com/openembedded/bitbake.git
Selectable versions in bugzilla don't match available tags in git so I didn't specify the version. The problem exists at least in the recent versions and on master.

Below patch seems to fix it. The idea is to make the get_hash function recursive in case of nested DataSmart objects and use their hash in the main hash. But I'm not sure if this is the best way to solve it. Someone more experienced with the bitbake implementation should take a look.

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 7b09af5..c0e4228 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -1014,7 +1014,10 @@ class DataSmart(MutableMapping):
                 continue

             value = d.getVar(key, False) or ""
-            data.update({key:value})
+            if type(value) is type(self):
+                data.update({key:value.get_hash()})
+            else:
+                data.update({key:value})

             varflags = d.getVarFlags(key, internalflags = True)
             if not varflags
Comment 1 Richard Purdie 2020-04-07 14:10:17 UTC
I did look into this and agree with your patch. We don't use BB_ORIGENV in the base configuration by default which is why we've not run into this.
Comment 2 dj 2020-04-08 11:32:55 UTC
Thank you for looking into this. Would it be possible for you to integrate this patch and release a new version? I'm not familiar with the processes in this project.
Comment 4 dj 2020-04-28 03:13:51 UTC
(In reply to comment #3)
> http://git.yoctoproject.org/cgit.cgi/poky/commit/
> ?id=98f9b05875c9a989490ad98b38f8f8bcaf0ca9d8

Thank you!