13824 2020-03-05 02:04:03 +0000 slow loading cache 2020-04-26 06:02:05 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED FIXED Medium+ normal 3.2 M1 1 153340508 richard.purdie akuster poky.bs.watcher poky.watcher --- --- Don't know oldest_to_newest 86661 0 153340508 2020-03-05 02:04:03 +0000 In our project,we have over 4000 patches and 300hundred bb's. Then loading cache in incremental build is very slow. I figured out that it is due to a problem algorithm in cache.py And i have give out a improvement. diff --git a/poky-sumo-19.0.1/bitbake/lib/bb/cache.py b/poky-sumo-19.0.1/bitbake/lib/bb/cache.py old mode 100644 new mode 100755 index 168a77ac0..753d074f7 --- a/poky-sumo-19.0.1/bitbake/lib/bb/cache.py +++ b/poky-sumo-19.0.1/bitbake/lib/bb/cache.py @@ -610,31 +610,38 @@ class Cache(NoCache): fn, f) self.remove(fn) return False - + import re if hasattr(info_array[0], 'file_checksums'): for _, fl in info_array[0].file_checksums.items(): fl = fl.strip() - while fl: - # A .split() would be simpler but means spaces or colons in filenames would break - a = fl.find(":True") - b = fl.find(":False") - if ((a < 0) and b) or ((b > 0) and (b < a)): - f = fl[:b+6] - fl = fl[b+7:] - elif ((b < 0) and a) or ((a > 0) and (a < b)): - f = fl[:a+5] - fl = fl[a+6:] - else: - break - fl = fl.strip() - if "*" in f: - continue - f, exist = f.split(":") - if (exist == "True" and not os.path.exists(f)) or (exist == "False" and os.path.exists(f)): - logger.debug(2, "Cache: %s's file checksum list file %s changed", - fn, f) + flist=re.split(r'(:True|:False)',fl) + for i in range(len(flist)): + if ( ":True" == flist[i].strip() and not os.path.exists(flist[i-1].strip())) or ( ":False" == flist[i].strip() and os.path.exists(flist[i-1].strip())): + logger.error(2, "Cache: %s's file checksum list file %s changed", + fn, flist[i-1]) self.remove(fn) return False + #while fl: + # A .split() would be simpler but means spaces or colons in filenames would break + #a = fl.find(":True") + #b = fl.find(":False") + #if ((a < 0) and b) or ((b > 0) and (b < a)): + #f = fl[:b+6] + #fl = fl[b+7:] + #elif ((b < 0) and a) or ((a > 0) and (a < b)): + #f = fl[:a+5] + #fl = fl[a+6:] + #else: + #break + #fl = fl.strip() + #if "*" in f: + #continue + #f, exist = f.split(":") + #if (exist == "True" and not os.path.exists(f)) or (exist == "False" and os.path.exists(f)): + #logger.debug(2, "Cache: %s's file checksum list file %s changed", + #fn, f) + #self.remove(fn) + #return False if appends != info_array[0].appends: logger.debug(2, "Cache: appends for %s changed", fn) 86662 1 richard.purdie 2020-03-05 13:16:05 +0000 Hi, Thanks for this. Could you describe a little bit more what the problem is and how this change fixes it please? 86663 2 153340508 2020-03-05 13:23:20 +0000 (In reply to comment #1) > Hi, > > Thanks for this. Could you describe a little bit more what the problem is > and how this change fixes it please? Hello Richard, thanks for replying me. The old method continually cut the "fl" string, while in python, this cut method takes O(n) time complexity,so in worst case ,it would take O(N^2) in total for loading cache process. In my compoany, we have over 4000 patches and in increamental build, "loading cache" takes about 3 min in my 32U128G machine. while the new method takes about 15seconds. 86664 3 153340508 2020-03-05 13:25:18 +0000 (In reply to comment #1) > Hi, > > Thanks for this. Could you describe a little bit more what the problem is > and how this change fixes it please? It happens in in incremental build when the termianl shows "loading cache ,loaded XXX entries form cache" 86673 4 153340508 2020-03-06 01:04:53 +0000 (In reply to comment #2) > (In reply to comment #1) > > Hi, > > > > Thanks for this. Could you describe a little bit more what the problem is > > and how this change fixes it please? > > Hello Richard, thanks for replying me. > The old method continually cut the "fl" string, while in python, this cut > method takes O(n) time complexity,so in worst case ,it would take O(N^2) in > total for loading cache process. In my compoany, we have over 4000 patches > and in increamental build, "loading cache" takes about 3 min in my 32U128G > machine. while the new method takes about 15seconds. this change fixes old algorithm from time complexity from O(n^2) to O(n). the improvemtnt is extremely big especially in big project. Kernel patches usually has a number of over 1000. So doing loading cache would save a lot of time 86678 5 richard.purdie 2020-03-06 22:04:02 +0000 Thanks for explaining, that helps and I can see the issue. I am a little concerned the case handing "*" is lost, I'm therefore worried there may be some corner cases which aren't handled correctly. We should also remove the commented code rather than just commenting it out. I appreciate you're constrained upon how you can share patches, do you want to fix up the patch here with a commit message and signed-off-by or does someone else need to do that. I am grateful you have reported the problem as it does sound like a good fix! 86687 6 153340508 2020-03-09 07:54:11 +0000 (In reply to comment #5) > Thanks for explaining, that helps and I can see the issue. > > I am a little concerned the case handing "*" is lost, I'm therefore worried > there may be some corner cases which aren't handled correctly. We should > also remove the commented code rather than just commenting it out. > > I appreciate you're constrained upon how you can share patches, do you want > to fix up the patch here with a commit message and signed-off-by or does > someone else need to do that. I am grateful you have reported the problem as > it does sound like a good fix! Hello,I have commit this fix. https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=happystudy-contrib/master&id=8c9e646c0f5ecc7d239f518e2e6ef3be2f312dbe In re module, i have difined the split symbol as True and False, so if there are "*" or blank or other strange characoter in filename,it won't be splited. So this method does not change the original idea in old method. Actually ,in my company, we have strict rules on filename, so we can simply use A.split(). BTW, i think it is better we have some rules on filenames for a well-trained engineer. XD 86816 7 153340508 2020-03-26 05:02:11 +0000 (In reply to comment #5) > Thanks for explaining, that helps and I can see the issue. > > I am a little concerned the case handing "*" is lost, I'm therefore worried > there may be some corner cases which aren't handled correctly. We should > also remove the commented code rather than just commenting it out. > > I appreciate you're constrained upon how you can share patches, do you want > to fix up the patch here with a commit message and signed-off-by or does > someone else need to do that. I am grateful you have reported the problem as > it does sound like a good fix! Hello Richard, Do you have any more ideas on my commit https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=happystudy-contrib/master&id=8c9e646c0f5ecc7d239f518e2e6ef3be2f312dbe. I know you are worrying about corner cases. I believe my fix has same function as the old one but faster,which i have explained before.And this will influence people building big project. If you find any more corner cases please do not hesiatate to contact me .I would be glad to work with you guys to improve YOCTO. 86936 8 richard.purdie 2020-04-08 15:25:20 +0000 I have queued a modified version of this in master-next. I modified it to use a more advanced regex, retain the "*" check and cache the regex. 87085 9 richard.purdie 2020-04-26 06:02:05 +0000 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=8b1636763d6df9d68a50ed017ff0ed5f358b7321