| Summary: | build-perf:time for build "rootfs" is increased in around 20% on 2.3 M1 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Jose Perez C <jose.perez.carranza> | ||||
| Component: | bitbake | Assignee: | Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez> | ||||
| Status: | VERIFIED FIXED | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | High | CC: | benjamin.esquivel, markus.lehtonen, poky.bs.watcher, poky.watcher, richard.purdie, ross.burton | ||||
| Version: | unspecified | ||||||
| Target Milestone: | 2.3 M3 | ||||||
| Hardware: | x86 | ||||||
| OS: | Multiple | ||||||
| Whiteboard: | |||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||
| Attachments: |
|
||||||
|
Description
Jose Perez C
2017-01-05 19:54:12 UTC
These are the numbers between the two mentioned commits (first column is hash and second is mean time in seconds for bitbake core-image-sato -c rootfs with no tmp and cache and with sstates) morty:dc8508f609974cc99606b9042bfa7f870ce80228,130.61714285714 master:003b053498aea91fdb8c5418d0d8bbda7b23dbc0,131.2425 master:c3d2df883a9d6d5036277114339673656d89a728,130.65333333333 master:edb5b524b3ee807d55a437cbab3d768c5f6f2283,130.37583333333 master:1655d55764eb0241b71217810500fcf1d4519a1c,134.80666666667 master:43e652f3d1fee5ce7fad67e6400315eab1b34270,133.66333333333 master:09278be8c209976da03f63b5a956d212ac171317,137.30357142857 master:9b1bd29ea99aa0dec51e603e4ac28e211af19765,138 master:7c3a47ed8965c3a3eb90a9a4678d5caedbba6337,137.6578125 master:12a0ee049e453b6d0d2ce2f3fa981d1b6e02bd78,136.58818181818 master:71b7c3a88ec5a20d5e4531751d0bf23d5c94e494,148.5175 master:3d34ae6c11bc2e800f784740c54b2dd2029658ed,149.23333333333 master:9e63f81c78e284c9b325fe04a1b59e61c7ad8a1a,0 master:e38775a1d82e6dc60fc96cf243ecb94be964d9b2,0 master:18e4688bd52ed7746e7687b225da10f025cafc68,0 master:dae00fbb913c24b6bab00d5b99bd61f3455b4728,0 master:36e178a62f04e7f2611b26964efe46b9af97189a,0 master:a0374e92a82e8c31964163f2eefa471bfe1eb0d4,0 master:14539203655913b3304bf98d2dd1f2e1fed3d7ca,0 master:a92d27acd8f42b61a2fc575311838b45d171f0af,151.796 master:573c646d4cc62dcd0c230381df4940bdf314d495,155.664 The greatest delta between times is master:12a0ee049e453b6d0d2ce2f3fa981d1b6e02bd78,136.58818181818 master:71b7c3a88ec5a20d5e4531751d0bf23d5c94e494,148.5175 and if we do a git-log shortlog between these two $ git ol 12a0ee049e453b6d0d2ce2f3fa981d1b6e02bd78^..71b7c3a88ec5a20d5e4531751d0bf23d5c94e494 71b7c3a: Ross Burton: Mon Nov 28 15:49:15 2016 +0000: Revert "classes/populate_sdk_ext: require uninative" 5fd4cad: Ross Burton: Mon Nov 28 15:49:15 2016 +0000: conf: add C++ flags for uninative interoperatility d351c87: Jose Lamego: Mon Nov 28 14:50:05 2016 +0000: maintainers: Change maintainers for python(3) modules 12a0ee0: Scott Rifenbark: Wed Nov 23 11:10:35 2016 +0000: dev-manual: Added note about RPM not dealing with post-install meaning that either 71b7c3a or 5fd4cad are the culprits (because the other two are just documentation stuff) These are two results using the scripts/contrib/build-perf-test-wraper.sh for two consecutive commits bef4e00bdadcd912574f73fde93e58d3943b18b5 and 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d (http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d, with title 'python3: Build and package precompiled modules') --- ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5,uninative-1.4-783-gbef4e00,1:05:27.49,0:05:52.75,1:04:40.51,0\ :04:06.59,0:00:21.87,0:00:14.36,0:00:01.59,0:06:49.33,26731216,6467604,1442455,5332459 ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d,uninative-1.4-784-g3f2ab3f,1:26:48.03,0:14:00.12,1:06:15.55,0\ :04:34.95,0:00:21.83,0:00:14.41,0:00:01.54,0:07:32.28,26963776,6619312,1500925,5579739 --- Looking into the tmp size column created by core-image-sato, we see this tmp sizes (in unit of blocks, consisting of 1024 bytes) respectively * 26731216 * 26963776 meaning around 232k more blocks or 227Mbytes are due 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d, impacting the sshare-cache folders (not in the same proportion) size thus the current measure time in this bug ('rm -rf tmp cache; bitbake core-image-sato -c rootfs'). All commits following 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d basically have similar tmp sizes, sshare-cache and core-images-sato rootfs creation times. In resume, increase times on core-image-sato rootfs creation are due to larger python3 packages, increased considerably by 3f2ab3f. (In reply to comment #2) > ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5, > uninative-1.4-783-gbef4e00,1:05:27.49,0:05:52.75,1:04:40.51,0\ > :04:06.59,0:00:21.87,0:00:14.36,0:00:01.59,0:06:49.33,26731216,6467604, > 1442455,5332459 > > ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d, > uninative-1.4-784-g3f2ab3f,1:26:48.03,0:14:00.12,1:06:15.55,0\ > :04:34.95,0:00:21.83,0:00:14.41,0:00:01.54,0:07:32.28,26963776,6619312, > 1500925,5579739 Why does the overall build time go from 1:05 to 1:26 here? Makes me worry about the other measurements... RP detected something not yet understood: build times for 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d are increasing considerably (by 20 minutes) but the weird thing is that commit after the latter (which of course includes 3f2ab) drops again to previous times. Performance numbers for bef4e00bdadcd912574f73fde93e58d3943b18b5, 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d and 8a81facc23fd54b095940372e05de2356579c97c (run twice for each commit) ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5,uninative-1.4-783-gbef4e00,1:05:45.73,0:05:39.27,1:02:37.37,0:02:31.80,0:00:21.51,0:00:13.79,0:00:01.38,0:04:17.40,26734880,6464912,1442084,5331747 ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5,uninative-1.4-783-gbef4e00,1:05:27.49,0:05:52.75,1:04:40.51,0:04:06.59,0:00:21.87,0:00:14.36,0:00:01.59,0:06:49.33,26731216,6467604,1442455,5332459 ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d,uninative-1.4-784-g3f2ab3f,1:26:48.03,0:14:00.12,1:06:15.55,0:04:34.95,0:00:21.83,0:00:14.41,0:00:01.54,0:07:32.28,26963776,6619312,1500925,5579739 ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d,uninative-1.4-784-g3f2ab3f,1:28:05.02,0:15:21.08,1:10:55.63,0:02:50.81,0:00:21.51,0:00:14.07,0:00:01.38,0:04:28.12,26965460,6621804,1500767,5580167 ypperf01.zpn.intel.com,None:8a81facc23fd54b095940372e05de2356579c97c,uninative-1.4-785-g8a81fac,1:05:49.71,0:05:41.60,1:02:37.45,0:02:33.16,0:00:21.49,0:00:13.79,0:00:01.40,0:04:44.46,26974204,6620092,1500059,5579960 ypperf01.zpn.intel.com,None:8a81facc23fd54b095940372e05de2356579c97c,uninative-1.4-785-g8a81fac,1:05:35.45,0:05:44.76,1:02:46.00,0:02:29.40,0:00:21.37,0:00:14.00,0:00:01.37,0:04:28.13,26961344,6621488,1500037,5580332 merge order (newest to oldest) 8a81fac * binutils: Fix build for c293pcie PPC machine 3f2ab3f * python3: Build and package precompiled modules bef4e00 * taglib: Update to 1.11.1 Looking at the buildstats from those builds it is do_rootfs that has taken significantly longer. And that is in walltime. There is no big difference in cputime, write operations or so. Is the tester machine ok? Try re-running the tests. PKG TASK ABSDIFF RELDIFF WALLTIME1 -> WALLTIME2 core-image-sato do_rootfs 960.0s +782.7% 122.7s -> 1082.7s Cumulative walltime: 2759.0s +10.4% 7:21:10.3 (26470.3s) -> 8:07:09.3 (29229.3s) (In reply to comment #5) > Looking at the buildstats from those builds it is do_rootfs that has taken > significantly longer. This may be also the cause: '3f2ab3f * python3: Build and package precompiled modules' >And that is in walltime. There is no big difference in > cputime, write operations or so. Is the tester machine ok? Try re-running > the tests. > I am re-running the tests again, this time in different order (newer to older) > > PKG TASK ABSDIFF RELDIFF > WALLTIME1 -> WALLTIME2 > > core-image-sato do_rootfs 960.0s +782.7% > 122.7s -> 1082.7s > > Cumulative walltime: > 2759.0s +10.4% 7:21:10.3 (26470.3s) -> 8:07:09.3 (29229.3s) (In reply to comment #3) > (In reply to comment #2) > > > ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5, > > uninative-1.4-783-gbef4e00,1:05:27.49,0:05:52.75,1:04:40.51,0\ > > :04:06.59,0:00:21.87,0:00:14.36,0:00:01.59,0:06:49.33,26731216,6467604, > > 1442455,5332459 > > > > ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d, > > uninative-1.4-784-g3f2ab3f,1:26:48.03,0:14:00.12,1:06:15.55,0\ > > :04:34.95,0:00:21.83,0:00:14.41,0:00:01.54,0:07:32.28,26963776,6619312, > > 1500925,5579739 > > Why does the overall build time go from 1:05 to 1:26 here? Makes me worry > about the other measurements... I re-run the perf script, this time the commit order was different * 8a81fac: Fabio Berton: Wed Nov 30 15:48:05 2016 +0000: binutils: Fix build for c293pcie PPC machine * 3f2ab3f: Dominic Sacré: Wed Nov 30 15:48:05 2016 +0000: python3: Build and package precompiled modules * bef4e00: Ross Burton: Wed Nov 30 15:48:04 2016 +0000: taglib: Update to 1.11.1 These tree commits are consecutive, 8a merged (in master) after 3f and 3f after b4. There are the results (runned twice): ypperf01.zpn.intel.com,None:8a81facc23fd54b095940372e05de2356579c97c,uninative-1.4-785-g8a81fac,1:05:55.78,0:05:31.66,1:03:07.73,0:02:37.57,0:00:21.36,0:00:13.95,0:00:01.38,0:04:32.92,26970828,6622316,1500221,5580396 ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d,uninative-1.4-784-g3f2ab3f,1:05:50.14,0:05:27.28,1:03:44.12,0:03:25.24,0:00:21.42,0:00:14.03,0:00:01.37,0:04:10.64,26965840,6619020,1500736,5579598 ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5,uninative-1.4-783-gbef4e00,1:07:31.13,0:05:26.34,1:04:05.45,0:03:08.13,0:00:21.50,0:00:13.83,0:00:01.37,0:04:11.35,26733588,6465168,1441951,5331989 ypperf01.zpn.intel.com,None:8a81facc23fd54b095940372e05de2356579c97c,uninative-1.4-785-g8a81fac,1:09:14.36,0:05:28.98,1:05:42.65,0:03:08.24,0:00:21.41,0:00:13.83,0:00:01.39,0:05:12.66,26962760,6622060,1499941,5580351 ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d,uninative-1.4-784-g3f2ab3f,1:06:48.66,0:05:29.10,1:05:15.37,0:03:25.78,0:00:21.38,0:00:14.04,0:00:01.38,0:04:40.93,26964980,6619252,1500590,5579838 ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5,uninative-1.4-783-gbef4e00,1:08:53.37,0:05:32.37,1:05:59.83,0:03:08.96,0:00:21.41,0:00:13.85,0:00:01.37,0:05:09.28,26725968,6468220,1442188,5332517 The core-image-sato building times (column 4) are now in the same range, no big deltas between them, which is expected. The are two remaining items to be clarify yet: 1) On this comment's results, commit 8a is not adding up the time that was increased by 3f on the rootfs with sstate test (column 7) 2) the 20 minutes increase on the results from comment 2 More data is needed to understand both items. After discussing with Richard Purdie, moving this bug to M3. We are not seeing increasing numbers after 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d thus further analysis is needed. I run some tests and it looks like the following commit is causing a 20% regression in the rootfs test in my setup: commit efd120e71418bc1cf3431194f38adf6986d1956e Author: Richard Purdie <richard.purdie@linuxfoundation.org> Date: Thu Jan 19 16:53:53 2017 +0000 sstate: Drop the depchain isPostDep() checks I don't yet quite understand why that would happen, though Created attachment 3623 [details]
Total time
This file was created from a set of results coming from fedora perf machine, from date 20161220010003 to 20170110160004. RAID was added into the machine December 27th 2016, and as shown, numbers (the numbers indicate the total time for the poky/scripts/contrib/build-perf-test-wrapper.sh) increase after it, indicating a perf penalty with this setup.
No RAID, HD directly connected:
results-dbb247c-20161225110004/results.json:2: "elapsed_time": 10553.688936,
results-dbb247c-20161225160004/results.json:2: "elapsed_time": 10474.749505,
results-dbb247c-20161225210003/results.json:2: "elapsed_time": 10518.647101,
results-dbb247c-20161226010004/results.json:2: "elapsed_time": 10523.633608,
RAID added:
results-dbb247c-20161227160008/results.json:2: "elapsed_time": 14215.856867,
results-dbb247c-20161227210005/results.json:2: "elapsed_time": 11296.378797,
results-dbb247c-20161228010004/results.json:2: "elapsed_time": 11095.741204,
results-dbb247c-20161228110005/results.json:2: "elapsed_time": 12382.883796,
results-dbb247c-20161228160004/results.json:2: "elapsed_time": 11654.096605,
results-dbb247c-20161228210004/results.json:2: "elapsed_time": 11515.103425,
results-dbb247c-20161229010004/results.json:2: "elapsed_time": 11535.710448,
Pipeline used:
grep -irn ' "elapsed_time' `ls -rt -1 results*/results.json` | grep ':2:'
These are some numbers (for 2.3 M3 RC1 and RC3) obtained without the RAID and with the RAID but using different builders. Unfortunately, we do not have numbers numbers for the same builder machine with and without RAID. Nummbers indicate this time more than one minute penalty between these two configurations. Columns are sato image and its rootfs build time (columns 4 and 7 from csv file. # NO RAID, ypperf01 (fedora) (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:12:00.59 0:02:53.15 (nobranch):226a508da955439b881b2f0a544a3aee76e59919 1:08:22.72 0:02:51.83 (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:11:53.42 0:02:54.05 (nobranch):226a508da955439b881b2f0a544a3aee76e59919 1:08:28.47 0:03:00.55 [ypperf01@ypperf01 build-perf-test]$ # RAID, ypperf02 (ubuntu) (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:15:22.73 0:04:26.78 (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:13:15.07 0:03:58.71 (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:14:19.69 0:04:07.55 (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:15:48.83 0:04:46.06 (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:16:27.07 0:03:51.65 (nobranch):226a508da955439b881b2f0a544a3aee76e59919 1:11:03.78 0:03:52.65 (nobranch):226a508da955439b881b2f0a544a3aee76e59919 1:10:44.12 0:03:55.83 (nobranch):226a508da955439b881b2f0a544a3aee76e59919 1:11:33.72 0:04:07.95 Resolving as not-a-bug. The main problem was the inclusion of the RAID setup, so numbers went higher. Reopening. As I said, my testing seems to indicate that there is around 20% regression in the do_rootfs test, caused by efd120e71418bc1. ypperf-opensuse422,None:bc7162a46cebfec3896f95081bc2f635efa509f2,2.3_M1-337-gbc7162a,1:19:28.41,0:06:40.52,1:20:42.19,0:02:48.25,0:00:26.65,0:00:17.39,0:00:01.79,0:03:14.10,27108480,6621684,1521853,5028288 ypperf-opensuse422,None:efd120e71418bc1cf3431194f38adf6986d1956e,2.3_M1-338-gefd120e,1:19:35.25,0:06:41.00,1:21:06.46,0:03:20.83,0:00:26.49,0:00:17.63,0:00:01.80,0:03:29.16,27105376,6620064,1522122,5027883 Just wanted to know if this is something we understand and accept, in which case we can close this bug. Or, do we want to try to do someething about this(?) (In reply to comment #13) > Reopening. As I said, my testing seems to indicate that there is around 20% > regression in the do_rootfs test, caused by efd120e71418bc1. > > ypperf-opensuse422,None:bc7162a46cebfec3896f95081bc2f635efa509f2,2.3_M1-337- > gbc7162a,1:19:28.41,0:06:40.52,1:20:42.19,0:02:48.25,0:00:26.65,0:00:17.39,0: > 00:01.79,0:03:14.10,27108480,6621684,1521853,5028288 > ypperf-opensuse422,None:efd120e71418bc1cf3431194f38adf6986d1956e,2.3_M1-338- > gefd120e,1:19:35.25,0:06:41.00,1:21:06.46,0:03:20.83,0:00:26.49,0:00:17.63,0: > 00:01.80,0:03:29.16,27105376,6620064,1522122,5027883 > > > Just wanted to know if this is something we understand and accept, in which > case we can close this bug. Or, do we want to try to do someething about > this(?) Let me further investigate what you have observed. (In reply to comment #13) > Reopening. As I said, my testing seems to indicate that there is around 20% > regression in the do_rootfs test, caused by efd120e71418bc1. > > ypperf-opensuse422,None:bc7162a46cebfec3896f95081bc2f635efa509f2,2.3_M1-337- > gbc7162a,1:19:28.41,0:06:40.52,1:20:42.19,0:02:48.25,0:00:26.65,0:00:17.39,0: > 00:01.79,0:03:14.10,27108480,6621684,1521853,5028288 > ypperf-opensuse422,None:efd120e71418bc1cf3431194f38adf6986d1956e,2.3_M1-338- > gefd120e,1:19:35.25,0:06:41.00,1:21:06.46,0:03:20.83,0:00:26.49,0:00:17.63,0: > 00:01.80,0:03:29.16,27105376,6620064,1522122,5027883 > > > Just wanted to know if this is something we understand and accept, in which > case we can close this bug. Or, do we want to try to do someething about > this(?) Latest number from the fedora perf machine for the following commits: # eae2dcd * gtk-icon-cache: Add PACKAGE_WRITE_DEPS for postinst # efd120e * sstate: Drop the depchain isPostDep() checks # bc7162a * classes/package*: Add support for PACKAGE_WRITE_DEPS ypperf01.zpn.intel.com,None:bc7162a46cebfec3896f95081bc2f635efa509f2,2.3_M1-337-gbc7162a,1:01:07.14,0:05:28.46,1:00:23.99,0:02:22.02,0:00:22.31,0:00:14.31,0:00:01.41,0:03:01.36,27133248,6655296,1511769,5012496 ypperf01.zpn.intel.com,None:efd120e71418bc1cf3431194f38adf6986d1956e,2.3_M1-338-gefd120e,1:01:03.05,0:05:18.91,1:00:13.11,0:02:55.37,0:00:22.28,0:00:14.24,0:00:01.40,0:02:45.50,27132660,6656604,1511608,5012905 ypperf01.zpn.intel.com,None:eae2dcd5742d9aaad743c7b9e954afe3605f4aec,2.3_M1-339-geae2dcd,1:01:05.05,0:05:33.54,1:00:26.32,0:02:50.99,0:00:22.44,0:00:14.38,0:00:01.40,0:02:40.78,27134800,6656064,1511642,5012675 ypperf01.zpn.intel.com,None:bc7162a46cebfec3896f95081bc2f635efa509f2,2.3_M1-337-gbc7162a,1:01:13.58,0:05:25.12,1:00:37.90,0:02:16.99,0:00:22.12,0:00:14.33,0:00:01.41,0:02:46.51,27135012,6656944,1512116,5012781 ypperf01.zpn.intel.com,None:efd120e71418bc1cf3431194f38adf6986d1956e,2.3_M1-338-gefd120e,1:01:11.06,0:05:26.81,1:00:27.04,0:02:50.43,0:00:22.16,0:00:14.42,0:00:01.42,0:02:48.88,27132300,6656316,1511550,5012775 So, the delta between aea and efd is around 30 seconds, thus approximately a 20% increment, as indicated by Markus. These are the tmp/sysroots folders for two consecutive commits (bc71 and efd1, the later is newer and these are consecutive in the repo history). A few observations based on this data: * the efd1 commit increase three times the size of the tmp/sysroots folder comparing to previous commit (bc71) * the commit efd1 creates this folder with size 732M tmp/sysroots/qemux86/usr which does not exit on bc71. lsandov1@yctb05:~/bug10884/poky-bc7162a46cebfec3896f95081bc2f635efa509f2/build> du -h -d 2 tmp/sysroots 46M tmp/sysroots/qemux86/pkgdata 16K tmp/sysroots/qemux86/usr 16K tmp/sysroots/qemux86/sysroot-providers 20K tmp/sysroots/qemux86/etc 46M tmp/sysroots/qemux86 549M tmp/sysroots/x86_64-linux/usr 412K tmp/sysroots/x86_64-linux/bin 2.0M tmp/sysroots/x86_64-linux/sbin 452K tmp/sysroots/x86_64-linux/etc 4.0K tmp/sysroots/x86_64-linux/var 464K tmp/sysroots/x86_64-linux/sysroot-providers 1.2M tmp/sysroots/x86_64-linux/lib 554M tmp/sysroots/x86_64-linux 599M tmp/sysroots lsandov1@yctb05:~/bug10884/poky-efd120e71418bc1cf3431194f38adf6986d1956e/build> du -h -d 2 tmp/sysroots 46M tmp/sysroots/qemux86/pkgdata 40K tmp/sysroots/qemux86/etc 1.5M tmp/sysroots/qemux86/sysroot-providers 732M tmp/sysroots/qemux86/usr 28M tmp/sysroots/qemux86/lib 96K tmp/sysroots/qemux86/var 807M tmp/sysroots/qemux86 937M tmp/sysroots/x86_64-linux/usr 412K tmp/sysroots/x86_64-linux/bin 2.0M tmp/sysroots/x86_64-linux/sbin 480K tmp/sysroots/x86_64-linux/etc 4.0K tmp/sysroots/x86_64-linux/var 544K tmp/sysroots/x86_64-linux/sysroot-providers 1.2M tmp/sysroots/x86_64-linux/lib 942M tmp/sysroots/x86_64-linux 1.8G tmp/sysroots I went digging on this. Its not expected that dropping the postinstall dependency code should cause such an increase in the size of the sysroot but the investigation is correct and it does.
What seems to happen is that the sstate code seems to think that for example, coreutils:do_package_write_rpm depends on libcap:do_populate_sysroot. The code sees that dependency and others and therefore ends up populating the target sysroot.
The target sysroot shouldn't be populated and this dependency is clearly bogus (along with a number of other similar looking odd ones).
I have a patch to bitbake's runqueue.py which fixes that code, its worth noting that with that applied, this test case will then fail as there are dependency bugs in oe-core. To
To make this older revision work, you need to do:
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6b08649..6599643 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -274,7 +274,7 @@ fakeroot python do_rootfs () {
do_rootfs[dirs] = "${TOPDIR}"
do_rootfs[cleandirs] += "${S} ${IMGDEPLOYDIR}"
do_rootfs[umask] = "022"
-addtask rootfs before do_build
+addtask rootfs before do_build after do_configure
fakeroot python do_image () {
from oe.utils import execute_pre_post_process
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index f1faf48..d11928f 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -877,7 +877,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d):
bb.debug(2, "Considering setscene task: %s" % (str(taskdependees[task])))
def isNativeCross(x):
- return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x
+ return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
# We only need to trigger populate_lic through direct dependencies
if taskdependees[task][1] == "do_populate_lic":
however these changes were made as part of Recipe Specific Sysroots which identified the same bugs. With these bugs fixed, my bitbake patch avoids the population of the target sysroot as it should.
Testing your suggested patch. (In reply to comment #17) > I went digging on this. Its not expected that dropping the postinstall > dependency code should cause such an increase in the size of the sysroot but > the investigation is correct and it does. > > What seems to happen is that the sstate code seems to think that for > example, coreutils:do_package_write_rpm depends on > libcap:do_populate_sysroot. The code sees that dependency and others and > therefore ends up populating the target sysroot. > > The target sysroot shouldn't be populated and this dependency is clearly > bogus (along with a number of other similar looking odd ones). > > I have a patch to bitbake's runqueue.py which fixes that code, its worth > noting that with that applied, this test case will then fail as there are > dependency bugs in oe-core. To > > To make this older revision work, you need to do: > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 6b08649..6599643 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -274,7 +274,7 @@ fakeroot python do_rootfs () { > do_rootfs[dirs] = "${TOPDIR}" > do_rootfs[cleandirs] += "${S} ${IMGDEPLOYDIR}" > do_rootfs[umask] = "022" > -addtask rootfs before do_build > +addtask rootfs before do_build after do_configure > > fakeroot python do_image () { > from oe.utils import execute_pre_post_process > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass > index f1faf48..d11928f 100644 > --- a/meta/classes/sstate.bbclass > +++ b/meta/classes/sstate.bbclass > @@ -877,7 +877,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d): > bb.debug(2, "Considering setscene task: %s" % > (str(taskdependees[task]))) > > def isNativeCross(x): > - return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x > + return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x > or x.endswith("-cross") > > # We only need to trigger populate_lic through direct dependencies > if taskdependees[task][1] == "do_populate_lic": > > > however these changes were made as part of Recipe Specific Sysroots which > identified the same bugs. With these bugs fixed, my bitbake patch avoids the > population of the target sysroot as it should. I have verified two things: * suggested patch by RP is already merged * tmp/sysroot folder sizes are basically the same # previous RSS commit lsandov1@yctb05:~/bug10884/poky-efd120e71418bc1cf3431194f38adf6986d1956e/build> du -h -d 1 tmp/sysroots 942M tmp/sysroots/x86_64-linux 807M tmp/sysroots/qemux86 1.8G tmp/sysroots # commit introducing RSS lsandov1@yctb05:~/bug10884/poky-9107d6ca144ac1f6169830dee25dae2dad775333/build> du -h -d 1 tmp/sysroots-components/ 946M tmp/sysroots-components/x86_64 36K tmp/sysroots-components/qemux86 759M tmp/sysroots-components/i586 3.4M tmp/sysroots-components/all 1.7G tmp/sysroots-components/ My tests indicate the same, i.e. that the regression is fixed by Richards patch. do_rootfs time dropped back to the level it was before the regression Fixed by: ff734f0560649c0d8743aa1ebdaef8cfb20e6628 Author: Richard Purdie <richard.purdie@linuxfoundation.org> AuthorDate: Sat Feb 18 13:09:39 2017 -0800 Commit: Richard Purdie <richard.purdie@linuxfoundation.org> CommitDate: Sun Feb 19 06:28:47 2017 -0800 Parent: 10968b4 bitbake: bitbake-layers: Handle exception raised bytinfoil.prepare() Containing: master Follows: 2.3_M1 (897) bitbake: runqueue: Fix collapsed setscene dependency tree Verified on 2.3 M3 rc2 poky commit: 415b72ffcbd26e5f3664370d8b2a9b8105fb6342 |