10884 2017-01-05 19:54:12 +0000 build-perf:time for build "rootfs" is increased in around 20% on 2.3 M1 2017-04-11 20:22:39 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple VERIFIED FIXED High normal 2.3 M3 1 jose.perez.carranza leonardo.sandoval.gonzalez benjamin.esquivel markus.lehtonen poky.bs.watcher poky.watcher richard.purdie ross.burton --- --- No (bug/feature does not impact docs) oldest_to_newest 69551 0 jose.perez.carranza 2017-01-05 19:54:12 +0000 Compared with 2.2 rc6 (morty) the tiome of building rootfs has been increased in around 20% (30 seconds). Charts : https://wiki.yoctoproject.org/charts/perf_milestone_GDC/bitbake-core-image-sato-rootfs.html --- Times --- 2.2 rc6 commit: dc8508f609974cc99606b9042bfa7f870ce80228 2.3 rc1 commit: 573c646d4cc62dcd0c230381df4940bdf314d495 - 2.2 rc6 --> Ubuntu15: 00:02:14 - 2.3 rc1 --> Ubuntu15: 00:02:47 - 2.2 rc6 --> Fedora23: 00:02:08 - 2.3 rc1 --> Fedora23: 00:02:34 --- Steps --- 1- $ time bitbake core-image-sato -c rootfs 69804 1 leonardo.sandoval.gonzalez 2017-01-13 17:10:54 +0000 These are the numbers between the two mentioned commits (first column is hash and second is mean time in seconds for bitbake core-image-sato -c rootfs with no tmp and cache and with sstates) morty:dc8508f609974cc99606b9042bfa7f870ce80228,130.61714285714 master:003b053498aea91fdb8c5418d0d8bbda7b23dbc0,131.2425 master:c3d2df883a9d6d5036277114339673656d89a728,130.65333333333 master:edb5b524b3ee807d55a437cbab3d768c5f6f2283,130.37583333333 master:1655d55764eb0241b71217810500fcf1d4519a1c,134.80666666667 master:43e652f3d1fee5ce7fad67e6400315eab1b34270,133.66333333333 master:09278be8c209976da03f63b5a956d212ac171317,137.30357142857 master:9b1bd29ea99aa0dec51e603e4ac28e211af19765,138 master:7c3a47ed8965c3a3eb90a9a4678d5caedbba6337,137.6578125 master:12a0ee049e453b6d0d2ce2f3fa981d1b6e02bd78,136.58818181818 master:71b7c3a88ec5a20d5e4531751d0bf23d5c94e494,148.5175 master:3d34ae6c11bc2e800f784740c54b2dd2029658ed,149.23333333333 master:9e63f81c78e284c9b325fe04a1b59e61c7ad8a1a,0 master:e38775a1d82e6dc60fc96cf243ecb94be964d9b2,0 master:18e4688bd52ed7746e7687b225da10f025cafc68,0 master:dae00fbb913c24b6bab00d5b99bd61f3455b4728,0 master:36e178a62f04e7f2611b26964efe46b9af97189a,0 master:a0374e92a82e8c31964163f2eefa471bfe1eb0d4,0 master:14539203655913b3304bf98d2dd1f2e1fed3d7ca,0 master:a92d27acd8f42b61a2fc575311838b45d171f0af,151.796 master:573c646d4cc62dcd0c230381df4940bdf314d495,155.664 The greatest delta between times is master:12a0ee049e453b6d0d2ce2f3fa981d1b6e02bd78,136.58818181818 master:71b7c3a88ec5a20d5e4531751d0bf23d5c94e494,148.5175 and if we do a git-log shortlog between these two $ git ol 12a0ee049e453b6d0d2ce2f3fa981d1b6e02bd78^..71b7c3a88ec5a20d5e4531751d0bf23d5c94e494 71b7c3a: Ross Burton: Mon Nov 28 15:49:15 2016 +0000: Revert "classes/populate_sdk_ext: require uninative" 5fd4cad: Ross Burton: Mon Nov 28 15:49:15 2016 +0000: conf: add C++ flags for uninative interoperatility d351c87: Jose Lamego: Mon Nov 28 14:50:05 2016 +0000: maintainers: Change maintainers for python(3) modules 12a0ee0: Scott Rifenbark: Wed Nov 23 11:10:35 2016 +0000: dev-manual: Added note about RPM not dealing with post-install meaning that either 71b7c3a or 5fd4cad are the culprits (because the other two are just documentation stuff) 69996 2 leonardo.sandoval.gonzalez 2017-01-23 03:36:12 +0000 These are two results using the scripts/contrib/build-perf-test-wraper.sh for two consecutive commits bef4e00bdadcd912574f73fde93e58d3943b18b5 and 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d (http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d, with title 'python3: Build and package precompiled modules') --- ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5,uninative-1.4-783-gbef4e00,1:05:27.49,0:05:52.75,1:04:40.51,0\ :04:06.59,0:00:21.87,0:00:14.36,0:00:01.59,0:06:49.33,26731216,6467604,1442455,5332459 ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d,uninative-1.4-784-g3f2ab3f,1:26:48.03,0:14:00.12,1:06:15.55,0\ :04:34.95,0:00:21.83,0:00:14.41,0:00:01.54,0:07:32.28,26963776,6619312,1500925,5579739 --- Looking into the tmp size column created by core-image-sato, we see this tmp sizes (in unit of blocks, consisting of 1024 bytes) respectively * 26731216 * 26963776 meaning around 232k more blocks or 227Mbytes are due 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d, impacting the sshare-cache folders (not in the same proportion) size thus the current measure time in this bug ('rm -rf tmp cache; bitbake core-image-sato -c rootfs'). All commits following 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d basically have similar tmp sizes, sshare-cache and core-images-sato rootfs creation times. In resume, increase times on core-image-sato rootfs creation are due to larger python3 packages, increased considerably by 3f2ab3f. 70002 3 richard.purdie 2017-01-23 14:16:18 +0000 (In reply to comment #2) > ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5, > uninative-1.4-783-gbef4e00,1:05:27.49,0:05:52.75,1:04:40.51,0\ > :04:06.59,0:00:21.87,0:00:14.36,0:00:01.59,0:06:49.33,26731216,6467604, > 1442455,5332459 > > ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d, > uninative-1.4-784-g3f2ab3f,1:26:48.03,0:14:00.12,1:06:15.55,0\ > :04:34.95,0:00:21.83,0:00:14.41,0:00:01.54,0:07:32.28,26963776,6619312, > 1500925,5579739 Why does the overall build time go from 1:05 to 1:26 here? Makes me worry about the other measurements... 70009 4 leonardo.sandoval.gonzalez 2017-01-23 15:03:30 +0000 RP detected something not yet understood: build times for 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d are increasing considerably (by 20 minutes) but the weird thing is that commit after the latter (which of course includes 3f2ab) drops again to previous times. Performance numbers for bef4e00bdadcd912574f73fde93e58d3943b18b5, 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d and 8a81facc23fd54b095940372e05de2356579c97c (run twice for each commit) ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5,uninative-1.4-783-gbef4e00,1:05:45.73,0:05:39.27,1:02:37.37,0:02:31.80,0:00:21.51,0:00:13.79,0:00:01.38,0:04:17.40,26734880,6464912,1442084,5331747 ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5,uninative-1.4-783-gbef4e00,1:05:27.49,0:05:52.75,1:04:40.51,0:04:06.59,0:00:21.87,0:00:14.36,0:00:01.59,0:06:49.33,26731216,6467604,1442455,5332459 ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d,uninative-1.4-784-g3f2ab3f,1:26:48.03,0:14:00.12,1:06:15.55,0:04:34.95,0:00:21.83,0:00:14.41,0:00:01.54,0:07:32.28,26963776,6619312,1500925,5579739 ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d,uninative-1.4-784-g3f2ab3f,1:28:05.02,0:15:21.08,1:10:55.63,0:02:50.81,0:00:21.51,0:00:14.07,0:00:01.38,0:04:28.12,26965460,6621804,1500767,5580167 ypperf01.zpn.intel.com,None:8a81facc23fd54b095940372e05de2356579c97c,uninative-1.4-785-g8a81fac,1:05:49.71,0:05:41.60,1:02:37.45,0:02:33.16,0:00:21.49,0:00:13.79,0:00:01.40,0:04:44.46,26974204,6620092,1500059,5579960 ypperf01.zpn.intel.com,None:8a81facc23fd54b095940372e05de2356579c97c,uninative-1.4-785-g8a81fac,1:05:35.45,0:05:44.76,1:02:46.00,0:02:29.40,0:00:21.37,0:00:14.00,0:00:01.37,0:04:28.13,26961344,6621488,1500037,5580332 merge order (newest to oldest) 8a81fac * binutils: Fix build for c293pcie PPC machine 3f2ab3f * python3: Build and package precompiled modules bef4e00 * taglib: Update to 1.11.1 70026 5 markus.lehtonen 2017-01-23 16:36:06 +0000 Looking at the buildstats from those builds it is do_rootfs that has taken significantly longer. And that is in walltime. There is no big difference in cputime, write operations or so. Is the tester machine ok? Try re-running the tests. PKG TASK ABSDIFF RELDIFF WALLTIME1 -> WALLTIME2 core-image-sato do_rootfs 960.0s +782.7% 122.7s -> 1082.7s Cumulative walltime: 2759.0s +10.4% 7:21:10.3 (26470.3s) -> 8:07:09.3 (29229.3s) 70028 6 leonardo.sandoval.gonzalez 2017-01-23 16:43:53 +0000 (In reply to comment #5) > Looking at the buildstats from those builds it is do_rootfs that has taken > significantly longer. This may be also the cause: '3f2ab3f * python3: Build and package precompiled modules' >And that is in walltime. There is no big difference in > cputime, write operations or so. Is the tester machine ok? Try re-running > the tests. > I am re-running the tests again, this time in different order (newer to older) > > PKG TASK ABSDIFF RELDIFF > WALLTIME1 -> WALLTIME2 > > core-image-sato do_rootfs 960.0s +782.7% > 122.7s -> 1082.7s > > Cumulative walltime: > 2759.0s +10.4% 7:21:10.3 (26470.3s) -> 8:07:09.3 (29229.3s) 70082 7 leonardo.sandoval.gonzalez 2017-01-24 15:27:50 +0000 (In reply to comment #3) > (In reply to comment #2) > > > ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5, > > uninative-1.4-783-gbef4e00,1:05:27.49,0:05:52.75,1:04:40.51,0\ > > :04:06.59,0:00:21.87,0:00:14.36,0:00:01.59,0:06:49.33,26731216,6467604, > > 1442455,5332459 > > > > ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d, > > uninative-1.4-784-g3f2ab3f,1:26:48.03,0:14:00.12,1:06:15.55,0\ > > :04:34.95,0:00:21.83,0:00:14.41,0:00:01.54,0:07:32.28,26963776,6619312, > > 1500925,5579739 > > Why does the overall build time go from 1:05 to 1:26 here? Makes me worry > about the other measurements... I re-run the perf script, this time the commit order was different * 8a81fac: Fabio Berton: Wed Nov 30 15:48:05 2016 +0000: binutils: Fix build for c293pcie PPC machine * 3f2ab3f: Dominic Sacré: Wed Nov 30 15:48:05 2016 +0000: python3: Build and package precompiled modules * bef4e00: Ross Burton: Wed Nov 30 15:48:04 2016 +0000: taglib: Update to 1.11.1 These tree commits are consecutive, 8a merged (in master) after 3f and 3f after b4. There are the results (runned twice): ypperf01.zpn.intel.com,None:8a81facc23fd54b095940372e05de2356579c97c,uninative-1.4-785-g8a81fac,1:05:55.78,0:05:31.66,1:03:07.73,0:02:37.57,0:00:21.36,0:00:13.95,0:00:01.38,0:04:32.92,26970828,6622316,1500221,5580396 ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d,uninative-1.4-784-g3f2ab3f,1:05:50.14,0:05:27.28,1:03:44.12,0:03:25.24,0:00:21.42,0:00:14.03,0:00:01.37,0:04:10.64,26965840,6619020,1500736,5579598 ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5,uninative-1.4-783-gbef4e00,1:07:31.13,0:05:26.34,1:04:05.45,0:03:08.13,0:00:21.50,0:00:13.83,0:00:01.37,0:04:11.35,26733588,6465168,1441951,5331989 ypperf01.zpn.intel.com,None:8a81facc23fd54b095940372e05de2356579c97c,uninative-1.4-785-g8a81fac,1:09:14.36,0:05:28.98,1:05:42.65,0:03:08.24,0:00:21.41,0:00:13.83,0:00:01.39,0:05:12.66,26962760,6622060,1499941,5580351 ypperf01.zpn.intel.com,None:3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d,uninative-1.4-784-g3f2ab3f,1:06:48.66,0:05:29.10,1:05:15.37,0:03:25.78,0:00:21.38,0:00:14.04,0:00:01.38,0:04:40.93,26964980,6619252,1500590,5579838 ypperf01.zpn.intel.com,None:bef4e00bdadcd912574f73fde93e58d3943b18b5,uninative-1.4-783-gbef4e00,1:08:53.37,0:05:32.37,1:05:59.83,0:03:08.96,0:00:21.41,0:00:13.85,0:00:01.37,0:05:09.28,26725968,6468220,1442188,5332517 The core-image-sato building times (column 4) are now in the same range, no big deltas between them, which is expected. The are two remaining items to be clarify yet: 1) On this comment's results, commit 8a is not adding up the time that was increased by 3f on the rootfs with sstate test (column 7) 2) the 20 minutes increase on the results from comment 2 More data is needed to understand both items. 70113 8 leonardo.sandoval.gonzalez 2017-01-25 15:42:57 +0000 After discussing with Richard Purdie, moving this bug to M3. We are not seeing increasing numbers after 3f2ab3f7cc1bf1646f2acfcace9b9cf546cc3a5d thus further analysis is needed. 70257 9 markus.lehtonen 2017-01-30 15:28:44 +0000 I run some tests and it looks like the following commit is causing a 20% regression in the rootfs test in my setup: commit efd120e71418bc1cf3431194f38adf6986d1956e Author: Richard Purdie <richard.purdie@linuxfoundation.org> Date: Thu Jan 19 16:53:53 2017 +0000 sstate: Drop the depchain isPostDep() checks I don't yet quite understand why that would happen, though 70395 10 3623 leonardo.sandoval.gonzalez 2017-02-02 18:08:29 +0000 Created attachment 3623 Total time This file was created from a set of results coming from fedora perf machine, from date 20161220010003 to 20170110160004. RAID was added into the machine December 27th 2016, and as shown, numbers (the numbers indicate the total time for the poky/scripts/contrib/build-perf-test-wrapper.sh) increase after it, indicating a perf penalty with this setup. No RAID, HD directly connected: results-dbb247c-20161225110004/results.json:2: "elapsed_time": 10553.688936, results-dbb247c-20161225160004/results.json:2: "elapsed_time": 10474.749505, results-dbb247c-20161225210003/results.json:2: "elapsed_time": 10518.647101, results-dbb247c-20161226010004/results.json:2: "elapsed_time": 10523.633608, RAID added: results-dbb247c-20161227160008/results.json:2: "elapsed_time": 14215.856867, results-dbb247c-20161227210005/results.json:2: "elapsed_time": 11296.378797, results-dbb247c-20161228010004/results.json:2: "elapsed_time": 11095.741204, results-dbb247c-20161228110005/results.json:2: "elapsed_time": 12382.883796, results-dbb247c-20161228160004/results.json:2: "elapsed_time": 11654.096605, results-dbb247c-20161228210004/results.json:2: "elapsed_time": 11515.103425, results-dbb247c-20161229010004/results.json:2: "elapsed_time": 11535.710448, Pipeline used: grep -irn ' "elapsed_time' `ls -rt -1 results*/results.json` | grep ':2:' 70422 11 leonardo.sandoval.gonzalez 2017-02-03 16:42:35 +0000 These are some numbers (for 2.3 M3 RC1 and RC3) obtained without the RAID and with the RAID but using different builders. Unfortunately, we do not have numbers numbers for the same builder machine with and without RAID. Nummbers indicate this time more than one minute penalty between these two configurations. Columns are sato image and its rootfs build time (columns 4 and 7 from csv file. # NO RAID, ypperf01 (fedora) (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:12:00.59 0:02:53.15 (nobranch):226a508da955439b881b2f0a544a3aee76e59919 1:08:22.72 0:02:51.83 (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:11:53.42 0:02:54.05 (nobranch):226a508da955439b881b2f0a544a3aee76e59919 1:08:28.47 0:03:00.55 [ypperf01@ypperf01 build-perf-test]$ # RAID, ypperf02 (ubuntu) (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:15:22.73 0:04:26.78 (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:13:15.07 0:03:58.71 (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:14:19.69 0:04:07.55 (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:15:48.83 0:04:46.06 (nobranch):62d7d4130202d8ede16abf9e7d779361ca70847e 1:16:27.07 0:03:51.65 (nobranch):226a508da955439b881b2f0a544a3aee76e59919 1:11:03.78 0:03:52.65 (nobranch):226a508da955439b881b2f0a544a3aee76e59919 1:10:44.12 0:03:55.83 (nobranch):226a508da955439b881b2f0a544a3aee76e59919 1:11:33.72 0:04:07.95 70494 12 leonardo.sandoval.gonzalez 2017-02-07 16:19:43 +0000 Resolving as not-a-bug. The main problem was the inclusion of the RAID setup, so numbers went higher. 70530 13 markus.lehtonen 2017-02-08 07:24:13 +0000 Reopening. As I said, my testing seems to indicate that there is around 20% regression in the do_rootfs test, caused by efd120e71418bc1. ypperf-opensuse422,None:bc7162a46cebfec3896f95081bc2f635efa509f2,2.3_M1-337-gbc7162a,1:19:28.41,0:06:40.52,1:20:42.19,0:02:48.25,0:00:26.65,0:00:17.39,0:00:01.79,0:03:14.10,27108480,6621684,1521853,5028288 ypperf-opensuse422,None:efd120e71418bc1cf3431194f38adf6986d1956e,2.3_M1-338-gefd120e,1:19:35.25,0:06:41.00,1:21:06.46,0:03:20.83,0:00:26.49,0:00:17.63,0:00:01.80,0:03:29.16,27105376,6620064,1522122,5027883 Just wanted to know if this is something we understand and accept, in which case we can close this bug. Or, do we want to try to do someething about this(?) 70550 14 leonardo.sandoval.gonzalez 2017-02-08 15:47:29 +0000 (In reply to comment #13) > Reopening. As I said, my testing seems to indicate that there is around 20% > regression in the do_rootfs test, caused by efd120e71418bc1. > > ypperf-opensuse422,None:bc7162a46cebfec3896f95081bc2f635efa509f2,2.3_M1-337- > gbc7162a,1:19:28.41,0:06:40.52,1:20:42.19,0:02:48.25,0:00:26.65,0:00:17.39,0: > 00:01.79,0:03:14.10,27108480,6621684,1521853,5028288 > ypperf-opensuse422,None:efd120e71418bc1cf3431194f38adf6986d1956e,2.3_M1-338- > gefd120e,1:19:35.25,0:06:41.00,1:21:06.46,0:03:20.83,0:00:26.49,0:00:17.63,0: > 00:01.80,0:03:29.16,27105376,6620064,1522122,5027883 > > > Just wanted to know if this is something we understand and accept, in which > case we can close this bug. Or, do we want to try to do someething about > this(?) Let me further investigate what you have observed. 70583 15 leonardo.sandoval.gonzalez 2017-02-09 15:27:55 +0000 (In reply to comment #13) > Reopening. As I said, my testing seems to indicate that there is around 20% > regression in the do_rootfs test, caused by efd120e71418bc1. > > ypperf-opensuse422,None:bc7162a46cebfec3896f95081bc2f635efa509f2,2.3_M1-337- > gbc7162a,1:19:28.41,0:06:40.52,1:20:42.19,0:02:48.25,0:00:26.65,0:00:17.39,0: > 00:01.79,0:03:14.10,27108480,6621684,1521853,5028288 > ypperf-opensuse422,None:efd120e71418bc1cf3431194f38adf6986d1956e,2.3_M1-338- > gefd120e,1:19:35.25,0:06:41.00,1:21:06.46,0:03:20.83,0:00:26.49,0:00:17.63,0: > 00:01.80,0:03:29.16,27105376,6620064,1522122,5027883 > > > Just wanted to know if this is something we understand and accept, in which > case we can close this bug. Or, do we want to try to do someething about > this(?) Latest number from the fedora perf machine for the following commits: # eae2dcd * gtk-icon-cache: Add PACKAGE_WRITE_DEPS for postinst # efd120e * sstate: Drop the depchain isPostDep() checks # bc7162a * classes/package*: Add support for PACKAGE_WRITE_DEPS ypperf01.zpn.intel.com,None:bc7162a46cebfec3896f95081bc2f635efa509f2,2.3_M1-337-gbc7162a,1:01:07.14,0:05:28.46,1:00:23.99,0:02:22.02,0:00:22.31,0:00:14.31,0:00:01.41,0:03:01.36,27133248,6655296,1511769,5012496 ypperf01.zpn.intel.com,None:efd120e71418bc1cf3431194f38adf6986d1956e,2.3_M1-338-gefd120e,1:01:03.05,0:05:18.91,1:00:13.11,0:02:55.37,0:00:22.28,0:00:14.24,0:00:01.40,0:02:45.50,27132660,6656604,1511608,5012905 ypperf01.zpn.intel.com,None:eae2dcd5742d9aaad743c7b9e954afe3605f4aec,2.3_M1-339-geae2dcd,1:01:05.05,0:05:33.54,1:00:26.32,0:02:50.99,0:00:22.44,0:00:14.38,0:00:01.40,0:02:40.78,27134800,6656064,1511642,5012675 ypperf01.zpn.intel.com,None:bc7162a46cebfec3896f95081bc2f635efa509f2,2.3_M1-337-gbc7162a,1:01:13.58,0:05:25.12,1:00:37.90,0:02:16.99,0:00:22.12,0:00:14.33,0:00:01.41,0:02:46.51,27135012,6656944,1512116,5012781 ypperf01.zpn.intel.com,None:efd120e71418bc1cf3431194f38adf6986d1956e,2.3_M1-338-gefd120e,1:01:11.06,0:05:26.81,1:00:27.04,0:02:50.43,0:00:22.16,0:00:14.42,0:00:01.42,0:02:48.88,27132300,6656316,1511550,5012775 So, the delta between aea and efd is around 30 seconds, thus approximately a 20% increment, as indicated by Markus. 70661 16 leonardo.sandoval.gonzalez 2017-02-13 15:53:06 +0000 These are the tmp/sysroots folders for two consecutive commits (bc71 and efd1, the later is newer and these are consecutive in the repo history). A few observations based on this data: * the efd1 commit increase three times the size of the tmp/sysroots folder comparing to previous commit (bc71) * the commit efd1 creates this folder with size 732M tmp/sysroots/qemux86/usr which does not exit on bc71. lsandov1@yctb05:~/bug10884/poky-bc7162a46cebfec3896f95081bc2f635efa509f2/build> du -h -d 2 tmp/sysroots 46M tmp/sysroots/qemux86/pkgdata 16K tmp/sysroots/qemux86/usr 16K tmp/sysroots/qemux86/sysroot-providers 20K tmp/sysroots/qemux86/etc 46M tmp/sysroots/qemux86 549M tmp/sysroots/x86_64-linux/usr 412K tmp/sysroots/x86_64-linux/bin 2.0M tmp/sysroots/x86_64-linux/sbin 452K tmp/sysroots/x86_64-linux/etc 4.0K tmp/sysroots/x86_64-linux/var 464K tmp/sysroots/x86_64-linux/sysroot-providers 1.2M tmp/sysroots/x86_64-linux/lib 554M tmp/sysroots/x86_64-linux 599M tmp/sysroots lsandov1@yctb05:~/bug10884/poky-efd120e71418bc1cf3431194f38adf6986d1956e/build> du -h -d 2 tmp/sysroots 46M tmp/sysroots/qemux86/pkgdata 40K tmp/sysroots/qemux86/etc 1.5M tmp/sysroots/qemux86/sysroot-providers 732M tmp/sysroots/qemux86/usr 28M tmp/sysroots/qemux86/lib 96K tmp/sysroots/qemux86/var 807M tmp/sysroots/qemux86 937M tmp/sysroots/x86_64-linux/usr 412K tmp/sysroots/x86_64-linux/bin 2.0M tmp/sysroots/x86_64-linux/sbin 480K tmp/sysroots/x86_64-linux/etc 4.0K tmp/sysroots/x86_64-linux/var 544K tmp/sysroots/x86_64-linux/sysroot-providers 1.2M tmp/sysroots/x86_64-linux/lib 942M tmp/sysroots/x86_64-linux 1.8G tmp/sysroots 70817 17 richard.purdie 2017-02-19 01:02:32 +0000 I went digging on this. Its not expected that dropping the postinstall dependency code should cause such an increase in the size of the sysroot but the investigation is correct and it does. What seems to happen is that the sstate code seems to think that for example, coreutils:do_package_write_rpm depends on libcap:do_populate_sysroot. The code sees that dependency and others and therefore ends up populating the target sysroot. The target sysroot shouldn't be populated and this dependency is clearly bogus (along with a number of other similar looking odd ones). I have a patch to bitbake's runqueue.py which fixes that code, its worth noting that with that applied, this test case will then fail as there are dependency bugs in oe-core. To To make this older revision work, you need to do: diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 6b08649..6599643 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -274,7 +274,7 @@ fakeroot python do_rootfs () { do_rootfs[dirs] = "${TOPDIR}" do_rootfs[cleandirs] += "${S} ${IMGDEPLOYDIR}" do_rootfs[umask] = "022" -addtask rootfs before do_build +addtask rootfs before do_build after do_configure fakeroot python do_image () { from oe.utils import execute_pre_post_process diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index f1faf48..d11928f 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -877,7 +877,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d): bb.debug(2, "Considering setscene task: %s" % (str(taskdependees[task]))) def isNativeCross(x): - return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x + return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross") # We only need to trigger populate_lic through direct dependencies if taskdependees[task][1] == "do_populate_lic": however these changes were made as part of Recipe Specific Sysroots which identified the same bugs. With these bugs fixed, my bitbake patch avoids the population of the target sysroot as it should. 70831 18 leonardo.sandoval.gonzalez 2017-02-20 15:04:07 +0000 Testing your suggested patch. 70842 19 leonardo.sandoval.gonzalez 2017-02-20 20:40:12 +0000 (In reply to comment #17) > I went digging on this. Its not expected that dropping the postinstall > dependency code should cause such an increase in the size of the sysroot but > the investigation is correct and it does. > > What seems to happen is that the sstate code seems to think that for > example, coreutils:do_package_write_rpm depends on > libcap:do_populate_sysroot. The code sees that dependency and others and > therefore ends up populating the target sysroot. > > The target sysroot shouldn't be populated and this dependency is clearly > bogus (along with a number of other similar looking odd ones). > > I have a patch to bitbake's runqueue.py which fixes that code, its worth > noting that with that applied, this test case will then fail as there are > dependency bugs in oe-core. To > > To make this older revision work, you need to do: > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 6b08649..6599643 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -274,7 +274,7 @@ fakeroot python do_rootfs () { > do_rootfs[dirs] = "${TOPDIR}" > do_rootfs[cleandirs] += "${S} ${IMGDEPLOYDIR}" > do_rootfs[umask] = "022" > -addtask rootfs before do_build > +addtask rootfs before do_build after do_configure > > fakeroot python do_image () { > from oe.utils import execute_pre_post_process > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass > index f1faf48..d11928f 100644 > --- a/meta/classes/sstate.bbclass > +++ b/meta/classes/sstate.bbclass > @@ -877,7 +877,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d): > bb.debug(2, "Considering setscene task: %s" % > (str(taskdependees[task]))) > > def isNativeCross(x): > - return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x > + return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x > or x.endswith("-cross") > > # We only need to trigger populate_lic through direct dependencies > if taskdependees[task][1] == "do_populate_lic": > > > however these changes were made as part of Recipe Specific Sysroots which > identified the same bugs. With these bugs fixed, my bitbake patch avoids the > population of the target sysroot as it should. I have verified two things: * suggested patch by RP is already merged * tmp/sysroot folder sizes are basically the same # previous RSS commit lsandov1@yctb05:~/bug10884/poky-efd120e71418bc1cf3431194f38adf6986d1956e/build> du -h -d 1 tmp/sysroots 942M tmp/sysroots/x86_64-linux 807M tmp/sysroots/qemux86 1.8G tmp/sysroots # commit introducing RSS lsandov1@yctb05:~/bug10884/poky-9107d6ca144ac1f6169830dee25dae2dad775333/build> du -h -d 1 tmp/sysroots-components/ 946M tmp/sysroots-components/x86_64 36K tmp/sysroots-components/qemux86 759M tmp/sysroots-components/i586 3.4M tmp/sysroots-components/all 1.7G tmp/sysroots-components/ 70854 20 markus.lehtonen 2017-02-21 10:55:11 +0000 My tests indicate the same, i.e. that the regression is fixed by Richards patch. do_rootfs time dropped back to the level it was before the regression 70866 21 leonardo.sandoval.gonzalez 2017-02-21 15:58:13 +0000 Fixed by: ff734f0560649c0d8743aa1ebdaef8cfb20e6628 Author: Richard Purdie <richard.purdie@linuxfoundation.org> AuthorDate: Sat Feb 18 13:09:39 2017 -0800 Commit: Richard Purdie <richard.purdie@linuxfoundation.org> CommitDate: Sun Feb 19 06:28:47 2017 -0800 Parent: 10968b4 bitbake: bitbake-layers: Handle exception raised bytinfoil.prepare() Containing: master Follows: 2.3_M1 (897) bitbake: runqueue: Fix collapsed setscene dependency tree 72325 22 jose.perez.carranza 2017-04-11 20:22:39 +0000 Verified on 2.3 M3 rc2 poky commit: 415b72ffcbd26e5f3664370d8b2a9b8105fb6342 3623 2017-02-02 18:08:29 +0000 2017-02-02 18:08:29 +0000 Total time raid.log text/plain 8076 leonardo.sandoval.gonzalez cmVzdWx0cy01NzNjNjQ2LTIwMTYxMjIwMDEwMDAzL3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNl ZF90aW1lIjogMTA1MDIuMjc4Mjc2LApyZXN1bHRzLTU3M2M2NDYtMjAxNjEyMjAwNjAwMDMvcmVz dWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMDQ5Mi4wMDcwMzgsCnJlc3VsdHMtNTcz YzY0Ni0yMDE2MTIyMDExMDAwNC9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDEw NTAzLjQzNjQ0LApyZXN1bHRzLTU3M2M2NDYtMjAxNjEyMjAxNjAwMDQvcmVzdWx0cy5qc29uOjI6 ICAgICJlbGFwc2VkX3RpbWUiOiAxMDUzMy45MTM5NjYsCnJlc3VsdHMtNTczYzY0Ni0yMDE2MTIy MDIxMDAwNS9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDEwNTA4LjAwNDIyLApy ZXN1bHRzLTU3M2M2NDYtMjAxNjEyMjEwMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2Vk X3RpbWUiOiAxMDUxOS40NTkxMzEsCnJlc3VsdHMtNTczYzY0Ni0yMDE2MTIyMTA2MDAwNC9yZXN1 bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDEwNDk2LjU2ODE3MywKcmVzdWx0cy01NzNj NjQ2LTIwMTYxMjIxMTEwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTA0 NzkuNTE2NDczLApyZXN1bHRzLTU3M2M2NDYtMjAxNjEyMjExNjAwMDQvcmVzdWx0cy5qc29uOjI6 ICAgICJlbGFwc2VkX3RpbWUiOiAxMDQ3OS42NTg4NjUsCnJlc3VsdHMtNTczYzY0Ni0yMDE2MTIy MTIxMDAwMy9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDEwNTMyLjU0NzAxNywK cmVzdWx0cy01NzNjNjQ2LTIwMTYxMjIyMDEwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNl ZF90aW1lIjogMTA1MjcuMDcwNzg2LApyZXN1bHRzLTU3M2M2NDYtMjAxNjEyMjIwNjAwMDQvcmVz dWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMDU2OC42NjgwNTcsCnJlc3VsdHMtNTcz YzY0Ni0yMDE2MTIyMjExMDAwMy9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDEw NTgzLjMzMjc4NCwKcmVzdWx0cy01NzNjNjQ2LTIwMTYxMjIyMTYwMDAzL3Jlc3VsdHMuanNvbjoy OiAgICAiZWxhcHNlZF90aW1lIjogMTA1MTcuNTgzMzY0LApyZXN1bHRzLTU3M2M2NDYtMjAxNjEy MjIyMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMDUxNy42MDMzMTks CnJlc3VsdHMtNTczYzY0Ni0yMDE2MTIyMzAxMDAwMy9yZXN1bHRzLmpzb246MjogICAgImVsYXBz ZWRfdGltZSI6IDEwNTE5Ljg1MjkxOCwKcmVzdWx0cy01NzNjNjQ2LTIwMTYxMjIzMDYwMDA0L3Jl c3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTA1MTQuMjYzMjc3LApyZXN1bHRzLWRi YjI0N2MtMjAxNjEyMjMxMTAwMDMvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAx MDQ4OC41MTU5MDQsCnJlc3VsdHMtZGJiMjQ3Yy0yMDE2MTIyMzE2MDAwNC9yZXN1bHRzLmpzb246 MjogICAgImVsYXBzZWRfdGltZSI6IDEwNTMzLjEwOTc5LApyZXN1bHRzLWRiYjI0N2MtMjAxNjEy MjMyMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMDUzNS42NDExMzUs CnJlc3VsdHMtZGJiMjQ3Yy0yMDE2MTIyNDAxMDAwNC9yZXN1bHRzLmpzb246MjogICAgImVsYXBz ZWRfdGltZSI6IDEwNTI4LjM0MTAxMSwKcmVzdWx0cy1kYmIyNDdjLTIwMTYxMjI0MDYwMDAzL3Jl c3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTA1NTAuNTQzNjAyLApyZXN1bHRzLWRi YjI0N2MtMjAxNjEyMjQxMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAx MDU0MS42NjA0NDMsCnJlc3VsdHMtZGJiMjQ3Yy0yMDE2MTIyNDE2MDAwMy9yZXN1bHRzLmpzb246 MjogICAgImVsYXBzZWRfdGltZSI6IDEwNDc3LjgyODQ2MSwKcmVzdWx0cy1kYmIyNDdjLTIwMTYx MjI0MjEwMDAzL3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTA0OTQuNjEzNjY4 LApyZXN1bHRzLWRiYjI0N2MtMjAxNjEyMjUwMTAwMDMvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFw c2VkX3RpbWUiOiAxMDQ4Mi40MDMyNjgsCnJlc3VsdHMtZGJiMjQ3Yy0yMDE2MTIyNTA2MDAwNC9y ZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDEwNDk4Ljk4NTMwNywKcmVzdWx0cy1k YmIyNDdjLTIwMTYxMjI1MTEwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjog MTA1NTMuNjg4OTM2LApyZXN1bHRzLWRiYjI0N2MtMjAxNjEyMjUxNjAwMDQvcmVzdWx0cy5qc29u OjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMDQ3NC43NDk1MDUsCnJlc3VsdHMtZGJiMjQ3Yy0yMDE2 MTIyNTIxMDAwMy9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDEwNTE4LjY0NzEw MSwKcmVzdWx0cy1kYmIyNDdjLTIwMTYxMjI2MDEwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxh cHNlZF90aW1lIjogMTA1MjMuNjMzNjA4LApyZXN1bHRzLWRiYjI0N2MtMjAxNjEyMjcxNjAwMDgv cmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxNDIxNS44NTY4NjcsCnJlc3VsdHMt ZGJiMjQ3Yy0yMDE2MTIyNzIxMDAwNS9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6 IDExMjk2LjM3ODc5NywKcmVzdWx0cy1kYmIyNDdjLTIwMTYxMjI4MDEwMDA0L3Jlc3VsdHMuanNv bjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTEwOTUuNzQxMjA0LApyZXN1bHRzLWRiYjI0N2MtMjAx NjEyMjgxMTAwMDUvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMjM4Mi44ODM3 OTYsCnJlc3VsdHMtZGJiMjQ3Yy0yMDE2MTIyODE2MDAwNC9yZXN1bHRzLmpzb246MjogICAgImVs YXBzZWRfdGltZSI6IDExNjU0LjA5NjYwNSwKcmVzdWx0cy1kYmIyNDdjLTIwMTYxMjI4MjEwMDA0 L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE1MTUuMTAzNDI1LApyZXN1bHRz LWRiYjI0N2MtMjAxNjEyMjkwMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUi OiAxMTUzNS43MTA0NDgsCnJlc3VsdHMtZGJiMjQ3Yy0yMDE2MTIyOTA2MDAwNC9yZXN1bHRzLmpz b246MjogICAgImVsYXBzZWRfdGltZSI6IDEyMzc4LjUxODMyOSwKcmVzdWx0cy1kYmIyNDdjLTIw MTYxMjI5MTEwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE0MTIuMTU4 ODc4LApyZXN1bHRzLWRiYjI0N2MtMjAxNjEyMjkxNjAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJl bGFwc2VkX3RpbWUiOiAxMTQzMS42NTU5MjksCnJlc3VsdHMtZGJiMjQ3Yy0yMDE2MTIyOTIxMDAw NC9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDExNTM1LjgzODAxNywKcmVzdWx0 cy1kYmIyNDdjLTIwMTYxMjMwMDEwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1l IjogMTEzODYuOTU3NjIzLApyZXN1bHRzLWRiYjI0N2MtMjAxNjEyMzAwNjAwMDUvcmVzdWx0cy5q c29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTQzNy41NjI2MTEsCnJlc3VsdHMtZGJiMjQ3Yy0y MDE2MTIzMDExMDAwNS9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDExNDYwLjg3 MzMzNywKcmVzdWx0cy1kYmIyNDdjLTIwMTYxMjMwMTYwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAi ZWxhcHNlZF90aW1lIjogMTE0NzMuOTg1NDI2LApyZXN1bHRzLWRiYjI0N2MtMjAxNjEyMzAyMTAw MDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTUxNi4xNTgxNTgsCnJlc3Vs dHMtZGJiMjQ3Yy0yMDE2MTIzMTAxMDAwNC9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGlt ZSI6IDExNTExLjk5ODc3NSwKcmVzdWx0cy1kYmIyNDdjLTIwMTYxMjMxMDYwMDA0L3Jlc3VsdHMu anNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE0ODIuNjQwMzY2LApyZXN1bHRzLWRiYjI0N2Mt MjAxNjEyMzExMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTQ4Mi41 NTA3NTUsCnJlc3VsdHMtZGJiMjQ3Yy0yMDE2MTIzMTE2MDAwNC9yZXN1bHRzLmpzb246MjogICAg ImVsYXBzZWRfdGltZSI6IDExNTM5Ljk0NjQ2OSwKcmVzdWx0cy1kYmIyNDdjLTIwMTYxMjMxMjEw MDA1L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE0OTAuNDExOTYzLApyZXN1 bHRzLWRiYjI0N2MtMjAxNzAxMDEwMTAwMDUvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3Rp bWUiOiAxNjc0NS42NzQ5NjcsCnJlc3VsdHMtZGJiMjQ3Yy0yMDE3MDEwMTA2MDAwNS9yZXN1bHRz Lmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDE3NjA5LjMxNDM0NCwKcmVzdWx0cy1kYmIyNDdj LTIwMTcwMTAxMTEwMDEwL3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTI2MjQu MjMzNTgzLApyZXN1bHRzLWRiYjI0N2MtMjAxNzAxMDExNjAwMDUvcmVzdWx0cy5qc29uOjI6ICAg ICJlbGFwc2VkX3RpbWUiOiAxMTQ3Ni4wNjIyMDIsCnJlc3VsdHMtZGJiMjQ3Yy0yMDE3MDEwMTIx MDAwNC9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDExNDYzLjcxODI1MywKcmVz dWx0cy1kYmIyNDdjLTIwMTcwMTAyMDEwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90 aW1lIjogMTE0NDAuOTk2OTA2LApyZXN1bHRzLWRiYjI0N2MtMjAxNzAxMDIwNjAwMDUvcmVzdWx0 cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTQzNC45MjY1MjUsCnJlc3VsdHMtZGJiMjQ3 Yy0yMDE3MDEwMjExMDAwNS9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDExNTA5 LjM2ODk0OSwKcmVzdWx0cy1kYmIyNDdjLTIwMTcwMTAyMTYwMDA0L3Jlc3VsdHMuanNvbjoyOiAg ICAiZWxhcHNlZF90aW1lIjogMTE0MjYuOTc1NDMyLApyZXN1bHRzLWRiYjI0N2MtMjAxNzAxMDIy MTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTUxNi40MzgxMjYsCnJl c3VsdHMtZGJiMjQ3Yy0yMDE3MDEwMzAxMDAwNC9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRf dGltZSI6IDExNDQ4LjA2ODM2MSwKcmVzdWx0cy1kYmIyNDdjLTIwMTcwMTAzMDYwMDA0L3Jlc3Vs dHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE1MzUuOTI4OTIsCnJlc3VsdHMtZGJiMjQ3 Yy0yMDE3MDEwMzExMDAwNS9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDExNDIx Ljg5MjMyOCwKcmVzdWx0cy1kYmIyNDdjLTIwMTcwMTAzMTYwMDA0L3Jlc3VsdHMuanNvbjoyOiAg ICAiZWxhcHNlZF90aW1lIjogMTE0OTIuMDIwMDAyLApyZXN1bHRzLWRiYjI0N2MtMjAxNzAxMDMy MTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTU0NC4yMDQ1MzgsCnJl c3VsdHMtZGJiMjQ3Yy0yMDE3MDEwNDAxMDAwNC9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRf dGltZSI6IDExNDY2LjIxMzgzNSwKcmVzdWx0cy1kYmIyNDdjLTIwMTcwMTA0MDYwMDA0L3Jlc3Vs dHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE0NTIuMDI0ODA4LApyZXN1bHRzLWRiYjI0 N2MtMjAxNzAxMDQxMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTQz OS41ODU5NjMsCnJlc3VsdHMtZGJiMjQ3Yy0yMDE3MDEwNDE2MDAwNC9yZXN1bHRzLmpzb246Mjog ICAgImVsYXBzZWRfdGltZSI6IDExNDYyLjk5NjU4MiwKcmVzdWx0cy1kYmIyNDdjLTIwMTcwMTA0 MjEwMDA1L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE0MzkuMzIyOTY2LApy ZXN1bHRzLWRiYjI0N2MtMjAxNzAxMDUwMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2Vk X3RpbWUiOiAxMTQ4My42Mjc1OTMsCnJlc3VsdHMtOWY2YTEwNC0yMDE3MDEwNTA2MDAwNS9yZXN1 bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDExNTI2LjQ2MTEzMSwKcmVzdWx0cy05ZjZh MTA0LTIwMTcwMTA1MTEwMDA1L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTEz NzkuNTYzMDM2LApyZXN1bHRzLWUwMTZlYjEtMjAxNzAxMDUxNjAwMDQvcmVzdWx0cy5qc29uOjI6 ICAgICJlbGFwc2VkX3RpbWUiOiAxMTQ3Mi45NDEyNTIsCnJlc3VsdHMtZTAxNmViMS0yMDE3MDEw NTIxMDAwNC9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDExNDAxLjkzODgxNCwK cmVzdWx0cy1lMDE2ZWIxLTIwMTcwMTA2MDEwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNl ZF90aW1lIjogMTE0MzIuMTM1MDg2LApyZXN1bHRzLWUwMTZlYjEtMjAxNzAxMDYwNjAwMDUvcmVz dWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTQ3Ni41NTE2OTIsCnJlc3VsdHMtZTAx NmViMS0yMDE3MDEwNjExMDAwNS9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDEx NTE0LjI4ODYwOSwKcmVzdWx0cy1lMDE2ZWIxLTIwMTcwMTA2MTYwMDA1L3Jlc3VsdHMuanNvbjoy OiAgICAiZWxhcHNlZF90aW1lIjogMTE1MTkuMzUxNzc3LApyZXN1bHRzLWUwMTZlYjEtMjAxNzAx MDYyMTAwMDUvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTUxMy41OTk3NjQs CnJlc3VsdHMtZTAxNmViMS0yMDE3MDEwNzAxMDAwNC9yZXN1bHRzLmpzb246MjogICAgImVsYXBz ZWRfdGltZSI6IDExNTI1LjI5OTI5NywKcmVzdWx0cy1lMDE2ZWIxLTIwMTcwMTA3MDYwMDA0L3Jl c3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE1MzguNTA2MjQ1LApyZXN1bHRzLWUw MTZlYjEtMjAxNzAxMDcxMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAx MTQ0OS4zMjc5OTMsCnJlc3VsdHMtZTAxNmViMS0yMDE3MDEwNzE2MDAwNC9yZXN1bHRzLmpzb246 MjogICAgImVsYXBzZWRfdGltZSI6IDExNDA5LjczMzQ2MiwKcmVzdWx0cy1lMDE2ZWIxLTIwMTcw MTA3MjEwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE0MjYuNTY2NDM4 LApyZXN1bHRzLWUwMTZlYjEtMjAxNzAxMDgwMTAwMTMvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFw c2VkX3RpbWUiOiAxNjc5OC45MTI5MzQsCnJlc3VsdHMtZTAxNmViMS0yMDE3MDEwODA2MDAwOC9y ZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDE3NDE2LjAyMTg1MywKcmVzdWx0cy1l MDE2ZWIxLTIwMTcwMTA4MTEwMDA3L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjog MTI2NjMuNjIzNzIzLApyZXN1bHRzLWUwMTZlYjEtMjAxNzAxMDgxNjAwMDQvcmVzdWx0cy5qc29u OjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTQ5OC41NDI1NzcsCnJlc3VsdHMtZTAxNmViMS0yMDE3 MDEwODIxMDAwNS9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6IDExNTAzLjExNjM5 MywKcmVzdWx0cy1lMDE2ZWIxLTIwMTcwMTA5MDEwMDA0L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxh cHNlZF90aW1lIjogMTE0OTEuNTEwMDM1LApyZXN1bHRzLTg0MGUyMjEtMjAxNzAxMDkwNjAwMDQv cmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTUyOS45OTI3MDUsCnJlc3VsdHMt ODQwZTIyMS0yMDE3MDEwOTExMDAwNS9yZXN1bHRzLmpzb246MjogICAgImVsYXBzZWRfdGltZSI6 IDExNTYyLjE2NTgwNywKcmVzdWx0cy04NDBlMjIxLTIwMTcwMTA5MTYwMDA1L3Jlc3VsdHMuanNv bjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE0NTUuODE3OTQ1LApyZXN1bHRzLTg0MGUyMjEtMjAx NzAxMDkyMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUiOiAxMTQ5NS40MjQ1 NzksCnJlc3VsdHMtODQwZTIyMS0yMDE3MDExMDAxMDAwNS9yZXN1bHRzLmpzb246MjogICAgImVs YXBzZWRfdGltZSI6IDExNTE4LjI1NDk3MiwKcmVzdWx0cy04NDBlMjIxLTIwMTcwMTEwMDYwMDA1 L3Jlc3VsdHMuanNvbjoyOiAgICAiZWxhcHNlZF90aW1lIjogMTE1MzEuNjA0ODAzLApyZXN1bHRz LTg0MGUyMjEtMjAxNzAxMTAxMTAwMDQvcmVzdWx0cy5qc29uOjI6ICAgICJlbGFwc2VkX3RpbWUi OiAxMTQ0Ny4yMjMxNjIsCnJlc3VsdHMtODQwZTIyMS0yMDE3MDExMDE2MDAwNC9yZXN1bHRzLmpz b246MjogICAgImVsYXBzZWRfdGltZSI6IDExNTExLjk4NjM3MywK