10695 2016-11-21 12:53:57 +0000 sqlite: upstream performance problem 2017-01-26 16:39:24 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 2.3 x86 Multiple RESOLVED FIXED Medium+ normal 2.3 M2 1 patrick.ohly maxin.john maxin.john meta.mr.watcher meta.watcher ross.burton sgw --- --- No (bug/feature does not impact docs) oldest_to_newest 68456 0 patrick.ohly 2016-11-21 12:53:57 +0000 Patch https://patchwork.openembedded.org/patch/132885/ reverted an upstream patch in sqlite which caused a performance regression (see bug #10367). As discussed in response to that patch, upstream needs to be notified about that problem and either also needs to revert the patch or find some other solution for whatever they were trying to achieve in the first place. If we don't do that, updating to future versions of sqlite will be hard. 69465 1 jianxun.zhang 2017-01-04 01:51:04 +0000 I have sent an email to sqlite user mailing list, CCing Patrick, pseudo maintainer and Saul. 69549 2 3588 jianxun.zhang 2017-01-05 19:31:32 +0000 Created attachment 3588 Correspondence with Sqlite3 team 69550 3 jianxun.zhang 2017-01-05 19:39:03 +0000 Refer to the attached correspondence between us and Sqlite3 team. Although the fix is in 3.16.0 but we are suggested to wait for 3.16.2 release which should come out very soon. I think once the Sqlite 3 component in YP is updated to 3.16.2 or any newer versions than it we can re-benchmark without the reverting patch then decide the next to do. Add maxin.john@intel.com the maintainer of sqlite 3 in watch list. Thanks 69702 4 maxin.john 2017-01-11 10:03:24 +0000 (In reply to comment #3) ... > I think once the Sqlite 3 component in YP is updated to 3.16.2 or any newer > versions than it we can re-benchmark without the reverting patch then decide > the next to do. > > Add maxin.john@intel.com the maintainer of sqlite 3 in watch list. > Patch for sqlite3 version upgrade to 3.16.2 sent to oe-core mailing list: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=ross/mut&id=41b9dd7ddc0b270638ccfb57429d355b1eea5276 69731 5 jianxun.zhang 2017-01-11 20:32:39 +0000 Maxin. Thanks lot! BTW, did you have any chance to verify no performance regression with the new version in the patch? 69740 6 maxin.john 2017-01-12 09:38:03 +0000 (In reply to comment #5) > Maxin. > Thanks lot! > BTW, did you have any chance to verify no performance regression with the > new version in the patch? # /usr/bin/time -v bitbake core-image-minimal Without the upgrade: Summary: There were 2 WARNING messages shown. Command being timed: "bitbake core-image-minimal" User time (seconds): 11889.38 System time (seconds): 3087.18 Percent of CPU this job got: 300% Elapsed (wall clock) time (h:mm:ss or m:ss): 1:23:08 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 802816 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 7888 Minor (reclaiming a frame) page faults: 721841293 Voluntary context switches: 19513415 Involuntary context switches: 5332647 Swaps: 0 File system inputs: 5243800 File system outputs: 54152904 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 After upgrade: Summary: There were 2 WARNING messages shown. Command being timed: "bitbake core-image-minimal" User time (seconds): 12026.36 System time (seconds): 3064.97 Percent of CPU this job got: 341% Elapsed (wall clock) time (h:mm:ss or m:ss): 1:13:40 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 802860 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 7551 Minor (reclaiming a frame) page faults: 721237606 Voluntary context switches: 17455269 Involuntary context switches: 7620493 Swaps: 0 File system inputs: 5461224 File system outputs: 53941600 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 In addition to this, Ross mentioned that this upgrade will be tested on performance cluster before pushing to master. 69764 7 sgw 2017-01-12 16:45:59 +0000 Can this patch be tested also with the Ostro failing issue, I know it's not being supported, but it did show the worst performance regression. 70189 8 ross.burton 2017-01-26 16:39:24 +0000 Upstream fixed this in 3.16.2, merged in oe-core 05317fe9f11565d40b84ad71300b39c990a53f6d. 3588 2017-01-05 19:31:32 +0000 2017-01-05 19:31:32 +0000 Correspondence with Sqlite3 team Re_ [sqlite] Commit ad601c7962 degrade performance in a client program.eml message/rfc822 1659 jianxun.zhang Q29udGVudC1UeXBlOiB0ZXh0L3BsYWluOw0KCWNoYXJzZXQ9dXMtYXNjaWkNCk1pbWUtVmVyc2lv bjogMS4wIChNYWMgT1MgWCBNYWlsIDEwLjIgXCgzMjU5XCkpDQpTdWJqZWN0OiBSZTogW3NxbGl0 ZV0gQ29tbWl0IGFkNjAxYzc5NjIgZGVncmFkZSBwZXJmb3JtYW5jZSBpbiBhIGNsaWVudA0KIHBy b2dyYW0NClgtVW5pdmVyc2FsbHktVW5pcXVlLUlkZW50aWZpZXI6IDE2NkMwOUIyLTlENjUtNDEz NS05QTI4LTREMzg2NDUyRDdDMw0KRnJvbTogSmlhbnh1biBaaGFuZyA8amlhbnh1bi56aGFuZ0Bs aW51eC5pbnRlbC5jb20+DQpJbi1SZXBseS1UbzogPENBTHdKPU13QnppPXlraXgzbnc4dC01Nlgw T2lfaFBZeFFLSEErUVhXb3lBZDhvc3dnQUBtYWlsLmdtYWlsLmNvbT4NCkRhdGU6IFRodSwgNSBK YW4gMjAxNyAxMToyODo1MSAtMDgwMA0KQ2M6IG1hcmsuaGF0bGVAd2luZHJpdmVyLmNvbSwNCiBT YXVsIFdvbGQgPHNnd0BsaW51eC5pbnRlbC5jb20+DQpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5n OiBxdW90ZWQtcHJpbnRhYmxlDQpNZXNzYWdlLUlkOiA8NERDOUFBRDUtQUMwRi00ODBELTg0MjMt QTJCQ0NGQTM0NDRGQGxpbnV4LmludGVsLmNvbT4NClJlZmVyZW5jZXM6IDw4QTMwNjU5MC1CNDc5 LTQ1RDMtOUI0Ri1GNzM0RDFBRUY3QjVAbGludXguaW50ZWwuY29tPg0KIDxDQUx3Sj1Nd0J6aT15 a2l4M253OHQtNTZYME9pX2hQWXhRS0hBK1FYV295QWQ4b3N3Z0FAbWFpbC5nbWFpbC5jb20+DQpU bzogUmljaGFyZCBIaXBwIDxkcmhAc3FsaXRlLm9yZz4NCg0KUmljaGFyZCwNCkkgYXBwcmVjaWF0 ZSB5b3VyIHF1aWNrIHJlcGx5LiBXZSB3aWxsIHdhaXQgZm9yIHRoZSBuZXcgdmVyc2lvbiBhcyB5 b3UgPQ0Kc3VnZ2VzdGVkLg0KDQpUaGFua3MgbG90IQ0KDQo+IE9uIEphbiA1LCAyMDE3LCBhdCAx MDoxOCBBTSwgUmljaGFyZCBIaXBwIDxkcmhAc3FsaXRlLm9yZz4gd3JvdGU6DQo+PTIwDQo+IEkg YXBvbG9naXplIGZvciBiZWluZyBhIGxpdHRsZSBzaG9ydCBpbiBteSByZXBseSB0byB5b3VyIG9y aWdpbmFsDQo+IHBvc3QuICBJdCdzIGJlZW4gYSBzdHJlc3NmdWwgd2VlayBhbmQgSSBmZWFyIHRo YXQgSSBtYXkgYmUgYSBsaXR0bGUNCj4gZ3J1bXB5Lg0KPj0yMA0KPiBXZSByZWFsbHkgZG8gYXBw cmVjaWF0ZSB5b3VyIHBlcmZvcm1hbmNlIGlzc3VlIHJlcG9ydC4gIFRoZSBwcm9ibGVtDQo+IHNo b3VsZCBiZSBmaXhlZCBpbiB0aGUgMy4xNi4yIHJlbGVhc2UuICBJdCBzIGFsc28gZml4ZWQgaW4g dGhlIDMuMTYuMA0KPiBhbmQgMy4xNi4xIHJlbGVhc2VzLCBidXQgeW91IHNob3VsZCBub3QgdXNl IHRob3NlIGZvciBvdGhlciByZWFzb25zLg0KPiBXYWl0IG9uIHRoZSAzLjE2LjIgcmVsZWFzZSB3 aGljaCB3aWxsIGJlIG91dCAoSSBob3BlKSBsYXRlciB0b2RheS4NCj49MjANCj4gUGxlYXNlIGZl ZWwgZnJlZSB0byByZXBvcnQgYW55IG90aGVyIHByb2JsZW1zIG9yIGRpZmZpY3VsdGllcyB5b3Ug aGF2ZQ0KPiB0byB0aGUgc3FsaXRlLXVzZXJzQG1haWxpbmdsaXN0cy5zcWxpdGUub3JnIG1haWxp bmcgbGlzdCwgb3IgZGlyZWN0bHkNCj4gdG8gbWUuDQo+PTIwDQo+IFRoYW5rcyBmb3IgdXNpbmcg U1FMaXRlLg0KPj0yMA0KPiAtLT0yMA0KPiBELiBSaWNoYXJkIEhpcHANCj4gZHJoQHNxbGl0ZS5v cmcNCg0K