9034 2016-02-02 13:25:46 +0000 The PR Service deletes the -wal and -shm sqlite files when daemonizing 2016-03-14 15:21:18 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 2.0.2 x86 Multiple VERIFIED FIXED High normal 2.1 M3 1 Diego.SantaCruz richard.purdie bogdanx.a.voiculescu costin.c.constantin poky.bs.watcher poky.watcher costin.c.constantin --- --- No (bug/feature does not impact docs) oldest_to_newest 58443 0 2962 Diego.SantaCruz 2016-02-02 13:25:46 +0000 Created attachment 2962 Patch fixing -wal and -shm sqlite file deletion When daemonizing the PR service the -wal and -shm sqlite files are being deleted, although they should never be (see https://www.sqlite.org/wal.html and https://www.sqlite.org/tempfiles.html). While the daemonized process keeps the file descriptors open (seen from /prid<pid>/fd on linux) and thus a clean exit does not loose any data, a power outage would loose all data in the WAL as the -wal file is unavailable when sqlite does its recovery, so even committed transactions would be lost. Removing these files also breaks sqlite collaboration between processes and furthermore prevents taking proper backups without stopping the PR service (e.g., with sqlite3 shell's ".backup" command). The reason the files are deleted is that the DB connection is opened in the initial process, before forking for daemonization. When the DB connection is closed by the exiting parent processes it can delete the -wal and -shm files if it considers itself to be the last connection to the DB. This is easily fixed by opening the DB connection after all forking. The attached patch does this (on top of jethro branch). With this I have been able to start the PR Service in daemon mode and the -wal and -shm files do exist and I can take proper backups. 58447 1 richard.purdie 2016-02-02 14:17:10 +0000 Could you send this patch and those for the other PR server bugs you've opened to the bitbake-devel list please? 58450 2 Diego.SantaCruz 2016-02-02 14:20:02 +0000 Sure! I'll subscribe to the list and send them in. 58922 3 richard.purdie 2016-02-11 20:58:15 +0000 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=be032fc40eb315841fdd1f69c780726619ea1bad 59496 4 costin.c.constantin 2016-02-26 14:31:19 +0000 Hello, Tested on latest (23056103c949b498c23b47579e8dd57ce78e6ed9) master on an Ubuntu 15.10 host. Steps performed: 1) add in local.conf: PRSERV_HOST = "localhost:0" INHERIT += "buildhistory" BUILDHISTORY_COMMIT = "1" 2) start prserv: bitbake-prserv --start 3) ls builddir: ls -1 poky/build ... prserv.sqlite3 prserv.sqlite3-shm prserv.sqlite3-wal ... 4) a) kill 9 prserv_pid ps xau | grep python ... 24681 ... python /media/yp_builds/2poky/bitbake/bin/bitbake-prserv --start ... kill -9 24681 b) simulate power outage by unplugging PC for both 4.a) and 4.b) -shm and -wal files were still present Marking as verified. 2962 2016-02-02 13:25:46 +0000 2016-02-02 13:25:46 +0000 Patch fixing -wal and -shm sqlite file deletion 0001-bitbake-prserv-wal-and-shm-sqlite-lost-when-daemoniz.patch text/plain 2650 Diego.SantaCruz RnJvbSAyYTg2NmNmMDliZTdmNmNkMWU3ZDYxNzQxOGU4ODBlNDEwMDEzMjY3IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBEaWVnbyBTYW50YSBDcnV6IDxEaWVnby5TYW50YUNydXpAc3Bp bmV0aXguY29tPgpEYXRlOiBUdWUsIDIgRmViIDIwMTYgMTM6MDQ6MjAgKzAxMDAKU3ViamVjdDog W1BBVENIXSBiaXRiYWtlOiBwcnNlcnY6IC13YWwgYW5kIC1zaG0gc3FsaXRlIGxvc3Qgd2hlbiBk YWVtb25pemluZwoKV2hlbiBkYWVtb25pemluZyB0aGUgUFIgc2VydmljZSB0aGUgLXdhbCBhbmQg LXNobSBzcWxpdGUgZmlsZXMgd2VyZQpiZWluZyBkZWxldGVkLCBhbHRob3VnaCB0aGV5IHNob3Vs ZCBuZXZlciBiZS4gV2hpbGUgdGhlIGRhZW1vbml6ZWQKcHJvY2VzcyBrZWVwcyB0aGUgZmlsZSBk ZXNjcmlwdG9ycyBvcGVuIGFuZCB0aHVzIGEgY2xlYW4gZXhpdCBkb2VzCm5vdCBsb29zZSBhbnkg ZGF0YSwgYSBwb3dlciBvdXRhZ2Ugd291bGQgbG9vc2UgYWxsIGRhdGEgaW4gdGhlIFdBTC4KUmVt b3ZpbmcgdGhlc2UgZmlsZXMgYWxzbyBicmVha3Mgc3FsaXRlIGNvbGxhYm9yYXRpb24gYmV0d2Vl bgpwcm9jZXNzZXMgYW5kIGZ1cnRoZXJtb3JlIHByZXZlbnRzIHRha2luZyBwcm9wZXIgYmFja3Vw cyB3aXRob3V0CnN0b3BwaW5nIHRoZSBQUiBzZXJ2aWNlLgoKVGhlIHJlYXNvbiB0aGlzIGhhcHBl bnMgaXMgdGhhdCB0aGUgREIgY29ubmVjdGlvbiBpcyBvcGVuZWQgaW4KdGhlIGluaXRpYWwgcHJv Y2VzcywgYmVmb3JlIGZvcmtpbmcgZm9yIGRhZW1vbml6YXRpb24uIFdoZW4gdGhlCkRCIGNvbm5l Y3Rpb24gaXMgY2xvc2VkIGJ5IHRoZSBleGl0aW5nIHBhcmVudCBwcm9jZXNzZXMgaXQgY2FuCmRl bGV0ZSB0aGUgLXdhbCBhbmQgLXNobSBmaWxlcyBpZiBpdCBjb25zaWRlcnMgaXRzZWxmIHRvIGJl IHRoZQpsYXN0IGNvbm5lY3Rpb24gdG8gdGhlIERCLiBUaGlzIGlzIGVhc2lseSBmaXhlZCBieSBv cGVuaW5nIHRoZQpEQiBjb25uZWN0aW9uIGFmdGVyIGFsbCBmb3JraW5nLgoKU2lnbmVkLW9mZi1i eTogRGllZ28gU2FudGEgQ3J1eiA8RGllZ28uU2FudGFDcnV6QHNwaW5ldGl4LmNvbT4KLS0tCiBi aXRiYWtlL2xpYi9wcnNlcnYvc2Vydi5weSB8IDEzICsrKysrKysrLS0tLS0KIDEgZmlsZSBjaGFu Z2VkLCA4IGluc2VydGlvbnMoKyksIDUgZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvYml0YmFr ZS9saWIvcHJzZXJ2L3NlcnYucHkgYi9iaXRiYWtlL2xpYi9wcnNlcnYvc2Vydi5weQppbmRleCBl YWZjM2FhLi5jNjVkZDNjIDEwMDY0NAotLS0gYS9iaXRiYWtlL2xpYi9wcnNlcnYvc2Vydi5weQor KysgYi9iaXRiYWtlL2xpYi9wcnNlcnYvc2Vydi5weQpAQCAtNjIsOSArNjIsNiBAQCBjbGFzcyBQ UlNlcnZlcihTaW1wbGVYTUxSUENTZXJ2ZXIpOgogICAgICAgICBzZWxmLnJlZ2lzdGVyX2Z1bmN0 aW9uKHNlbGYuaW1wb3J0b25lLCAiaW1wb3J0b25lIikKICAgICAgICAgc2VsZi5yZWdpc3Rlcl9p bnRyb3NwZWN0aW9uX2Z1bmN0aW9ucygpCiAKLSAgICAgICAgc2VsZi5kYiA9IHByc2Vydi5kYi5Q UkRhdGEoc2VsZi5kYmZpbGUpCi0gICAgICAgIHNlbGYudGFibGUgPSBzZWxmLmRiWyJQUk1BSU4i XQotCiAgICAgICAgIHNlbGYucmVxdWVzdHF1ZXVlID0gUXVldWUuUXVldWUoKQogICAgICAgICBz ZWxmLmhhbmRsZXJ0aHJlYWQgPSB0aHJlYWRpbmcuVGhyZWFkKHRhcmdldCA9IHNlbGYucHJvY2Vz c19yZXF1ZXN0X3RocmVhZCkKICAgICAgICAgc2VsZi5oYW5kbGVydGhyZWFkLmRhZW1vbiA9IEZh bHNlCkBAIC05OCwxMCArOTUsMTIgQEAgY2xhc3MgUFJTZXJ2ZXIoU2ltcGxlWE1MUlBDU2VydmVy KToKICAgICAgICAgICAgIHNlbGYudGFibGUuc3luY19pZl9kaXJ0eSgpCiAKICAgICBkZWYgc2ln aW50X2hhbmRsZXIoc2VsZiwgc2lnbnVtLCBzdGFjayk6Ci0gICAgICAgIHNlbGYudGFibGUuc3lu YygpCisgICAgICAgIGlmIHNlbGYudGFibGU6CisgICAgICAgICAgICBzZWxmLnRhYmxlLnN5bmMo KQogCiAgICAgZGVmIHNpZ3Rlcm1faGFuZGxlcihzZWxmLCBzaWdudW0sIHN0YWNrKToKLSAgICAg ICAgc2VsZi50YWJsZS5zeW5jKCkKKyAgICAgICAgaWYgc2VsZi50YWJsZToKKyAgICAgICAgICAg IHNlbGYudGFibGUuc3luYygpCiAgICAgICAgIHJhaXNlIFN5c3RlbUV4aXQKIAogICAgIGRlZiBw cm9jZXNzX3JlcXVlc3Qoc2VsZiwgcmVxdWVzdCwgY2xpZW50X2FkZHJlc3MpOgpAQCAtMTQxLDYg KzE0MCwxMCBAQCBjbGFzcyBQUlNlcnZlcihTaW1wbGVYTUxSUENTZXJ2ZXIpOgogICAgICAgICBz ZWxmLnF1aXQgPSBGYWxzZQogICAgICAgICBzZWxmLnRpbWVvdXQgPSAwLjUKIAorICAgICAgICAj IERCIGNvbm5lY3Rpb24gbXVzdCBiZSBjcmVhdGVkIGFmdGVyIGFsbCBmb3JrcworICAgICAgICBz ZWxmLmRiID0gcHJzZXJ2LmRiLlBSRGF0YShzZWxmLmRiZmlsZSkKKyAgICAgICAgc2VsZi50YWJs ZSA9IHNlbGYuZGJbIlBSTUFJTiJdCisKICAgICAgICAgbG9nZ2VyLmluZm8oIlN0YXJ0ZWQgUFJT ZXJ2ZXIgd2l0aCBEQmZpbGU6ICVzLCBJUDogJXMsIFBPUlQ6ICVzLCBQSUQ6ICVzIiAlCiAgICAg ICAgICAgICAgICAgICAgICAoc2VsZi5kYmZpbGUsIHNlbGYuaG9zdCwgc2VsZi5wb3J0LCBzdHIo b3MuZ2V0cGlkKCkpKSkKIAotLSAKMS45LjQKCg==