Bug 9034

Summary: The PR Service deletes the -wal and -shm sqlite files when daemonizing
Product: [Build System, Metadata & Runtime] BitBake Reporter: Diego Santa cruz <Diego.SantaCruz>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: VERIFIED FIXED QA Contact: Costin <costin.c.constantin>
Severity: normal    
Priority: High CC: bogdanx.a.voiculescu, costin.c.constantin, poky.bs.watcher, poky.watcher
Version: 2.0.2   
Target Milestone: 2.1 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Patch fixing -wal and -shm sqlite file deletion none

Description Diego Santa cruz 2016-02-02 13:25:46 UTC
Created attachment 2962 [details]
Patch fixing -wal and -shm sqlite file deletion

When daemonizing the PR service the -wal and -shm sqlite files are being deleted, although they should never be (see https://www.sqlite.org/wal.html and https://www.sqlite.org/tempfiles.html).

While the daemonized process keeps the file descriptors open (seen from /prid<pid>/fd on linux) and thus a clean exit does not loose any data, a power outage would loose all data in the WAL as the -wal file is unavailable when sqlite does its recovery, so even committed transactions would be lost.

Removing these files also breaks sqlite collaboration between processes and furthermore prevents taking proper backups without stopping the PR service (e.g., with sqlite3 shell's ".backup" command).

The reason the files are deleted is that the DB connection is opened in the initial process, before forking for daemonization. When the DB connection is closed by the exiting parent processes it can delete the -wal and -shm files if it considers itself to be the last connection to the DB. This is easily fixed by opening the DB connection after all forking.

The attached patch does this (on top of jethro branch). With this I have been able to start the PR Service in daemon mode and the -wal and -shm files do exist and I can take proper backups.
Comment 1 Richard Purdie 2016-02-02 14:17:10 UTC
Could you send this patch and those for the other PR server bugs you've opened to the bitbake-devel list please?
Comment 2 Diego Santa cruz 2016-02-02 14:20:02 UTC
Sure! I'll subscribe to the list and send them in.
Comment 4 Costin 2016-02-26 14:31:19 UTC
Hello,

Tested on latest (23056103c949b498c23b47579e8dd57ce78e6ed9) master on an Ubuntu 15.10 host. 

Steps performed:
1) add in local.conf: 
PRSERV_HOST = "localhost:0"
INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"

2) start prserv:
bitbake-prserv --start

3) ls builddir:
ls -1 poky/build
...
prserv.sqlite3  
prserv.sqlite3-shm  
prserv.sqlite3-wal
...

4) a) kill 9 prserv_pid 
     ps xau | grep python
      ...
     24681  ...  python /media/yp_builds/2poky/bitbake/bin/bitbake-prserv --start
     ...
     kill -9 24681

   b) simulate power outage by unplugging PC

for both 4.a) and 4.b) -shm and -wal files were still present

Marking as verified.