The current persist_data uses a sqlite database that is shared among all the threads that wish to access it. The bitbake worker processes that spawn off are forked off of other processes which are also accessing the database, which causes the underlying sqlite database object to be cloned into the new child process. Once this is done, the game is over however because while the python sqlite state and underlying file descriptor are copied to the new process, the POSIX advisory locks that it had aren't. This means that there is essential no record to any other process that the new child exists and needs to be synchronized with other sqlite connections. This can cause all sorts of synchronization problems since the database pages might get flushed, or even worse unmapped from memory while the "hidden" child process is trying to read from it (this is the specific case I found; the child processes were dying with a SIGBUS error in sqlite).

sqlite admits this is a problem a little more succinctly and recommends that if you *must* carry an open connection across a fork(), you immediately close() (not sqlite3_close()!) the underlying file descriptor: https://www.sqlite.org/howtocorrupt.html#_carrying_an_open_database_connection_across_a_fork_ . However, python being what it is, this isn't a very tenable solution since 
1) we don't have direct access to the underlying file descriptor
2) couldn't prevent python from garbage collecting the sqlite database object
3) The os.register_at_fork() API wasn't added until Python 3.7
4) Even if the os.register_at_fork() handler was available, it's a pretty ugly hack.

Note that this problem probably isn't biting us at all currently because of the read-only and single process nature of the persist_data; that is data is mostly only read and only done so by a single process, but this is likely to change with https://bugzilla.yoctoproject.org/show_bug.cgi?id=13030 .

I believe that the best long term solution is to rework persist data to use a client server model where a single server process handles the sqlite database and manages requests on behalf of the clients.

I've started incubating a python module that can do such a thing efficiently for  read-mostly uses cases using shared memory here: https://github.com/JPEWdev/pyradur