Bug 13108 - log show many fetch error after commit: bitbake: persist_data: Retry database setup
Summary: log show many fetch error after commit: bitbake: persist_data: Retry database...
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 2.6
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.7 M2
Assignee: Joshua Watt
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2018-12-28 15:00 UTC by openatv
Modified: 2019-01-05 15:09 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description openatv 2018-12-28 15:00:09 UTC 
we test master branch and after this commit
https://github.com/openembedded/bitbake/commit/5a2a95b0396e39662968690b3065d2f88167a71c

we get many fetch issue i must start build 4-5 times and its running without the fetch error any idea  here the log

ERROR: Traceback (most recent call last):
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 423, in expandWithRefs
    s = __expand_python_regexp__.sub(varparse.python_sub, s)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 148, in python_sub
    value = utils.better_eval(codeobj, DataContext(self.d), {'d' : self.d})
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/utils.py", line 429, in better_eval
    return eval(source, ctx, locals)
  File "Var <SRCPV>", line 1, in <module>
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 758, in get_srcrev
    fetcher = Fetch(d.getVar('SRC_URI').split(), d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 1604, in __init__
    self.ud[url] = FetchData(url, d, localonly)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 1275, in __init__
    self.method.urldata_init(self, d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/git.py", line 242, in urldata_init
    ud.setup_revisions(d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 1305, in setup_revisions
    self.revisions[name] = srcrev_internal_helper(self, d, name)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 1170, in srcrev_internal_helper
    srcrev = ud.method.latest_revision(ud, d, name)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 1560, in latest_revision
    revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 308, in persist
    return SQLTable(cachefile, domain)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 98, in __init__
    self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT PRIMARY KEY NOT NULL, value TEXT);" % table)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 55, in wrap_func
    self.reconnect()
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 115, in reconnect
    self._setup_database()
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 60, in wrap_func
    return f(self, *args, **kwargs)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 89, in wrap_func
    return f(self, cursor, *args, **kwargs)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 103, in _setup_database
    cursor.execute("pragma synchronous = off;")
sqlite3.OperationalError: locking protocol

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/oe1/nextp3/build-enviroment/bitbake/bin/bitbake-worker", line 236, in child
    the_data = bb_cache.loadDataFull(fn, appends)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/cache.py", line 339, in loadDataFull
    bb_data = self.load_bbfile(virtualfn, appends, virtonly=True)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/cache.py", line 352, in load_bbfile
    datastores = parse_recipe(bb_data, bbfile, appends, mc)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/cache.py", line 315, in parse_recipe
    bb_data = bb.parse.handle(bbfile, bb_data)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/parse/__init__.py", line 117, in handle
    return h['handle'](fn, data, include)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 155, in handle
    return ast.multi_finalize(fn, d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/parse/ast.py", line 399, in multi_finalize
    finalize(fn, d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/parse/ast.py", line 358, in finalize
    runAnonFuncs(d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/parse/ast.py", line 342, in runAnonFuncs
    bb.utils.better_exec("\n".join(code), {"d": d})
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/utils.py", line 404, in better_exec
    exec(code, get_context(), context)
  File "<code>", line 3, in <module>
  File "/home/oe1/nextp3/build-enviroment/openembedded-core/meta/classes/base.bbclass", line 666, in __anon_694__home_oe1_nextp3_build_enviroment_openembedded_core_meta_classes_base_bbclass
    fpaths = (d.getVar('FILESPATH') or '').split(':')
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 616, in getVar
    return self.getVarFlag(var, "_content", expand, noweakdefault, parsing)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 814, in getVarFlag
    parser = self.expandWithRefs(value, cachename)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 421, in expandWithRefs
    s = __expand_var_regexp__.sub(varparse.var_sub, s)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 109, in var_sub
    var = self.d.getVarFlag(key, "_content")
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 814, in getVarFlag
    parser = self.expandWithRefs(value, cachename)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 421, in expandWithRefs
    s = __expand_var_regexp__.sub(varparse.var_sub, s)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 109, in var_sub
    var = self.d.getVarFlag(key, "_content")
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 814, in getVarFlag
    parser = self.expandWithRefs(value, cachename)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 421, in expandWithRefs
    s = __expand_var_regexp__.sub(varparse.var_sub, s)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 109, in var_sub
    var = self.d.getVarFlag(key, "_content")
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 814, in getVarFlag
    parser = self.expandWithRefs(value, cachename)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 436, in expandWithRefs
    raise ExpansionError(varname, s, exc).with_traceback(tb) from exc
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 423, in expandWithRefs
    s = __expand_python_regexp__.sub(varparse.python_sub, s)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/data_smart.py", line 148, in python_sub
    value = utils.better_eval(codeobj, DataContext(self.d), {'d' : self.d})
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/utils.py", line 429, in better_eval
    return eval(source, ctx, locals)
  File "Var <SRCPV>", line 1, in <module>
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 758, in get_srcrev
    fetcher = Fetch(d.getVar('SRC_URI').split(), d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 1604, in __init__
    self.ud[url] = FetchData(url, d, localonly)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 1275, in __init__
    self.method.urldata_init(self, d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/git.py", line 242, in urldata_init
    ud.setup_revisions(d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 1305, in setup_revisions
    self.revisions[name] = srcrev_internal_helper(self, d, name)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 1170, in srcrev_internal_helper
    srcrev = ud.method.latest_revision(ud, d, name)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/fetch2/__init__.py", line 1560, in latest_revision
    revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 308, in persist
    return SQLTable(cachefile, domain)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 98, in __init__
    self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT PRIMARY KEY NOT NULL, value TEXT);" % table)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 55, in wrap_func
    self.reconnect()
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 115, in reconnect
    self._setup_database()
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 60, in wrap_func
    return f(self, *args, **kwargs)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 89, in wrap_func
    return f(self, cursor, *args, **kwargs)
  File "/home/oe1/nextp3/build-enviroment/bitbake/lib/bb/persist_data.py", line 103, in _setup_database
    cursor.execute("pragma synchronous = off;")
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception OperationalError: locking protocol
Comment 2 openatv 2019-01-03 17:39:17 UTC 
this fix the issue

i test new on fresh build no fetch issue 

thx for fix it
Comment 3 nickersk 2019-01-03 17:52:19 UTC 
I can confirm, this patch fixed the issue.
Thank you.