12374 2017-11-22 07:31:38 +0000 do_rootfs failed when len(TMPDIR) == 410 2023-01-16 15:09:09 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 2.5 x86 Multiple RESOLVED FIXED Medium+ normal 4.2 M3 1 liezhi.yang yoann.congal chee.yang.lee meta.mr.watcher meta.watcher randy.macleod yoann.congal --- --- No (bug/feature does not impact docs) oldest_to_newest 78314 0 liezhi.yang 2017-11-22 07:31:38 +0000 when len(TMPDIR) == 410 and run bitbake core-image-minimal File "/workspace2/lyang1/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaa/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/dnf/base.py", line 782, in _run_transaction lastdbv = self.history.last() File "/workspace2/lyang1/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaa/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/dnf/base.py", line 526, in <lambda> history = property(fget=lambda self: self._getHistory(), File "/workspace2/lyang1/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaa/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/dnf/base.py", line 523, in _getHistory releasever=releasever) File "/workspace2/lyang1/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaa/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/dnf/yum/history.py", line 775, in __init__ self._create_db_file() File "/workspace2/lyang1/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaa/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/dnf/yum/history.py", line 1710, in _create_db_file cur = self._get_cursor() File "/workspace2/lyang1/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaaaa/aaaaaaaaa/aaaaaa/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/dnf/yum/history.py", line 799, in _get_cursor self._conn = sqlite.connect(self._db_file) sqlite3.OperationalError: unable to open database file 94036 1 chee.yang.lee 2022-09-20 05:53:16 +0000 related bug at createrepo_c https://github.com/rpm-software-management/createrepo_c/issues/220 94642 2 yoann.congal 2023-01-08 09:10:47 +0000 I've looked into this and it really looks like a hard limit on path length in the sqlite code. I guess we can do 2 things about this : - Setup an explicit error when this path length is used (saying we don't support this, sadly) - Trying to push an improvement to sqlite to remove this limitation 94649 3 yoann.congal 2023-01-10 09:54:17 +0000 https://github.dev/sqlite/sqlite/blob/master/src/os_unix.c#L179-L180 : #define MAX_PATHNAME 512 This looks quite low... I can reproduce this limitation with the "sqlite3" CLI tool : T="aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/aaaaaaaaa/" # T is 480 char long sqlite3 t.db 'create table tbl(x int);' # create t.db mkdir -p "$T" # create long named directory sqlite3 t.db .dump # Opens OK cp t.db $T/ sqlite3 $T/t.db .dump # fails to open Error: unable to open database "aaaaaaaaa/[..snip]/aaaaaaaaa//t.db": unable to open database file => This confirms this bug is a limitation of sqlite 94651 4 yoann.congal 2023-01-10 17:02:03 +0000 Some elements discussed during the "Yocto Project Technical Team Meeting / Engineering Sync": * SQlite ticket about this limit closed as "Won't fix" : https://sqlite.org/cgi/src/tktview/c060923a5422590b3734 * meta/classes-global/sanity.bbclass contains a Sanity test: * status.addresult(check_path_length(tmpdir, "TMPDIR", 410)) * My guess is that bitbake tries to open a sqlite database *before* reaching the sanity.bbclass tests (Will check) * Maybe a trick where we exchange path length between CWD and "relative path" ? (Did not find it at first, will try further) * We may need to put this check in bitbake itself 94655 5 yoann.congal 2023-01-10 22:14:28 +0000 > Some elements discussed during the "Yocto Project Technical Team Meeting / > Engineering Sync": > * meta/classes-global/sanity.bbclass contains a Sanity test: > * status.addresult(check_path_length(tmpdir, "TMPDIR", 410)) > * My guess is that bitbake tries to open a sqlite database *before* reaching > the sanity.bbclass tests (Will check) Confirmed, bitbake does connect to the sqlite database before the sanity.bbclass checks. Note: those checks are not made every time (remove cache/sanity_info to force this sanity check) $ rm cache/sanity_info ; bitbake -p NOTE: Connecting to self.cachefile='.../cache/bb_persist_data.sqlite3' (len(self.cachefile)=109) # bitbake/lib/bb/persist_data.py:103 NOTE: Checking path length of TMPDIR : tmpdir='.../tmp' len(tmpdir)=83 vs 410 # => sanity.bbclass:735 Loading cache: 100% |##########################################################################################################################################################| Time: 0:00:00 Loaded 2897 entries from dependency cache. 94656 6 yoann.congal 2023-01-10 22:42:50 +0000 Backtrace of the call that happens before the sanity check : NOTE: bb.persist_data.persist called here : .../poky/bitbake/lib/bb/fetch2/__init__.py:501 (fetcher_init) NOTE: backtrace = NOTE: File ".../poky/bitbake/bin/bitbake-server", line 54, in <module> bb.server.process.execServer(lockfd, readypipeinfd, lockname, sockname, timeout, xmlrpcinterface, profile) NOTE: File ".../poky/bitbake/lib/bb/server/process.py", line 636, in execServer server.run() NOTE: File ".../poky/bitbake/lib/bb/server/process.py", line 153, in run ret = self.main() NOTE: File ".../poky/bitbake/lib/bb/server/process.py", line 274, in main self.command_channel_reply.send(self.cooker.command.runCommand(command, self)) NOTE: File ".../poky/bitbake/lib/bb/command.py", line 90, in runCommand result = command_method(self, commandline) NOTE: File ".../poky/bitbake/lib/bb/command.py", line 292, in updateConfig command.cooker.updateConfigOpts(options, environment, cmdline) NOTE: File ".../poky/bitbake/lib/bb/cooker.py", line 536, in updateConfigOpts self.reset() NOTE: File ".../poky/bitbake/lib/bb/cooker.py", line 1783, in reset self.initConfigurationData() NOTE: File ".../poky/bitbake/lib/bb/cooker.py", line 393, in initConfigurationData self.databuilder.parseBaseConfiguration() NOTE: File ".../poky/bitbake/lib/bb/cookerdata.py", line 272, in parseBaseConfiguration bb.fetch.fetcher_init(self.data) 94657 7 yoann.congal 2023-01-10 23:39:08 +0000 sqlite (either CLI tool or sqlite3.connect()) fails for absolute path >=505 chars. persist_data.py opens "${TOPDIR}/cache/bb_persist_data.sqlite3". len("/cache/bb_persist_data.sqlite3") = 30 => this does limit $TOPDIR to 474 chars (which is above the sanity check at ~410 btw) 94673 8 yoann.congal 2023-01-11 22:46:37 +0000 I've sent a patch [1] handling the error thrown by sqlite to display clear error message instead of the stacktrace... [1]: https://lore.kernel.org/bitbake-devel/20230111223729.3472181-1-yoann.congal@smile.fr/ 94712 9 yoann.congal 2023-01-12 19:39:31 +0000 V2 here (changed python string formating method) : https://lore.kernel.org/bitbake-devel/20230112193255.38994-1-yoann.congal@smile.fr/ 94732 10 yoann.congal 2023-01-16 15:09:09 +0000 Fixed by https://git.yoctoproject.org/poky/commit/?id=e1a661e3d2d4d62aa0c47c2abb0a6824e1b17ede => Print a nice error message instead of a backtrace FYI : The new SQlite post about this limit is here https://sqlite.org/forum/forumpost/0b1b8b5116 (no answer as of now)