11442 2017-05-03 09:12:31 +0000 svn AUTOREV recipe parse time explosion 2021-04-08 15:26:53 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 2.3 x86 Multiple RESOLVED OBSOLETE Medium normal 4.99 1 niko.mauno richard.purdie poky.bs.watcher poky.watcher randy.macleod ross.burton --- Regression (Used to work) No (bug/feature does not impact docs) oldest_to_newest 72990 0 niko.mauno 2017-05-03 09:12:31 +0000 While testing/deploying Yocto 2.3_M3 (Pyro) I noticed that bitbake initial parse phase duration is significantly longer than in yocto-2.2.1 (Morty). I pinpointed the problem to a company-specific recipe which has: SRC_URI = "svn://remote-subversion-server.company.com/foo;module=bar;\ protocol=https;trustcert=true" SRCREV = "${AUTOREV}" As a difference between yocto-2.2.1 and 2.3_M3, the latter seems to suffer from nearly 30 redundant hits to 'except KeyError:' in bitbake/lib/bb/fetch2/__init__.py (each of which leads to a new "svn --non-interactive --trust-server-cert log --limit 1 --no-auth-cache https://remote-subversion-server.company.com/foo/bar/" call under bitbake hood). Since each such 'svn log --limit 1' call takes (in this particular case) around 1.6 seconds to complete, this introduces an additional ~45..47 seconds initial parsing delay every time bitbake command is issued. Eg. while in yocto-2.2.1 a sample 'bitbake -c listtasks myrecipe' command takes about 7 or 8 seconds to complete, in 2.3_M3 the same command takes almost 55 seconds. If the "${AUTOREV}" is replaced with a static revision value, then in both yocto-2.2.1 and 2.3_M3 the sample command takes in both cases about 6.5 seconds -- thus to me this would strongly seem like a regression with recipes making use of svn/AUTOREV combination. 73189 1 niko.mauno 2017-05-11 14:40:57 +0000 As a related observation in 2.3_M3, svn+AUTOREV recipes prevent bitbake parsing without 'svn' in HOSTTOOLS. Failure looks like: bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Fetch command ... failed with exit code 127, output: /usr/bin/env: svn: No such file or directory 83460 2 ross.burton 2019-04-05 11:22:47 +0000 Another report of this on IRC so people are still using subversion. The problem is that we don't use host svn but build our own. AUTOREV has to be expanded at parse time though which is before we've built svn. Thus, the work around is to install subversion on the host and add svn to HOSTTOOLS so that binary is available at parse time. 89907 3 randy.macleod 2021-04-08 15:26:53 +0000 Fewer and fewer people are using subversion. The subversion + autorev time isn't something that we can do anything about and it isnt' an advised workflow. There is a cache policy in bitbake that would prevent hitting the network as much BB_SRCREV_POLICY is the variable. If you're going to use subversion, then as people have pointed out you are well advised to install it on the host. If there are still subversion users and this is a problem, please send a patch to deal with the issue.