I'm on yocto 2.4.2 (but this problem bothers me for quite some time) Here's what I do: I have a rather big image. Something like a core-image-sato-sdk with some extra tings (including sources and debug info as well) and would like to show the usage of opkg. The rootfs is exported via nfs: root@multi-v7-ml:~# cat /proc/cmdline console=ttymxc1,115200n8 ip=192.168.42.11:192.168.42.1:192.168.42.254:255.255.255.0:imx6q-phytec-mira-rdk-nand:eth0:off nfsroot=192.168.42.1:/opt/poky/mira-rootfs,v3,tcp noinitrd nohlt panic=1 earlyprintk=ttymxc1,115200n8 And the pkg feed is on the same machine exported via a web server. root@multi-v7-ml:~# opkg --version opkg version 0.3.5 (libsolv 0.6.28) root@multi-v7-ml:~# opkg update Downloading http://192.168.42.1/imx6q-phytec-mira-rdk-nand/ext-sdk-repo/2.4.2/ext-sdk-ipk/all/Packages.gz. Updated source 'all'. Downloading http://192.168.42.1/imx6q-phytec-mira-rdk-nand/ext-sdk-repo/2.4.2/ext-sdk-ipk/armv7a-neon/Packages.gz. Updated source 'armv7a-neon'. Downloading http://192.168.42.1/imx6q-phytec-mira-rdk-nand/ext-sdk-repo/2.4.2/ext-sdk-ipk/multi_v7_ml/Packages.gz. Updated source 'multi-v7-ml'. root@multi-v7-ml:~# opkg info strace-doc Package: strace-doc Version: 4.18-r0.0 Status: unknown ok not-installed Section: doc Architecture: armv7a-neon Maintainer: Poky <poky@yoctoproject.org> MD5Sum: 820391789d9710186c3bee8549415d21 Size: 11184 Filename: strace-doc_4.18-r0.0_armv7a-neon.ipk Source: strace_4.18.bb Description: System call tracing tool - Documentation files System call tracing tool - Documentation files root@multi-v7-ml:~# opkg list-installed | grep strace-doc root@multi-v7-ml:~# opkg list | grep strace-doc strace-doc - 4.18-r0.0 - System call tracing tool - Documentation files ---- so far so good --- root@multi-v7-ml:~# time opkg install strace-doc Installing strace-doc (4.18) on root Downloading http://192.168.42.1/imx6q-phytec-mira-rdk-nand/ext-sdk-repo/2.4.2/ext-sdk-ipk/armv7a-neon/strace-doc_4.18-r0.0_armv7a-neon.ipk. Configuring strace-doc. real 3m41.485s user 0m15.660s sys 1m41.225s which is quite long, I would say. so I tried strace opkg install strace-doc and see: ... openat(AT_FDCWD, "///var/lib/opkg/info/bluez5-dbg.list", O_RDONLY|O_LARGEFILE) = 4 fstat64(4, {st_mode=S_IFREG|0644, st_size=31790, ...}) = 0 read(4, "/usr/lib/debug/usr/bin/bluemoon."..., 4096) = 4096 lstat64("/usr/lib/debug/usr/bin/bluemoon.debug", {st_mode=S_IFREG|0755, st_size=81848, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/tools/hciattach.c", {st_mode=S_IFREG|0644, st_size=32144, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/src/shared/ecc.h", {st_mode=S_IFREG|0644, st_size=2443, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/src/shared/gatt-helpers.c", {st_mode=S_IFREG|0644, st_size=33891, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/tools/hciattach.h", {st_mode=S_IFREG|0644, st_size=2379, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/lib/bnep.h", {st_mode=S_IFREG|0644, st_size=3938, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/profiles/input/suspend-none.c", {st_mode=S_IFREG|0644, st_size=1124, ...}) = 0 lstat64("/usr/lib/debug/usr/lib/bluez5/ptest/unit/test-ecc.debug", {st_mode=S_IFREG|0755, st_size=86116, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/src/shared/gatt-helpers.h", {st_mode=S_IFREG|0644, st_size=4436, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/unit/test-avdtp.c", {st_mode=S_IFREG|0644, st_size=41652, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/tools/parser/hidp.c", {st_mode=S_IFREG|0644, st_size=3098, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/plugins/policy.c", {st_mode=S_IFREG|0644, st_size=19928, ...}) = 0 lstat64("/usr/lib/debug/usr/libexec/bluetooth/obexd.debug", {st_mode=S_IFREG|0755, st_size=1147820, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/tools/parser/ericsson.c", {st_mode=S_IFREG|0644, st_size=1388, ...}) = 0 lstat64("/usr/src/debug/bluez5/5.46-r0/bluez-5.46/client/display.c", {st_mode=S_IFREG|0644, st_size=3640, ...}) = 0 ... opkg reads /usr/src/debug and /usr/lib/debug, meaning the source and debug packages. And it does this each time. It looks like the bigger the rootfs gets, the longer it takes to install a small package. Can you please elaborate what's happening? I guess opkg tries to build some database and that's why it checks what's on the rootfs. If that's the case can I make it somehow non-volatile? Or even better pre-build it and add it to the rootfs? root@multi-v7-ml:~# time opkg install strace-doc No packages installed or removed. real 3m30.172s user 0m16.160s sys 1m40.425s It even takes 3m30 not to install anything;) root@multi-v7-ml:~# opkg info strace-doc Package: strace-doc Version: 4.18-r0.0 Status: install ok installed Section: doc Architecture: armv7a-neon Maintainer: Poky <poky@yoctoproject.org> MD5Sum: 820391789d9710186c3bee8549415d21 Size: 11184 Filename: strace-doc_4.18-r0.0_armv7a-neon.ipk Source: strace_4.18.bb Description: System call tracing tool - Documentation files System call tracing tool - Documentation files Installed-Size: 27614 Installed-Time: 1522825261
Today, I took a look at this and your right, this is a regression from 0.3.4. On commit 492dd1, the file mode (returned by lstat) was added to the list of things to track on a file. However, the mode information is not persisted, so opkg calls lstat on every installed file every time it runs, which is causing the performance slowdown. Michael: I think we talked about possibly persisting the mode in the *.list file at some point? I remember there were concerns about backwards compatibility. Since the performance hit is too big, we might need to add a check to use the new format if present, the old if not. The drawback would be that tools that read the *.list file might break if we change the format...but I think I am OK with that since the *.list files are a detail implementation of opkg, with no guarantees of it staying the same.
@Alejandro del Castillo: I don't quite understand what the reading of every file is needed for. Does it build some database? Also I was wondering if this could be done when we build the image on the host and not on identical targets once on every target.
The following patchset can give you context: https://groups.google.com/forum/#!topic/opkg-devel/afrFJrAva-I. The file mode for every file is a piece of metadata that is now needed, but is not being persisted. We should persist that information to avoid the expensive lstat call on every installed file.
fixed on commit 99c89c64665670a5547294e826bc709e218b1bd3 (Michael Hansen)
fixed
https://git.yoctoproject.org/cgit/cgit.cgi/opkg/commit/?id=99c89c64665670a5547294e826bc709e218b1bd3