<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>12648</bug_id>
          
          <creation_ts>2018-04-04 05:14:42 +0000</creation_ts>
          <short_desc>opkg install reads all files on rootfs?</short_desc>
          <delta_ts>2018-11-14 21:57:11 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>6</classification_id>
          <classification>Yocto Project Subprojects</classification>
          <product>opkg</product>
          <component>opkg</component>
          <version>0.3.5</version>
          <rep_platform>All</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Undecided</priority>
          <bug_severity>normal</bug_severity>
          <target_milestone>0.4.0</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Robert Berger">pokylinux</reporter>
          <assigned_to name="Alejandro del Castillo">alejandro.delcastillo</assigned_to>
          <cc>michael.j.hansen</cc>
    
    <cc>stephano</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>Don&apos;t know</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>80088</commentid>
    <comment_count>0</comment_count>
    <who name="Robert Berger">pokylinux</who>
    <bug_when>2018-04-04 05:14:42 +0000</bug_when>
    <thetext>I&apos;m on yocto 2.4.2 (but this problem bothers me for quite some time)

Here&apos;s what I do:

I have a rather big image. Something like a core-image-sato-sdk with some extra tings (including sources and debug info as well) and would like to show the usage of opkg.

The rootfs is exported via nfs:

root@multi-v7-ml:~# cat /proc/cmdline 
console=ttymxc1,115200n8 ip=192.168.42.11:192.168.42.1:192.168.42.254:255.255.255.0:imx6q-phytec-mira-rdk-nand:eth0:off nfsroot=192.168.42.1:/opt/poky/mira-rootfs,v3,tcp noinitrd nohlt panic=1 earlyprintk=ttymxc1,115200n8

And the pkg feed is on the same machine exported via a web server.

root@multi-v7-ml:~# opkg --version
opkg version 0.3.5 (libsolv 0.6.28)

root@multi-v7-ml:~# opkg update
Downloading http://192.168.42.1/imx6q-phytec-mira-rdk-nand/ext-sdk-repo/2.4.2/ext-sdk-ipk/all/Packages.gz.
Updated source &apos;all&apos;.
Downloading http://192.168.42.1/imx6q-phytec-mira-rdk-nand/ext-sdk-repo/2.4.2/ext-sdk-ipk/armv7a-neon/Packages.gz.
Updated source &apos;armv7a-neon&apos;.
Downloading http://192.168.42.1/imx6q-phytec-mira-rdk-nand/ext-sdk-repo/2.4.2/ext-sdk-ipk/multi_v7_ml/Packages.gz.
Updated source &apos;multi-v7-ml&apos;.

root@multi-v7-ml:~# opkg info strace-doc
Package: strace-doc
Version: 4.18-r0.0
Status: unknown ok not-installed
Section: doc
Architecture: armv7a-neon
Maintainer: Poky &lt;poky@yoctoproject.org&gt;
MD5Sum: 820391789d9710186c3bee8549415d21
Size: 11184
Filename: strace-doc_4.18-r0.0_armv7a-neon.ipk
Source: strace_4.18.bb
Description: System call tracing tool - Documentation files
 System call tracing tool - Documentation files

root@multi-v7-ml:~# opkg list-installed | grep strace-doc
root@multi-v7-ml:~# opkg list | grep strace-doc
strace-doc - 4.18-r0.0 - System call tracing tool - Documentation files

---- so far so good ---

root@multi-v7-ml:~# time opkg install strace-doc
Installing strace-doc (4.18) on root
Downloading http://192.168.42.1/imx6q-phytec-mira-rdk-nand/ext-sdk-repo/2.4.2/ext-sdk-ipk/armv7a-neon/strace-doc_4.18-r0.0_armv7a-neon.ipk.
Configuring strace-doc.

real    3m41.485s
user    0m15.660s
sys     1m41.225s


which is quite long, I would say.

so I tried strace opkg install strace-doc and see:

...

openat(AT_FDCWD, &quot;///var/lib/opkg/info/bluez5-dbg.list&quot;, O_RDONLY|O_LARGEFILE) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=31790, ...}) = 0
read(4, &quot;/usr/lib/debug/usr/bin/bluemoon.&quot;..., 4096) = 4096
lstat64(&quot;/usr/lib/debug/usr/bin/bluemoon.debug&quot;, {st_mode=S_IFREG|0755, st_size=81848, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/tools/hciattach.c&quot;, {st_mode=S_IFREG|0644, st_size=32144, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/src/shared/ecc.h&quot;, {st_mode=S_IFREG|0644, st_size=2443, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/src/shared/gatt-helpers.c&quot;, {st_mode=S_IFREG|0644, st_size=33891, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/tools/hciattach.h&quot;, {st_mode=S_IFREG|0644, st_size=2379, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/lib/bnep.h&quot;, {st_mode=S_IFREG|0644, st_size=3938, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/profiles/input/suspend-none.c&quot;, {st_mode=S_IFREG|0644, st_size=1124, ...}) = 0
lstat64(&quot;/usr/lib/debug/usr/lib/bluez5/ptest/unit/test-ecc.debug&quot;, {st_mode=S_IFREG|0755, st_size=86116, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/src/shared/gatt-helpers.h&quot;, {st_mode=S_IFREG|0644, st_size=4436, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/unit/test-avdtp.c&quot;, {st_mode=S_IFREG|0644, st_size=41652, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/tools/parser/hidp.c&quot;, {st_mode=S_IFREG|0644, st_size=3098, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/plugins/policy.c&quot;, {st_mode=S_IFREG|0644, st_size=19928, ...}) = 0
lstat64(&quot;/usr/lib/debug/usr/libexec/bluetooth/obexd.debug&quot;, {st_mode=S_IFREG|0755, st_size=1147820, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/tools/parser/ericsson.c&quot;, {st_mode=S_IFREG|0644, st_size=1388, ...}) = 0
lstat64(&quot;/usr/src/debug/bluez5/5.46-r0/bluez-5.46/client/display.c&quot;, {st_mode=S_IFREG|0644, st_size=3640, ...}) = 0

...

opkg reads /usr/src/debug and /usr/lib/debug, meaning the source and debug packages.

And it does this each time.

It looks like the bigger the rootfs gets, the longer it takes to install a small package.

Can you please elaborate what&apos;s happening? I guess opkg tries to build some database and that&apos;s why it checks what&apos;s on the rootfs. If that&apos;s the case can I make it somehow non-volatile? Or even better pre-build it and add it to the rootfs?

root@multi-v7-ml:~# time opkg install strace-doc
No packages installed or removed.

real    3m30.172s
user    0m16.160s
sys     1m40.425s

It even takes 3m30 not to install anything;)

root@multi-v7-ml:~# opkg info strace-doc
Package: strace-doc
Version: 4.18-r0.0
Status: install ok installed
Section: doc
Architecture: armv7a-neon
Maintainer: Poky &lt;poky@yoctoproject.org&gt;
MD5Sum: 820391789d9710186c3bee8549415d21
Size: 11184
Filename: strace-doc_4.18-r0.0_armv7a-neon.ipk
Source: strace_4.18.bb
Description: System call tracing tool - Documentation files
 System call tracing tool - Documentation files
Installed-Size: 27614
Installed-Time: 1522825261</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>82200</commentid>
    <comment_count>1</comment_count>
    <who name="Alejandro del Castillo">alejandro.delcastillo</who>
    <bug_when>2018-11-13 17:28:19 +0000</bug_when>
    <thetext>Today, I took a look at this and your right, this is a regression from 0.3.4. On commit 492dd1, the file mode (returned by lstat) was added to the list of things to track on a file. However, the mode information is not persisted, so opkg calls lstat on every installed file every time it runs, which is causing the performance slowdown.

Michael: I think we talked about possibly persisting the mode in the *.list file at some point? I remember there were concerns about backwards compatibility. Since the performance hit is too big, we might need to add a check to use the new format if present, the old if not. The drawback would be that tools that read the *.list file might break if we change the format...but I think I am OK with that since the *.list files are a detail implementation of opkg, with no guarantees of it staying the same.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>82205</commentid>
    <comment_count>2</comment_count>
    <who name="Robert Berger">pokylinux</who>
    <bug_when>2018-11-13 18:54:52 +0000</bug_when>
    <thetext>@Alejandro del Castillo:

I don&apos;t quite understand what the reading of every file is needed for. Does it build some database?

Also I was wondering if this could be done when we build the image on the host and not on identical targets once on every target.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>82207</commentid>
    <comment_count>3</comment_count>
    <who name="Alejandro del Castillo">alejandro.delcastillo</who>
    <bug_when>2018-11-13 19:32:31 +0000</bug_when>
    <thetext>The following patchset can give you context: https://groups.google.com/forum/#!topic/opkg-devel/afrFJrAva-I.

The file mode for every file is a piece of metadata that is now needed, but is not being persisted. We should persist that information to avoid the expensive lstat call on every installed file.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>82225</commentid>
    <comment_count>4</comment_count>
    <who name="Alejandro del Castillo">alejandro.delcastillo</who>
    <bug_when>2018-11-14 20:24:22 +0000</bug_when>
    <thetext>fixed on commit 99c89c64665670a5547294e826bc709e218b1bd3  (Michael Hansen)</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>82226</commentid>
    <comment_count>5</comment_count>
    <who name="Alejandro del Castillo">alejandro.delcastillo</who>
    <bug_when>2018-11-14 20:24:51 +0000</bug_when>
    <thetext>fixed</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>82228</commentid>
    <comment_count>6</comment_count>
    <who name="Robert Berger">pokylinux</who>
    <bug_when>2018-11-14 21:57:11 +0000</bug_when>
    <thetext>https://git.yoctoproject.org/cgit/cgit.cgi/opkg/commit/?id=99c89c64665670a5547294e826bc709e218b1bd3</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>