Bug 13638

Summary: opkg double free when reproducibile builds enabled
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: devtools / tool chainAssignee: Alejandro del Castillo <alejandro.delcastillo>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: High CC: alejandro.delcastillo, meta.mr.watcher, meta.watcher, randy.macleod
Version: 3.0   
Target Milestone: 3.1 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Package which reproduces the build failure none

Description Richard Purdie 2019-11-18 12:35:00 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/1233

When INHERIT += "reproducible_build" is added to builds, opkg rootfs construction fails

/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-test-builder-io-scalar_5.30.0-r0_core2-32.ipk.
Installing perl-module-pod-simple-progress (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-pod-simple-progress_5.30.0-r0_core2-32.ipk.
Installing perl-module-unicode-collate-locale-et (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-unicode-collate-locale-et_5.30.0-r0_core2-32.ipk.
Installing perl-module-unicode-collate-locale-es (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-unicode-collate-locale-es_5.30.0-r0_core2-32.ipk.
Installing perl-module-unicode-collate-locale-eo (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-unicode-collate-locale-eo_5.30.0-r0_core2-32.ipk.
Installing perl-module-b-op-private (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-b-op-private_5.30.0-r0_core2-32.ipk.
Installing perl-module-tie-hash (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-tie-hash_5.30.0-r0_core2-32.ipk.
Installing perl-module-unicode-collate-locale-si-dict (5.30.0) on root
DownloadRunning useradd commands...
NOTE: rpcbind: Performing useradd with [--root /home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/rootfs --system --no-create-home --home-dir /                        --shell /bin/false --user-group rpc]
Running groupadd commands...
NOTE: cronie: Performing groupadd with [--root /home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/rootfs --system crontab]
Running groupadd commands...
NOTE: lttng-tools: Performing groupadd with [--root /home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/rootfs tracing]
ing file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-unicode-collate-locale-si-dict_5.30.0-r0_core2-32.ipk.
Installing perl-module-unicode-collate-locale-ee (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-unicode-collate-locale-ee_5.30.0-r0_core2-32.ipk.
Installing perl-module-test2-formatter-tap (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-test2-formatter-tap_5.30.0-r0_core2-32.ipk.
Installing perl-module-dynaloader (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-dynaloader_5.30.0-r0_core2-32.ipk.
Installing libxml-parser-perl (2.44) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/libxml-parser-perl_2.44-r0_core2-32.ipk.
Installing perl-module-term-cap (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-term-cap_5.30.0-r0_core2-32.ipk.
Installing perl-module-exporter (5.30.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/perl-module-exporter_5.30.0-r0_core2-32.ipk.
Installing valgrind (3.15.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/valgrind_3.15.0-r0_core2-32.ipk.
Installing glibc-mtrace (2.30) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/glibc-mtrace_2.30-r0_core2-32.ipk.
Installing rpcbind (1.2.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/rpcbind_1.2.5-r0_core2-32.ipk.
Installing cronie (1.5.4) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/cronie_1.5.4-r0_core2-32.ipk.
Installing libdbus-1-3 (1.12.16) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/libdbus-1-3_1.12.16-r0_core2-32.ipk.
Installing lttng-tools (2.10.7) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/lttng-tools_2.10.7-r0_core2-32.ipk.
Installing alsa-utils-alsamixer (1.1.9) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/alsa-utils-alsamixer_1.1.9-r0_core2-32.ipk.
Installing btrfs-tools (5.3) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/btrfs-tools_5.3-r0_core2-32.ipk.
Installing python3-stringold (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-stringold_3.7.5-r0_core2-32.ipk.
Installing python3-resource (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-resource_3.7.5-r0_core2-32.ipk.
Installing python3-mime (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-mime_3.7.5-r0_core2-32.ipk.
Installing python3-profile (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-profile_3.7.5-r0_core2-32.ipk.
Installing python3-ctypes (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-ctypes_3.7.5-r0_core2-32.ipk.
Installing python3-pkgutil (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-pkgutil_3.7.5-r0_core2-32.ipk.
Installing python3-mmap (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-mmap_3.7.5-r0_core2-32.ipk.
Installing python3-threading (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-threading_3.7.5-r0_core2-32.ipk.
Installing python3-audio (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-audio_3.7.5-r0_core2-32.ipk.
Installing python3-datetime (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-datetime_3.7.5-r0_core2-32.ipk.
Installing python3-compile (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-compile_3.7.5-r0_core2-32.ipk.
Installing python3-image (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-image_3.7.5-r0_core2-32.ipk.
Installing python3-pprint (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-pprint_3.7.5-r0_core2-32.ipk.
Installing python3-fcntl (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-fcntl_3.7.5-r0_core2-32.ipk.
Installing python3-curses (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-curses_3.7.5-r0_core2-32.ipk.
Installing python3-syslog (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-syslog_3.7.5-r0_core2-32.ipk.
Installing python3-codecs (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-codecs_3.7.5-r0_core2-32.ipk.
Installing python3-distutils (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-distutils_3.7.5-r0_core2-32.ipk.
Installing python3-json (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-json_3.7.5-r0_core2-32.ipk.
Installing python3-pickle (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-pickle_3.7.5-r0_core2-3Running useradd commands...
NOTE: openssh: Performing useradd with [--root /home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/rootfs --system --no-create-home --home-dir /var/run/sshd --shell /bin/false --user-group sshd]
2.ipk.
Installing python3-xml (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-xml_3.7.5-r0_core2-32.ipk.
Installing python3-numbers (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-numbers_3.7.5-r0_core2-32.ipk.
Installing python3-db (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-db_3.7.5-r0_core2-32.ipk.
Installing python3-idle (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-idle_3.7.5-r0_core2-32.ipk.
Installing python3-difflib (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-difflib_3.7.5-r0_core2-32.ipk.
Installing python3-typing (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-typing_3.7.5-r0_core2-32.ipk.
Installing python3-html (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-html_3.7.5-r0_core2-32.ipk.
Installing python3-2to3 (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-2to3_3.7.5-r0_core2-32.ipk.
Installing python3-tkinter (3.7.5) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/python3-tkinter_3.7.5-r0_core2-32.ipk.
Installing openssh-sshd (8.0p1) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/openssh-sshd_8.0p1-r0_core2-32.ipk.
Installing cpp (9.2.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/cpp_9.2.0-r0_core2-32.ipk.
Installing e2fsprogs (1.45.4) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/e2fsprogs_1.45.4-r0_core2-32.ipk.
Installing util-linux (2.34) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/util-linux_2.34-r0_core2-32.ipk.
Installing fontconfig-utils (2.13.1) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/fontconfig-utils_2.13.1-r0_core2-32.ipk.
Installing expect (5.45.4) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/expect_5.45.4-r0_core2-32.ipk.
Installing packagegroup-base-pci (1.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/qemux86/packagegroup-base-pci_1.0-r83_qemux86.ipk.
Installing udev-extraconf (1.1) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/core2-32/udev-extraconf_1.1-r0_core2-32.ipk.
Installing packagegroup-core-boot (1.0) on root
Downloading file:/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/oe-rootfs-repo/qemux86/packagegroup-free(): double free detected in tcache 2

ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/temp/log.do_rootfs.6210
NOTE: recipe core-image-sato-sdk-1.0-r0: task do_rootfs: Failed
ERROR: Task (/home/pokybuild/yocto-worker/pkgman-non-rpm/build/meta/recipes-sato/images/core-image-sato-sdk.bb:do_rootfs) failed with exit code '1'
Comment 1 Richard Purdie 2019-11-18 23:47:00 UTC
I had some fun with this:

a) you have to realise that opkg doesn't print errors if it aborts, it has them stored in a buffer which it never prints.
b) the double free comes from opkg_archive:open_inner's err_cleanup call
c) when you account for a), its printing open_inner: Failed to open inner archive: Unrecognized archive format
d) the double free is because inner_close was already called on the buffer by libarchive
e) the underlying problem is the ipk in question has a 0 size data.tar, package attached.
Comment 2 Richard Purdie 2019-11-18 23:47:26 UTC
Created attachment 4588 [details]
Package which reproduces the build failure
Comment 3 Richard Purdie 2019-11-19 00:07:51 UTC
FWIW a fix for the double free is probably:

diff --git a/libopkg/opkg_archive.c b/libopkg/opkg_archive.c
index 1e8c9d1..3c9c89c 100644
--- a/libopkg/opkg_archive.c
+++ b/libopkg/opkg_archive.c
@@ -620,7 +621,7 @@ static struct archive *open_inner(struct archive *outer)
     if (r != ARCHIVE_OK) {
         opkg_msg(ERROR, "Failed to open inner archive: %s\n",
                  archive_error_string(inner));
-        goto err_cleanup;
+        return NULL;
     }
 
     return inner;
Comment 4 Alejandro del Castillo 2019-11-21 16:57:29 UTC
Fixed upstream (opkg). There were 2 issues:

- Opkg wasn't handling empty payloads
- Libarchive errors caused a double free

Commits

f22b35 open_inner: add support for empty payloads
23ed947 opkg_archive.c: avoid double free on uncompress error