Bug 13575 - Segmentation fault if processing a huge package index
Summary: Segmentation fault if processing a huge package index
Status: RESOLVED FIXED
Alias: None
Product: opkg
Classification: Yocto Project Subprojects
Component: opkg (show other bugs)
Version: 0.4.1
Hardware: All arm
: Undecided normal
Target Milestone: 0.4.4
Assignee: Alejandro del Castillo
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-10-04 20:35 UTC by Melanie
Modified: 2020-10-30 10:03 UTC (History)
0 users

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Stack Trace (19.43 KB, text/plain)
2020-03-08 19:01 UTC, Melanie
no flags Details
Stack Trace 1 (3.65 MB, text/x-log)
2020-07-25 16:19 UTC, Melanie
no flags Details
Stack Trace 2 (3.65 MB, text/x-log)
2020-07-25 16:23 UTC, Melanie
no flags Details
Stack Trace 3 (4.66 MB, text/x-log)
2020-07-25 16:24 UTC, Melanie
no flags Details
Stack Trace 4 (4.66 MB, text/x-log)
2020-07-25 16:25 UTC, Melanie
no flags Details
Stack Trace 5 (3.65 MB, text/x-log)
2020-07-25 16:27 UTC, Melanie
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Melanie 2019-10-04 20:35:56 UTC
If the opkg configuration includes a repository with a huge package index, the command 'opkg clean' terminates with a segmentation fault and exit code 139.

The command 'opkg clean -V4' prints the following line for each package in the index:
pkg_vec_insert_merge: Adding new pkg=<name> <version> <arch>

Is this amount of processing really necessary when cleaning the opkg cache directory?

Notes:
The command 'opkg info <package-name>' also terminates with segmentation fault.
The command 'opkg install <package-name>' is not affected from this problem and runs without any errors.

My system configuration:

Machine model: Raspberry Pi Model B Plus Rev 1.2
Linux version 5.2.11
opkg version 0.4.1

My opkg configuration file:

arch all 1
arch armhf 8
option no_install_recommends true
option cache_dir /tmp/opkg-cache
option lists_dir /tmp/opkg-lists
dist stable http://archive.raspbian.org/raspbian main
Comment 1 Alejandro del Castillo 2019-10-11 19:28:56 UTC
I just posted a patch [1] to speed up clean. As you mentioned, it doesn't need to load the feeds or the status of the installed packages.

Regarding the segfault, sounds to me like you are running your system out of memory. Install operations don't load all the metadata (there is a mask that makes sure only Description and Source are loaded), while info operations load ALL metadata. Could you confirm that you system is running out of memory?

There are optimizations that could be made to opkg info, to avoid loading all metadata...but before looking into that I would like to see if this is indeed a low memory condition.

[1] https://groups.google.com/forum/#!topic/opkg-devel/gKH7LiQWYSU
Comment 2 Melanie 2019-10-15 13:30:36 UTC
I was trying to reproduce the segmentation fault (on the same device and configuration) but now 'opkg clean' and 'opkg info <package-name>' always terminates with exit code 134 and the following error message:

realloc(): invalid next size
Aborted

The amount of free system memory is constantly decreasing while the commands are running (in my case from 200 MB to about 140 MB), so it does not seem to be a low memory condition ...
Comment 3 Alejandro del Castillo 2019-10-25 15:42:49 UTC
interesting, the error points to a heap corruption issue. Support for debian style repos is less than ideal, so its likely a problem with that code. I'll try to reproduce with the configuration you provided.
Comment 4 Alejandro del Castillo 2019-10-28 21:33:52 UTC
Wasn't able to reproduce on my system. In fact, the update operation should throw you an error since you are configuring an "all" arch, but there is no all repo on the raspbian online feed:

[adelcast@delcastillo3 libopkg]$ opkg -o /tmp/opkg update
Downloading http://archive.raspbian.org/raspbian/dists/stable/Release.
Downloaded release files for dist stable.
Downloading http://archive.raspbian.org/raspbian/dists/stable/main/binary-all/Packages.
Downloading http://archive.raspbian.org/raspbian/dists/stable/main/binary-armhf/Packages.
Downloading http://archive.raspbian.org/raspbian/dists/stable/main/binary-armhf/Packages.
Updated source 'stable-main-armhf'.
Collected errors:
 * opkg_validate_cached_file: Failed to download http://archive.raspbian.org/raspbian/dists/stable/main/binary-all/Packages headers: HTTP response code said error.


could you try to remove all your opkg temp files? (option c ache_dir /tmp/opkg-cache), as well as all your configured lists? do you still see the error? what if you comment out the "arch all" line? Also, I assume you are using upstream opkg from https://git.yoctoproject.org/cgit/cgit.cgi/opkg/?
Comment 5 Melanie 2019-11-02 04:39:13 UTC
I am using opkg from [1].

Although 'opkg update' throws an error for the all-arch, the command completes sucessfully (exit code 0).

The 'arch all 1' statement in the configuration file is required to be able to install the architecture-independant packages.
Nevertheless I tested it without 'arch all 1' but 'opkg info <package-name>' still fails (this time with the segmentation fault).

Further testing:

Config:
arch armhf 8
dist stable http://archive.raspbian.org/raspbian main

corrupted size vs. prev_size
Aborted






[1] http://downloads.yoctoproject.org/releases/opkg
Comment 6 Alejandro del Castillo 2019-11-20 23:22:00 UTC
I don't have a whole lot to go on since I can't reproduce the problem. Any chance you could provide me with a stack trace?
Comment 7 Melanie 2020-03-08 19:01:30 UTC
Created attachment 4648 [details]
Stack Trace

Sorry for my late response.
Stack Trace attached.
Please let me know if additional debug data is required ...
Comment 8 Alejandro del Castillo 2020-07-17 14:40:26 UTC
apologies for taking a while to get back to you. Looking at your callstack it seems like realloc is failing. Realloc is called during package index building, which point towards the index building operation potentially running the system out of memory. 

Could you run the opkg info operation with -V4 and attach the logs? I am hoping to get more information that way...
Comment 9 Melanie 2020-07-25 16:19:50 UTC
Created attachment 4705 [details]
Stack Trace 1
Comment 10 Melanie 2020-07-25 16:23:01 UTC
Created attachment 4706 [details]
Stack Trace 2
Comment 11 Melanie 2020-07-25 16:24:24 UTC
Created attachment 4707 [details]
Stack Trace 3
Comment 12 Melanie 2020-07-25 16:25:43 UTC
Created attachment 4708 [details]
Stack Trace 4
Comment 13 Melanie 2020-07-25 16:27:57 UTC
Created attachment 4709 [details]
Stack Trace 5
Comment 14 Melanie 2020-07-25 16:52:12 UTC
Not sure if it helps but I noticed that the kind of failure (segmentation fault vs realloc failure) depends on the arch setting in the opkg configuration.

Each of the attached files includes a opkg configuration followed by the stack trace output of the opkg info operation.
Comment 15 Alejandro del Castillo 2020-07-29 16:49:44 UTC
I just submitted a patch that address a memory corruption that I observed using the feeds on this ticket. Could you try it out to see if the error you are seeing goes away?

https://groups.google.com/forum/#!topic/opkg-devel/sW1lLcjo7Dk
Comment 16 Alejandro del Castillo 2020-10-30 10:03:43 UTC
I believe commit fd33426c2805f119201a8851171665c0b32e25df fixed this issue. Please re-open if that was not the case.