Bug 15876 - diffoscope has no memory limit
Summary: diffoscope has no memory limit
Status: RESOLVED FIXED
Alias: None
Product: Functional (self) Testing
Classification: QA/Testing
Component: oe-selftest (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Undecided normal
Target Milestone: ---
Assignee: Yoann Congal
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2025-05-22 15:46 UTC by Yoann Congal
Modified: 2025-05-28 06:42 UTC (History)
0 users

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Yoann Congal 2025-05-22 15:46:34 UTC
(Bug to note ideas discussed during the bug review call)

In its current form, the reproducibility build (using diffoscope) has no limit in memory usage. When testing meta-oe reproducibility, its high memory usage can trigger OOM kills on AB.

In first-to-last to try:
* this option:
--max-diff-block-lines-saved LINES
    Maximum number of lines saved per diff block. Most users should not need this, unless
    you run out of memory. This truncates diff(1) output before emitting it in a report, 
    and affects all types of output, including --text and --json. (0 to disable, default: 0)
* most likely need to be coupled with 
--max-diff-block-lines LINES
    Maximum number of lines output per unified-diff block, across all pages. 
    (0 to disable, default: 1024)
  ... to avoid generating no empty output
* Re-explore '--no-default-limits' option (see call[0]) => look at git history on why it was added.
* If limiting memory from diffoscope is not possible, use an external tool:
  * prlimit? (https://man7.org/linux/man-pages/man1/prlimit.1.html)
  * (during the call, I mentioned a "timeout" tools but it was not the GNU Coreutils tool, but this one https://github.com/pshved/timeout)
  * cgroups? (feasibility on AB is uncertain)


[0] https://web.git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/reproducible.py#n101
Comment 1 Yoann Congal 2025-05-22 17:03:58 UTC
Also,
* When diffoscop'ing 2 1GB binaries, diffoscope tried to use >26GB memory.
* RP suggested that if we have a proper reproducer, upstream diffoscope is interested.
Comment 2 Yoann Congal 2025-05-22 17:28:20 UTC
Note: the --no-default-limits option was there since diffoscope was added: 
https://git.yoctoproject.org/poky/commit/?id=d59bfa1334bd635bc1537ca20e58e624693167f1
Comment 3 Yoann Congal 2025-05-23 11:01:31 UTC
On the may 17th failures, the most likely suspect is nodejs-dbg which is almost 1GB (!):
https://valkyrie.yocto.io/pub/repro-fail-openembedded/meta-oe/oe-reproducible-20250517-nuhpy5cy/packages/reproducibleA/tmp/deploy/deb/core2-64/
Comment 4 Yoann Congal 2025-05-23 19:25:39 UTC
Patch sent:
[PATCH] selftest/reproducible: Limit memory used by diffoscope
https://lists.openembedded.org/g/openembedded-core/message/217218
Comment 5 Yoann Congal 2025-05-28 06:42:39 UTC
Fix merged with: selftest/reproducible: Limit memory used by diffoscope - openembedded-core - OpenEmbedded Core layer
https://git.openembedded.org/openembedded-core/commit/?id=04cbcfc7e09d19b0ba50e7940fc82d10e222fdbe