| Summary: | diffoscope has no memory limit | ||
|---|---|---|---|
| Product: | [QA/Testing] Functional (self) Testing | Reporter: | Yoann Congal <yoann.congal> |
| Component: | oe-selftest | Assignee: | Yoann Congal <yoann.congal> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Undecided | ||
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
Also, * When diffoscop'ing 2 1GB binaries, diffoscope tried to use >26GB memory. * RP suggested that if we have a proper reproducer, upstream diffoscope is interested. Note: the --no-default-limits option was there since diffoscope was added: https://git.yoctoproject.org/poky/commit/?id=d59bfa1334bd635bc1537ca20e58e624693167f1 On the may 17th failures, the most likely suspect is nodejs-dbg which is almost 1GB (!): https://valkyrie.yocto.io/pub/repro-fail-openembedded/meta-oe/oe-reproducible-20250517-nuhpy5cy/packages/reproducibleA/tmp/deploy/deb/core2-64/ Patch sent: [PATCH] selftest/reproducible: Limit memory used by diffoscope https://lists.openembedded.org/g/openembedded-core/message/217218 Fix merged with: selftest/reproducible: Limit memory used by diffoscope - openembedded-core - OpenEmbedded Core layer https://git.openembedded.org/openembedded-core/commit/?id=04cbcfc7e09d19b0ba50e7940fc82d10e222fdbe |
(Bug to note ideas discussed during the bug review call) In its current form, the reproducibility build (using diffoscope) has no limit in memory usage. When testing meta-oe reproducibility, its high memory usage can trigger OOM kills on AB. In first-to-last to try: * this option: --max-diff-block-lines-saved LINES Maximum number of lines saved per diff block. Most users should not need this, unless you run out of memory. This truncates diff(1) output before emitting it in a report, and affects all types of output, including --text and --json. (0 to disable, default: 0) * most likely need to be coupled with --max-diff-block-lines LINES Maximum number of lines output per unified-diff block, across all pages. (0 to disable, default: 1024) ... to avoid generating no empty output * Re-explore '--no-default-limits' option (see call[0]) => look at git history on why it was added. * If limiting memory from diffoscope is not possible, use an external tool: * prlimit? (https://man7.org/linux/man-pages/man1/prlimit.1.html) * (during the call, I mentioned a "timeout" tools but it was not the GNU Coreutils tool, but this one https://github.com/pshved/timeout) * cgroups? (feasibility on AB is uncertain) [0] https://web.git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/reproducible.py#n101