15876 2025-05-22 15:46:34 +0000 diffoscope has no memory limit 2025-05-28 06:42:39 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest unspecified x86 Multiple RESOLVED FIXED Undecided normal --- 1 yoann.congal yoann.congal --- --- No (bug/feature does not impact docs) oldest_to_newest 102048 0 yoann.congal 2025-05-22 15:46:34 +0000 (Bug to note ideas discussed during the bug review call) In its current form, the reproducibility build (using diffoscope) has no limit in memory usage. When testing meta-oe reproducibility, its high memory usage can trigger OOM kills on AB. In first-to-last to try: * this option: --max-diff-block-lines-saved LINES Maximum number of lines saved per diff block. Most users should not need this, unless you run out of memory. This truncates diff(1) output before emitting it in a report, and affects all types of output, including --text and --json. (0 to disable, default: 0) * most likely need to be coupled with --max-diff-block-lines LINES Maximum number of lines output per unified-diff block, across all pages. (0 to disable, default: 1024) ... to avoid generating no empty output * Re-explore '--no-default-limits' option (see call[0]) => look at git history on why it was added. * If limiting memory from diffoscope is not possible, use an external tool: * prlimit? (https://man7.org/linux/man-pages/man1/prlimit.1.html) * (during the call, I mentioned a "timeout" tools but it was not the GNU Coreutils tool, but this one https://github.com/pshved/timeout) * cgroups? (feasibility on AB is uncertain) [0] https://web.git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/reproducible.py#n101 102051 1 yoann.congal 2025-05-22 17:03:58 +0000 Also, * When diffoscop'ing 2 1GB binaries, diffoscope tried to use >26GB memory. * RP suggested that if we have a proper reproducer, upstream diffoscope is interested. 102052 2 yoann.congal 2025-05-22 17:28:20 +0000 Note: the --no-default-limits option was there since diffoscope was added: https://git.yoctoproject.org/poky/commit/?id=d59bfa1334bd635bc1537ca20e58e624693167f1 102061 3 yoann.congal 2025-05-23 11:01:31 +0000 On the may 17th failures, the most likely suspect is nodejs-dbg which is almost 1GB (!): https://valkyrie.yocto.io/pub/repro-fail-openembedded/meta-oe/oe-reproducible-20250517-nuhpy5cy/packages/reproducibleA/tmp/deploy/deb/core2-64/ 102066 4 yoann.congal 2025-05-23 19:25:39 +0000 Patch sent: [PATCH] selftest/reproducible: Limit memory used by diffoscope https://lists.openembedded.org/g/openembedded-core/message/217218 102077 5 yoann.congal 2025-05-28 06:42:39 +0000 Fix merged with: selftest/reproducible: Limit memory used by diffoscope - openembedded-core - OpenEmbedded Core layer https://git.openembedded.org/openembedded-core/commit/?id=04cbcfc7e09d19b0ba50e7940fc82d10e222fdbe