Bug 15536

Summary: When saving ptest artefacts for failed tests, recursive directories are being copied
Product: [QA/Testing] Package Testing (ptest) Reporter: Richard Purdie <richard.purdie>
Component: ptestAssignee: Alexis Lothoré <alexis.lothore>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: High CC: alexandre.belloni, alexis.lothore, randy.macleod
Version: unspecified   
Target Milestone: 5.1 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2024-06-30 22:22:07 UTC 
We've been seeing space issues on the autobuilder, sometimes with a lack of inodes.

Upon investigation, we found:

~/yocto-worker/qemux86-64-ptest/build/build/tmp/log/oeqa/artifacts

contained huge numbers of inodes (approx. 105063410 from du --inodes -s)

This was from a util-linux ptest failure which resulted in recursive directories of the form:

build/tmp/log/oeqa/artifacts/usr/lib/util-linux/ptest/tests/output/lsblk/dumps/simple-lvm/sys/block/loop0/holders/dm-0/slaves/loop0/holders/dm-0/slaves/loop0/holders/dm-0/slaves/loop0/holders/dm-0/slaves/loop0/holders

We need to make the artefact saving preserve symlinks and/or stop at a sensible depth.
Comment 1 Alexis Lothoré 2024-07-02 14:04:28 UTC 
I see two ways of fixing this without spawning an over-complicated solution evaluating each file to check if any is a symlink:
- create an archive with all the files we want to retrieve, and then scp/uncompress this archive
- switch to rsync

I kind of remember that there was constraints which prevented rsync usage for this use case, but I don't remember the details. I will check again and come up with a fix for this
Comment 2 Richard Purdie 2024-07-03 17:19:02 UTC 
(In reply to Alexis Lothoré from comment #1)
> I see two ways of fixing this without spawning an over-complicated solution
> evaluating each file to check if any is a symlink:
> - create an archive with all the files we want to retrieve, and then
> scp/uncompress this archive

I'd worry that we don't have the space on target to be able to do this in many cases unfortunately.

> - switch to rsync
> 
> I kind of remember that there was constraints which prevented rsync usage
> for this use case, but I don't remember the details. I will check again and
> come up with a fix for this

The issue with this one would be the requirement to have rsync in the target image which we can't always guarantee either.

There may be a third option which is to pipe tar over ssh since tar can handle symlinks and loops.

The rsync solution is probably most viable/easiest?