Bug 10880 - perf recipe contaminates linux shared workdir in do_configure_prepend()
Summary: perf recipe contaminates linux shared workdir in do_configure_prepend()
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: kernel (show other bugs)
Version: 2.4
Hardware: All Multiple
: Medium+ normal
Target Milestone: 4.99
Assignee: Hongxu Jia
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-01-05 12:42 UTC by Martin Hundeboll
Modified: 2018-06-01 02:11 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Hundeboll 2017-01-05 12:42:05 UTC
When baking perf the shared workdir is modified:

% git status
 HEAD detached at 43daed5beff6
 Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   tools/build/Makefile.build
        modified:   tools/build/Makefile.feature
        modified:   tools/lib/api/Makefile
        modified:   tools/perf/Makefile.perf
        modified:   tools/perf/arch/arm/tests/dwarf-unwind.c
        modified:   tools/perf/arch/arm/util/unwind-libunwind.c
        modified:   tools/perf/config/Makefile

no changes added to commit (use "git add" and/or "git commit -a")

Building the kernel (or its modules) afterwards (without running do_unpack) changes the kernelversion to *-dirty, which might lead to misplaced modules etc.
Comment 1 Bruce Ashfield 2017-01-05 15:33:10 UTC
There's no easy way to fix this. How the perf recipe modifies the kernel source is out of necessity, since perf is a separate recipe and not attached to a definitive non-kernel source tree.

That means that we can't just fix perf issues with patches (since they won't apply to everyone's kernel source tree), and we can't fix it directly in linux-yocto (my preference), since not everyone uses linux-yocto.

Also, we can't just commit the changes, since not everyone uses git based kernel source trees, hence having the perf recipe itself do git add/commits won't universally work (not to mention we have enough issues even with the kernel recipes ensuring that git is configured properly for commits). We could always do a commit for linux-yocto, but again, not everyone uses linux-yocto.

We could do kernel-recipe specific providers of perf, and those kernel trees could carry perf patches. Possible, but duplicated effort to maintain fixes.

We could split out the perf code into a separate tree, patch and build from it. Also possible, but then we'd deviate from upstream development and have kernel version skew.

We could test for a git based ${S} and conditionally commit changes, but that adds complexity to the recipe .. but is probably the best option.
Comment 2 Bruce Ashfield 2017-01-05 16:21:49 UTC
I see this got a milestone .. to be clear, I accepted this, but am not committing to a fix. 

Changing the milestone.
Comment 3 Martin Hundeboll 2017-01-05 18:08:04 UTC
Doing commits from the perf recipe doesn't fix the problem, as following kernel builds will have a new commit-id.

I lean towards building perf as part of the kernel-class, possibly conditional on a class-variable.
Comment 4 Bruce Ashfield 2017-01-05 18:26:52 UTC
perf was explicitly split out from the kernel recipes. I'm not going to be the one that attempts to put it back in. 

It has a separate set of depends/functionality/complications that the already complex kernel build process needs no part of.
Comment 5 Bruce Ashfield 2017-01-05 18:28:48 UTC
Alternatively, we can just have the kernel recipe clean up, and unpack the sources again on subsequent builds if the tree has been modified. (In reply to comment #4)
> perf was explicitly split out from the kernel recipes. I'm not going to be
> the one that attempts to put it back in. 
> 
> It has a separate set of depends/functionality/complications that the
> already complex kernel build process needs no part of.

Not to mention, you are still in the situation where perf fixes cannot be shared, which isn't going to fly.
Comment 6 Martin Hundeboll 2017-01-05 18:31:00 UTC
Would that race with other recipes using the shared kernel source? E.g. do_compile_kernelmodules...
Comment 7 Martin Hundeboll 2017-01-05 21:08:19 UTC
For your information I fixed this in my local layer (non-x86 multilib) with a bbappend:

% cat perf.bbappend 
RDEPENDS_perf_remove = "perl"

# remove tainting of shared workdir, see
# https://bugzilla.yoctoproject.org/show_bug.cgi?id=10880
do_configure_prepend() {
    # Fix for rebuilding
    rm -rf ${B}/
    mkdir -p ${B}/

    # return here to avoid running do_configure_append() from perf.bb
    return

}

# Fix the build failures caused by the early return in
# do_configure_prepend() by passing the correct options to make:
#  - Unlike other kernel builds, perf uses "OUTPUT" instead of "O"
#    to specify the build directory
#  - The perf build ignores the flags given in ${CC}, so pass them in
#    via EXTRA_CFLAGS instead
EXTRA_OEMAKE += " \
    OUTPUT=${B}/ \
    EXTRA_CFLAGS="${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS} -ldw" \
"
Comment 8 Martin Hundeboll 2017-01-05 21:13:01 UTC
Alternatively, the perf recipe could patch out the -dirty feature from ./scripts/setlocalversion
Comment 9 Bruce Ashfield 2017-04-12 17:31:49 UTC
I had asked Richard about this one via email, but there are other higher priority items for 2.3 to complete first.

I'll take my half done implementation and complete it in early 2.4
Comment 10 Stephen K Jolley 2017-08-03 15:19:56 UTC
See Bruce's question
Comment 11 Hongxu Jia 2018-06-01 02:11:54 UTC
Merged into oe-core

commit 9b38c824961fc9dce51bda95c25dac91a69fc64f
Author: Hongxu Jia <hongxu.jia@windriver.com>
Date:   Tue Apr 24 11:33:47 2018 +0800

    perf: make a copy of kernel source to perf workdir
    
    Since perf contaminates linux shared workdir, it probably caused
    kernel-devsrc compile failure at world build.
    ...
    |0 blocks
    |cpio: ./tools/perf/arch/arm/util/sedr7ORqk: Cannot stat:
    No such file or directory
    |0 blocks
    ...
    cpio tried to find a file at ${S}/tools/perf and failed
    if the input list is not valid.
    
    Make a copy of kernel shared source directory into a perf workdir
    could fix the issue.
    
    Drop `Fix for rebuilding' which is obsolete
    
    [YOCTO #10880]
    
    Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
    Signed-off-by: Ross Burton <ross.burton@intel.com>