Bug 14060

Summary: devtool modify fails when applied on recipes using patches
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Abdellatif El Khlifi <abdellatif.elkhlifi>
Component: devtools / tool chainAssignee: Saul Wold <saul.wold>
Status: RESOLVED WORKSFORME QA Contact:
Severity: major    
Priority: Medium+ CC: meta.mr.watcher, meta.watcher, randy.macleod, ross.burton, sgw, tim.orling
Version: 3.1.2   
Target Milestone: 3.4 M4   
Hardware: x86   
OS: arm   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
u-boot and linux-stable do_patch logs (concatenated in 1 file) none

Description Abdellatif El Khlifi 2020-09-23 11:37:36 UTC 
Created attachment 4730 [details]
u-boot and linux-stable do_patch logs (concatenated in 1 file)

Hi guys,

"devtool modify" fails when applied on recipes using patches.

For example, in case of the corstone700-fvp platform this happens with u-boot and linux-stable recipes. Both recipes use patches.

The steps to replicate in addition to the error logs are listed below.

log.do_patch files contents are attached. Please take a look.

1/ Steps to replicate

repo init -u ssh://gerrit.oss.arm.com/iot-sw/iota-manifest -b master -m corstone700.xml
export MACHINE=corstone700-fvp
export DISTRO=poky-tiny
. ./setup-environment

2/ u-boot

devtool modify u-boot --keep-temp

output:

Build Configuration:
BB_VERSION           = "1.46.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-18.04"
TARGET_SYS           = "arm-poky-linux-musleabi"
MACHINE              = "corstone700-fvp"
DISTRO               = "poky-tiny"
DISTRO_VERSION       = "3.1"
TUNE_FEATURES        = "armv8a cortexa32 crc"
TARGET_FPU           = "soft"
meta-filesystems     
meta-oe              = "HEAD:920161113533074d27dc93c521815380fdf20275"
meta-poky            = "HEAD:3f6857be9be9ab5e2ede9a416c212487357eec0d"
meta-arm             
meta-arm-bsp         = "HEAD:64131db424abad32b94c2690ff73f1a4f424a83d"
meta-kernel          = "HEAD:58a589c5aad5417abd099a961e3c1a5b083cdb90"
meta-arm-image       = "HEAD:080371d1b4536a68b8966e216be5b68d4cda6dce"
meta-arm-toolchain   = "HEAD:64131db424abad32b94c2690ff73f1a4f424a83d"
meta-python          = "HEAD:920161113533074d27dc93c521815380fdf20275"
meta                 = "HEAD:1795f30d8ab73d35710ca99064c51190dc84853e"
workspace            = "<unknown>:<unknown>"

NOTE: Fetching uninative binary shim from http://downloads.yoctoproject.org/releases/uninative/2.8/x86_64-nativesdk-libc.tar.xz;sha256sum=a09922172c3a439105e0ae6b943daad2d83505b17da0aba97961ff433b8c21ab
Initialising tasks: 100% |###################################################################################################################################################################| Time: 0:00:00
Sstate summary: Wanted 20 Found 0 Missed 20 Current 0 (0% match, 0% complete)
NOTE: Executing Tasks
ERROR: [Errno 2] No such file or directory: '/home/abdelk01/Downloads/devtool_demo/downloads/0001-armv7-add-mmio-timer.patch'
ERROR: Logfile of failure stored in: /home/abdelk01/Downloads/devtool_demo/build-poky-tiny/tmp-poky_tiny/work/corstone700_fvp-poky-linux-musleabi/u-boot/1_2020.04-r0/devtooltmp-axkujcmw/temp/log.do_patch.
7625                                                                                                                                                                                                       
NOTE: Tasks Summary: Attempted 93 tasks of which 0 didn't need to be rerun and 1 failed.
INFO: Preserving temporary directory /home/abdelk01/Downloads/devtool_demo/build-poky-tiny/tmp-poky_tiny/work/corstone700_fvp-poky-linux-musleabi/u-boot/1_2020.04-r0/devtooltmp-axkujcmw
ERROR: Extracting source for u-boot failed


NB: 0001-armv7-add-mmio-timer.patch is not used by corstone700-fvp machine

3/ linux-stable

devtool modify linux-stable --keep-temp

output:

Build Configuration:
BB_VERSION           = "1.46.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "arm-poky-linux-musleabi"
MACHINE              = "corstone700-fvp"
DISTRO               = "poky-tiny"
DISTRO_VERSION       = "3.1"
TUNE_FEATURES        = "armv8a cortexa32 crc"
TARGET_FPU           = "soft"
meta-filesystems     
meta-oe              = "HEAD:920161113533074d27dc93c521815380fdf20275"
meta-poky            = "HEAD:3f6857be9be9ab5e2ede9a416c212487357eec0d"
meta-arm             
meta-arm-bsp         = "HEAD:64131db424abad32b94c2690ff73f1a4f424a83d"
meta-kernel          = "HEAD:58a589c5aad5417abd099a961e3c1a5b083cdb90"
meta-arm-image       = "HEAD:080371d1b4536a68b8966e216be5b68d4cda6dce"
meta-arm-toolchain   = "HEAD:64131db424abad32b94c2690ff73f1a4f424a83d"
meta-python          = "HEAD:920161113533074d27dc93c521815380fdf20275"
meta                 = "HEAD:1795f30d8ab73d35710ca99064c51190dc84853e"
workspace            = "<unknown>:<unknown>"

Initialising tasks: 100% |###################################################################################################################################################################| Time: 0:00:02
Sstate summary: Wanted 74 Found 0 Missed 74 Current 20 (0% match, 21% complete)
NOTE: Executing Tasks
ERROR: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:devtool_post_patch(d)
     0003:
File: '/home/abdelk01/Downloads/devtool_demo/build-poky-tiny/conf/../../layers/openembedded-core/meta/classes/devtool-source.bbclass', lineno: 224, function: devtool_post_patch
     0220:                bb.build.exec_func('do_patch', localdata)
     0221:                rm_patches()
     0222:                # Now we need to reconcile the new branch with the no-overrides one
     0223:                # (otherwise we'd likely be left with identical commits that have different hashes)
 *** 0224:                bb.process.run('git rebase devtool-no-overrides', cwd=srcsubdir)
     0225:        bb.process.run('git checkout %s' % devbranch, cwd=srcsubdir)
     0226:    bb.process.run('git tag -f devtool-patched', cwd=srcsubdir)
     0227:}
     0228:
File: '/home/abdelk01/Downloads/devtool_demo/bitbake/lib/bb/process.py', lineno: 182, function: run
     0178:        if not stderr is None:
     0179:            stderr = stderr.decode("utf-8")
     0180:
     0181:    if pipe.returncode != 0:
 *** 0182:        raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
     0183:    return stdout, stderr
Exception: bb.process.ExecutionError: Execution of 'git rebase devtool-no-overrides' failed with exit code 1:

It seems that there is already a rebase-apply directory, and
I wonder if you are in the middle of another rebase.  If that is the
case, please try
        git rebase (--continue | --abort | --skip)
If that is not the case, please
        rm -fr "/home/abdelk01/Downloads/devtool_demo/build-poky-tiny/tmp-poky_tiny/work/corstone700_fvp-poky-linux-musleabi/linux-stable/5.6.14-r0/devtooltmp-0a1c56u_/workdir/git/.git/rebase-apply"
and run me again.  I am stopping in case you still have something
valuable there.


ERROR: Logfile of failure stored in: /home/abdelk01/Downloads/devtool_demo/build-poky-tiny/tmp-poky_tiny/work/corstone700_fvp-poky-linux-musleabi/linux-stable/5.6.14-r0/devtooltmp-0a1c56u_/temp/log.do_pat
ch.10763                                                                                                                                                                                                   
NOTE: Tasks Summary: Attempted 425 tasks of which 90 didn't need to be rerun and 1 failed.
INFO: Preserving temporary directory /home/abdelk01/Downloads/devtool_demo/build-poky-tiny/tmp-poky_tiny/work/corstone700_fvp-poky-linux-musleabi/linux-stable/5.6.14-r0/devtooltmp-0a1c56u_
ERROR: Extracting source for linux-stable failed
Comment 1 Abdellatif El Khlifi 2020-09-29 10:34:34 UTC 
Please use the following steps to replicate the issue:

repo init -u ssh://git@git.linaro.org/landing-teams/working/arm/arm-reference-platforms-manifest.git -b master -m corstone700.xml
repo sync -j8
export MACHINE=corstone700-fvp
export DISTRO=poky-tiny
. ./setup-environment
Comment 2 Ross Burton 2020-10-06 04:47:36 UTC 
The problem is when you have a recipe that uses overrides to add patches, but not all of the overrides have been built previously.

The workaround would be to explicitly fetch the missing overrides, in this example the missing file is 0001-armv7-add-mmio-timer.patch which is from a5ds:

$ MACHINE=a5ds bitbake u-boot -c fetch

Repeat for each missing patch.
Comment 3 Ross Burton 2020-10-06 05:19:48 UTC 
Retracting that, it's more complicated!
Comment 4 Abdellatif El Khlifi 2020-10-22 09:11:31 UTC 
(In reply to comment #3)
> Retracting that, it's more complicated!

Is there a way to list all patches that are enabled through overrides, then filtering them based on MACHINE and MACHINEOVERRIDES ? (ie: putting them in python dictionary and filtering them)
Comment 5 Saul Wold 2021-09-14 21:20:18 UTC 
I have been looking at this, and I note that the arm repos are using older layers and I was not able to reproduce this issue with current master, what's your update plan for the meta-arm layers?

Can you test it with master?

Or is it possible to create a reproducer with just Poky?
Comment 6 Ross Burton 2021-09-16 15:44:02 UTC 
I believe a reproducer was just this with meta-arm:

MACHINE=tc0 devtool modify u-boot

That works for me now, so I suspect this might have been resolved.
Comment 7 Saul Wold 2021-09-21 20:51:44 UTC 
I am going to close this as it was meta-arm specific and seems to be resolved in master.
Comment 8 Randy MacLeod 2025-02-14 16:46:27 UTC 
bulk change to add Saul's non-WR address.
Comment 9 Randy MacLeod 2025-02-14 16:48:18 UTC 
Bulk change: Remove Saul's old WR address.