Bug 9120 - The patch file is not applied to linux-yocto by yocto-kernel add patch command
Summary: The patch file is not applied to linux-yocto by yocto-kernel add patch command
Status: VERIFIED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: kernel (show other bugs)
Version: 2.0.1
Hardware: x86 Multiple
: Medium normal
Target Milestone: 2.2 M1
Assignee: Leonardo Sandoval Gonzalez
QA Contact: Yi Zhao
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2016-02-16 08:56 UTC by Yi Zhao
Modified: 2016-11-16 06:29 UTC (History)
8 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: Regression (Used to work)
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
log.do_patch (4.45 KB, application/octet-stream)
2016-02-16 08:56 UTC, Yi Zhao
no flags Details
log.do_patch.good (4.97 KB, application/octet-stream)
2016-02-16 09:03 UTC, Yi Zhao
no flags Details
yocto-testmod.patch (2.29 KB, patch)
2016-02-16 09:04 UTC, Yi Zhao
no flags Details | Diff
layer created with yocto-bsp command (5.05 KB, application/gzip)
2016-04-21 15:54 UTC, Leonardo Sandoval Gonzalez
no flags Details
Patch sent to the ML (1.04 KB, patch)
2016-04-25 19:05 UTC, Leonardo Sandoval Gonzalez
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Yi Zhao 2016-02-16 08:56:30 UTC
Created attachment 2986 [details]
log.do_patch

Git rev: jethro/5b12268f6e17574999f91628a60e21711cf62ee4

Maybe this issue is introduced by fixes for bug 8486 .

This issue *ONLY* happens when re-use an existing kernel branch. Please see my comment in the following steps:

Steps:
1. Follow the https://wiki.yoctoproject.org/wiki/Transcript:_Using_the_Yocto_BSP_tools_to_create_a_qemu_BSP to create a myqemuarm layer:

##############################
$ yocto-bsp create myqemuarm qemu
Checking basic git connectivity...
+Done.

Which qemu architecture would you like to use? [default: i386]
	1) i386    (32-bit)
	2) x86_64  (64-bit)
	3) ARM     (32-bit)
	4) PowerPC (32-bit)
	5) MIPS    (32-bit)
	6) MIPS64  (64-bit)
3
Would you like to use the default (4.1) kernel? (y/n) [default: y]
 

Do you need a new machine branch for this BSP (the alternative is to re-use an existing branch)? [y/n] [default: y] n     <--------- DO NOT use default setting, choose *NO*


Getting branches from remote repo git://git.yoctoproject.org/linux-yocto-4.1.git...
Please choose an existing machine branch to use for this BSP: [default: standard/arm-versatile-926ejs]
	1) standard/arm-versatile-926ejs
	2) standard/base
	3) standard/beagleboard
	4) standard/beaglebone
	5) standard/edgerouter
	6) standard/fsl-mpc8315e-rdb
	7) standard/mti-malta32
	8) standard/mti-malta64
	9) standard/qemuarm64
	10) standard/qemuppc
1
Would you like SMP support? (y/n) [default: y] 
Does your BSP have a touchscreen? (y/n) [default: n] 
Does your BSP have a keyboard? (y/n) [default: y] 

New qemu BSP created in meta-myqemuarm

################################

2. Follow the https://wiki.yoctoproject.org/wiki/Transcript:_Using_the_Yocto_BSP_tools_to_manage_kernel_patches_and_config_items to add a patch (see the attachment):

#######################
$ yocto-kernel patch add myqemuarm ./yocto-testmod.patch
Added patches:
	yocto-testmod.patch
$ yocto-kernel config add myqemuarm CONFIG_MISC_DEVICES=y
Added item:
	CONFIG_MISC_DEVICES=y
$ yocto-kernel config add myqemuarm CONFIG_YOCTO_TESTMOD=y
Added item:
	CONFIG_YOCTO_TESTMOD=y

#######################

3. Run bitbake linux-yocto

It could be built without errors. but the yocto-testmod isn't patched. Please see the attached log.
Comment 1 Yi Zhao 2016-02-16 09:02:04 UTC
If I choose *YES* to use a new machine branch for the BSP, everything works well.

Here is a piece from log.do_patch
#################
mark --> myqemuarm-user-patches.scc
         patch yocto-testmod.patch (.kernel-meta/patches/standard/arm-versatile-926ejs/myqemuarm/series)
   mark <-- myqemuarm-user-patches.scc
   mark <-- myqemuarm.scc
   mark <-- myqemuarm-standard.scc
(1/1) yocto-testmod.patch
[INFO]: check of .kernel-meta/patches/standard/arm-versatile-926ejs/myqemuarm/links/files/yocto-testmod.patch with "git am" did not pass, trying reduced context.
[INFO]: Context reduced git-am of .kernel-meta/patches/standard/arm-versatile-926ejs/myqemuarm/links/files/yocto-testmod.patch with "git am" did not work, trying "apply".
####################

The full log is also attached
Comment 2 Yi Zhao 2016-02-16 09:03:46 UTC
Created attachment 2987 [details]
log.do_patch.good
Comment 3 Yi Zhao 2016-02-16 09:04:25 UTC
Created attachment 2988 [details]
yocto-testmod.patch

Here is the patch file
Comment 4 Yi Zhao 2016-02-19 07:52:55 UTC
This issue also exists on 2.1_M2 rc3 build.
git rev: master/152914f2983c5d69001de1d46ce99547fa1e75fe
Comment 5 Leonardo Sandoval Gonzalez 2016-04-19 22:53:01 UTC
Yi

I am seeing the problem (patch does not get into the kernel) in both cases (reusing and not reusing an existing branch). So the problem is now even worse.

My testing was done under:

commit ee25d0e3987d7732a2e46e1640693b4cf419a9fc
Author: Belen Barros Pena <belen.barros.pena@linux.intel.com>
Date:   Mon Apr 18 16:08:18 2016 +0100

(tested using 4.4 and 4.1)

I am not sure if this problem came from the bellow commit because I have not reverted it an test it 

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=jethro&id=4e74b36458b36ee1202689388da5423ed9ee7b71

It may be that that the problem may be on the kern-tools. I will continue my investigation.
Comment 6 Bruce Ashfield 2016-04-20 00:20:34 UTC
(In reply to comment #5)
> Yi
> 
> I am seeing the problem (patch does not get into the kernel) in both cases
> (reusing and not reusing an existing branch). So the problem is now even
> worse.
> 
> My testing was done under:
> 
> commit ee25d0e3987d7732a2e46e1640693b4cf419a9fc
> Author: Belen Barros Pena <belen.barros.pena@linux.intel.com>
> Date:   Mon Apr 18 16:08:18 2016 +0100
> 
> (tested using 4.4 and 4.1)
> 
> I am not sure if this problem came from the bellow commit because I have not
> reverted it an test it 
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/
> ?h=jethro&id=4e74b36458b36ee1202689388da5423ed9ee7b71
> 
> It may be that that the problem may be on the kern-tools. I will continue my
> investigation.

I see the steps to reproduce this now. I'll have a look at the generated .scc
and config files to see if there's an issue there. I'll update the bug(s) in a few
hours with the results.
Comment 7 Bruce Ashfield 2016-04-20 03:13:47 UTC
I've found the problem, and yes, the yocto-bsp tool still needs another tweak
to work with the updated tools.

It was a misunderstanding on my part when the update to the tools was done. Let me
explain below:

- The tools used to always look at all patches in the series and determine which
  ones were applied to the tree. When a patch was found that wasn't on the tree that
  was declared the "Starting point" and all patches from then on were applied.

- That technique was effective, but sometimes had: 1) false positives for the resume
  point (due to similarly worded commits or diffstats) 2) performance isssues, since
  many git operations were used 3) complexity.

- To fix the above issues, the tools were updated to not "auto resume", but instead
  only push patches onto the tree if: 1) a new branch was created 2) the patches
  or .scc files were passed on the SRC_URI.

The .scc files that are being generated by the tool manage to shoot right through
the middle of the patching conditions.

You may ask: "But myqemuarm-standard.scc is passed on the SRC_URI and it then
includes myqemuarm.scc, which in turn finally includes myqemuarm-user-patches.scc". 
So why doesn't that trigger the patching, when one of the conditions was 2) "the
patches or .scc files were passed on the SRC_URI".

The answer is, since that is the BSP description and we've chosen to re-use existing
branches, any patch pushing is turned off when the existing branch is detected
(since you have to travers existing patches to get to the end). 

That's a complicated description of what is going on, but that is the *internals*
of the process (sort of like describing how bitbake processes and parses files).
It works the way it does for some important use cases, they just don't seem
obvious when walking through this one.

I could likely modify the tools to detect this special case of a user-space BSP
description that is (or isn't) re-using branches, but that will make the code
even more fragile. The plan is to simplify it more for the 2.2 release, not make
it more complex.

If we modify the output of the BSP tool, we can make this work without changing
the tools.

1) put the following in myqemuarm-user-patches.scc: 'mark patching start', and
that will force the patches that follow to be applied to the tree

2) put file://myqemuarm-user-patches.scc in the SRC_URI and remove it from 
myqemuarm.scc.

3) leave everything as-is, and add: "branch myqemuarm" to myqemuarm-standard.scc, 
since the new branch will trigger patching. But note, if you do this, you also
need to adjust KBRANCH to be the correct final name ( <whatever the BSP branch
is>/base). Creating this branch is acceptable since the real "Branch reuse" is
to ensure that the existing BSPs changes are inherited, not just that the name
would be directly used.

I'd suggest #1 or #2 for simplicities sake, since they are simple mods to the
bsp tool.

.. and yes, I tested all three solutions :)
Comment 8 Leonardo Sandoval Gonzalez 2016-04-20 19:36:08 UTC
(In reply to comment #7)
> 

Thanks for your description, it is a more clear now.


> 1) put the following in myqemuarm-user-patches.scc: 'mark patching start',
> and

For this option, I tried the following patch but I did not work (in both case, with new machine branch and without it). Is there something else I am missing when implementing your proposed change?

diff --git a/scripts/lib/bsp/kernel.py b/scripts/lib/bsp/kernel.py
index ba68b60..0783228 100644
--- a/scripts/lib/bsp/kernel.py
+++ b/scripts/lib/bsp/kernel.py
@@ -333,6 +333,7 @@ def write_patch_items(scripts_path, machine, patch_items):
     patch list [${machine}-user-patches.scc].
     """
     f = open_user_file(scripts_path, machine, machine+"-user-patches.scc", "w")
+    f.write("mark patching start\n")
     for item in patch_items:
         f.write("patch " + item + "\n")
     f.close()
Comment 9 Leonardo Sandoval Gonzalez 2016-04-20 20:13:24 UTC
(In reply to comment #7)
> 
> 2) put file://myqemuarm-user-patches.scc in the SRC_URI and remove it from 
> myqemuarm.scc.
>

This solution DID work, but again, I find it 1) less intrusive. Waiting for Bruce comments regarding my previous comment.
Comment 10 Bruce Ashfield 2016-04-20 20:37:54 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > 
> 
> Thanks for your description, it is a more clear now.
> 
> 
> > 1) put the following in myqemuarm-user-patches.scc: 'mark patching start',
> > and
> 
> For this option, I tried the following patch but I did not work (in both
> case, with new machine branch and without it). Is there something else I am
> missing when implementing your proposed change?
> 
> diff --git a/scripts/lib/bsp/kernel.py b/scripts/lib/bsp/kernel.py
> index ba68b60..0783228 100644
> --- a/scripts/lib/bsp/kernel.py
> +++ b/scripts/lib/bsp/kernel.py
> @@ -333,6 +333,7 @@ def write_patch_items(scripts_path, machine,
> patch_items):
>      patch list [${machine}-user-patches.scc].
>      """
>      f = open_user_file(scripts_path, machine, machine+"-user-patches.scc",
> "w")
> +    f.write("mark patching start\n")
>      for item in patch_items:
>          f.write("patch " + item + "\n")
>      f.close()

That looks like it should have worked. Let me make the same change here
and re-run the test.
Comment 11 Bruce Ashfield 2016-04-20 23:47:00 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > 
> 
> Thanks for your description, it is a more clear now.
> 
> 
> > 1) put the following in myqemuarm-user-patches.scc: 'mark patching start',
> > and
> 
> For this option, I tried the following patch but I did not work (in both
> case, with new machine branch and without it). Is there something else I am
> missing when implementing your proposed change?
> 
> diff --git a/scripts/lib/bsp/kernel.py b/scripts/lib/bsp/kernel.py
> index ba68b60..0783228 100644
> --- a/scripts/lib/bsp/kernel.py
> +++ b/scripts/lib/bsp/kernel.py
> @@ -333,6 +333,7 @@ def write_patch_items(scripts_path, machine,
> patch_items):
>      patch list [${machine}-user-patches.scc].
>      """
>      f = open_user_file(scripts_path, machine, machine+"-user-patches.scc",
> "w")
> +    f.write("mark patching start\n")
>      for item in patch_items:
>          f.write("patch " + item + "\n")
>      f.close()

I made this same change, and it worked here:

commit a198f511a770f276b75c7bdeef7179028f99b8bd
Author: invalid_git config <unknown@unknown>
Date:   Wed Apr 20 19:45:03 2016 -0400

:100644 100644 22892c701c63... be65df4897e2... M        drivers/misc/Kconfig
:100644 100644 537d7f3b78da... ff37ff70c0a0... M        drivers/misc/Makefile
:000000 100644 000000000000... 1e09d350bee5... A        drivers/misc/yocto-testmod.c

yow-bashfiel-d4 [/home/bruc...nel-source]> git branch
  master
* standard/arm-versatile-926ejs
  standard/base
  standard/beaglebone
  standard/edgerouter
  standard/fsl-mpc8315e-rdb
  standard/mti-malta32
  standard/mti-malta64
  standard/preempt-rt
  standard/qemuarm64
  standard/qemuppc
  standard/tiny/base
  standard/tiny/common-pc
Comment 12 Bruce Ashfield 2016-04-20 23:52:13 UTC
But I'm seeing a problem with the new branch creation. Checking to see what's happening there.
Comment 13 Bruce Ashfield 2016-04-21 00:04:11 UTC
(In reply to comment #12)
> But I'm seeing a problem with the new branch creation. Checking to see
> what's happening there.

Sorry for all the Bugzilla spam. I retract that statement, the patches are
being applied, but the KBRANCH Is not being set properly in the "new branch"
case, I selected "based on standard/base"

yow-bashfiel-d4 [/home/bruc...nel-source]> git branch
  master
  standard/arm-versatile-926ejs/base
  standard/arm-versatile-926ejs/myqemuarm
* standard/base
  standard/beaglebone
  standard/edgerouter
  standard/fsl-mpc8315e-rdb
  standard/mti-malta32
  standard/mti-malta64
  standard/preempt-rt
  standard/qemuarm64
  standard/qemuppc
  standard/tiny/base
  standard/tiny/common-pc

.. but that exposed a bug in a recent change I made to ensure that KBRANCH was
being built. KBRANCH is standard/base, and you can't set it to KBRANCH="standard/arm-versatile-926ejs/myqemuarm"
since the fetcher will error because the branch doesn't actually exist.

Not sure yow to fix this case (without breaking the reason I put the force
build code into the tree at the moment), the "create my own branch" really
isn't a valid use case if you are using a git repo, since you are circumventing
the fetcher in any case. So really, I'd recommend simply hiding that option
versus me hacking the classes to no longer force the branch in a special case
like this. The 'new branch' code pre-dated the fetcher update to sanitize
branches and SRCREVs, so it can go away now :)
Comment 14 Tom Zanussi 2016-04-21 02:22:23 UTC
Hi Bruce,

Coming in a bit late here, so hopefully am not out of context...

Coincidentally I was just trying to update the kernel labs to krogoth and ran into the following using the existing yocto-bsp tool:

WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch: After meta data application, the kernel tree branch is standard/arm-versatile-926ejs/lab5-qemuarm. The
WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch: SRC_URI specified branch standard/base. The branch will be forced to standard/base,
WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch: but this means the board meta data (.scc files) do not match the SRC_URI specification.
WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch: The meta data and branch standard/base should be inspected to ensure the proper
WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch: kernel is being built.

I guess this is basically the same problem, but am at a loss at what to do about it in either case...

The main use case of yocto-bsp was supposed to be to allow a user to create a new BSP assuming he would want a machine-specific branch to hold patches.  Then, once everything was finished and working properly, be able to push it all to the upstream repo (this was even supposed to be automated at some point), including the new branch with all the patches as commits, etc.  The case of using an existing branch was assumed to be the less common case, thus the default of creating a new branch.

I suppose one solution would be to have the user manually create the new branch, which assumes working from a local repo, but that assumes more from the user than a tool targeted at new users should assume.
Comment 15 Bruce Ashfield 2016-04-21 02:53:07 UTC
(In reply to comment #14)
> Hi Bruce,
> 
> Coming in a bit late here, so hopefully am not out of context...

not late, and not out of context! I'm happy to sort this one out in
a way that works for everyone.

> 
> Coincidentally I was just trying to update the kernel labs to krogoth and
> ran into the following using the existing yocto-bsp tool:
> 
> WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch:
> After meta data application, the kernel tree branch is
> standard/arm-versatile-926ejs/lab5-qemuarm. The
> WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch:
> SRC_URI specified branch standard/base. The branch will be forced to
> standard/base,
> WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch:
> but this means the board meta data (.scc files) do not match the SRC_URI
> specification.
> WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch:
> The meta data and branch standard/base should be inspected to ensure the
> proper
> WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch:
> kernel is being built.
> 
> I guess this is basically the same problem, but am at a loss at what to do
> about it in either case...

It is the same thing. I just added that "force the branch" code to solve
another problem when the meta-data took someone in a different direction
than they expected. 

> 
> The main use case of yocto-bsp was supposed to be to allow a user to create
> a new BSP assuming he would want a machine-specific branch to hold patches. 
> Then, once everything was finished and working properly, be able to push it
> all to the upstream repo (this was even supposed to be automated at some
> point), including the new branch with all the patches as commits, etc.  The
> case of using an existing branch was assumed to be the less common case,
> thus the default of creating a new branch.

And I agree with that goal .. the new fetcher has just thrown a bit
of a wrench into things for me :)

I think I have an idea. What if the branch switch code was capable of
being WARN_ONLY ? I could dump the message (and make it less alarming)
but not switch the branch.

The yocto-bsp could set the variable to warn only, and it would work,
and the user that is setting a branch to force a build .. would also
work.

> 
> I suppose one solution would be to have the user manually create the new
> branch, which assumes working from a local repo, but that assumes more from
> the user than a tool targeted at new users should assume.

Yah, I thought that as well. A local repo with the branch would work .. but
that's too much effort and git foo for a beginner.

Let me know what you think of the WARN idea. And I can do a patch.
Comment 16 Tom Zanussi 2016-04-21 03:27:27 UTC
(In reply to comment #15)
> (In reply to comment #14)
> > Hi Bruce,
> > 
> > Coming in a bit late here, so hopefully am not out of context...
> 
> not late, and not out of context! I'm happy to sort this one out in
> a way that works for everyone.
> 
> > 
> > Coincidentally I was just trying to update the kernel labs to krogoth and
> > ran into the following using the existing yocto-bsp tool:
> > 
> > WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch:
> > After meta data application, the kernel tree branch is
> > standard/arm-versatile-926ejs/lab5-qemuarm. The
> > WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch:
> > SRC_URI specified branch standard/base. The branch will be forced to
> > standard/base,
> > WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch:
> > but this means the board meta data (.scc files) do not match the SRC_URI
> > specification.
> > WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch:
> > The meta data and branch standard/base should be inspected to ensure the
> > proper
> > WARNING: linux-yocto-4.4.3+gitAUTOINC+bcc6509084_1a72cec834-r0.1 do_patch:
> > kernel is being built.
> > 
> > I guess this is basically the same problem, but am at a loss at what to do
> > about it in either case...
> 
> It is the same thing. I just added that "force the branch" code to solve
> another problem when the meta-data took someone in a different direction
> than they expected. 
> 
> > 
> > The main use case of yocto-bsp was supposed to be to allow a user to create
> > a new BSP assuming he would want a machine-specific branch to hold patches. 
> > Then, once everything was finished and working properly, be able to push it
> > all to the upstream repo (this was even supposed to be automated at some
> > point), including the new branch with all the patches as commits, etc.  The
> > case of using an existing branch was assumed to be the less common case,
> > thus the default of creating a new branch.
> 
> And I agree with that goal .. the new fetcher has just thrown a bit
> of a wrench into things for me :)
> 
> I think I have an idea. What if the branch switch code was capable of
> being WARN_ONLY ? I could dump the message (and make it less alarming)
> but not switch the branch.
> 
> The yocto-bsp could set the variable to warn only, and it would work,
> and the user that is setting a branch to force a build .. would also
> work.
> 
> > 
> > I suppose one solution would be to have the user manually create the new
> > branch, which assumes working from a local repo, but that assumes more from
> > the user than a tool targeted at new users should assume.
> 
> Yah, I thought that as well. A local repo with the branch would work .. but
> that's too much effort and git foo for a beginner.
> 
> Let me know what you think of the WARN idea. And I can do a patch.

Sounds like that would work, thanks!
Comment 17 Leonardo Sandoval Gonzalez 2016-04-21 14:24:15 UTC
(In reply to comment #12)
> But I'm seeing a problem with the new branch creation. Checking to see
> what's happening there.

Bruce, just to be more specific: I am seeing the problem either choosing 'y' or 'n' on this question:

"Do you need a new machine branch for this BSP (the alternative is to re-use an existing branch)?"

But as I can see, you are seeing the problem just when 'y'. Although the patch was reported for Jethro, I am seeing the problem in latest master (you may be working on Jethro, thus we are seeing different things).

Anyway, I wait for further directions.
Comment 18 Bruce Ashfield 2016-04-21 14:25:50 UTC
(In reply to comment #17)
> (In reply to comment #12)
> > But I'm seeing a problem with the new branch creation. Checking to see
> > what's happening there.
> 
> Bruce, just to be more specific: I am seeing the problem either choosing 'y'
> or 'n' on this question:
> 
> "Do you need a new machine branch for this BSP (the alternative is to re-use
> an existing branch)?"
> 
> But as I can see, you are seeing the problem just when 'y'. Although the
> patch was reported for Jethro, I am seeing the problem in latest master (you
> may be working on Jethro, thus we are seeing different things).
> 
> Anyway, I wait for further directions.

I only ever work on master :)

I'm definitely not seeing the problem when I say 'no', so without access
to one of your builds, there's nothing more I can do.
Comment 19 Leonardo Sandoval Gonzalez 2016-04-21 15:54:32 UTC
Created attachment 3143 [details]
layer created with yocto-bsp command

This layer was created with the following commands:

$ yocto-bsp create myqemuarm qemu
Checking basic git connectivity...
Done.

Which qemu architecture would you like to use? [default: i386]
        1) i386    (32-bit)
        2) x86_64  (64-bit)
        3) ARM     (32-bit)
        4) PowerPC (32-bit)
        5) MIPS    (32-bit)
        6) MIPS64  (64-bit)
3
Would you like to use the default (4.4) kernel? (y/n) [default: y] 
Do you need a new machine branch for this BSP (the alternative is to re-use an existing branch)? [y/n] [default: y] n
Getting branches from remote repo git://git.yoctoproject.org/linux-yocto-4.4.git...
Please choose an existing machine branch to use for this BSP: [default: standard/arm-versatile-926ejs]
        1) standard/arm-versatile-926ejs
        2) standard/base
        3) standard/beaglebone
        4) standard/edgerouter
        5) standard/fsl-mpc8315e-rdb
        6) standard/mti-malta32
        7) standard/mti-malta64
        8) standard/preempt-rt
        9) standard/qemuarm64
        10) standard/qemuppc
1
Would you like SMP support? (y/n) [default: y] 
Does your BSP have a touchscreen? (y/n) [default: n] 
Does your BSP have a keyboard? (y/n) [default: y] 

New qemu BSP created in meta-myqemuarm

$ yocto-kernel patch add myqemuarm ./yocto-testmod.patch
$ yocto-kernel config add myqemuarm CONFIG_MISC_DEVICES=y
$ yocto-kernel config add myqemuarm CONFIG_YOCTO_TESTMOD=y
Comment 20 Leonardo Sandoval Gonzalez 2016-04-21 15:59:41 UTC
(In reply to comment #18)

> I'm definitely not seeing the problem when I say 'no', so without access
> to one of your builds, there's nothing more I can do.

You are completely right. I have attached a layer, where the 'bitbake -c patch linux-yocto' does not patch the corresponding patch file.

The way to check if it was patch or now, is the following

$bitbake -c devshell linux-yocto
$yctb05:~/poky/build-qemuarm/tmp/work-shared/qemuarm/kernel-source # grep -irn yocto drivers/misc/
$yctb05:~/poky/build-qemuarm/tmp/work-shared/qemuarm/kernel-source #
Comment 21 Bruce Ashfield 2016-04-22 19:58:22 UTC
(In reply to comment #20)
> (In reply to comment #18)
> 
> > I'm definitely not seeing the problem when I say 'no', so without access
> > to one of your builds, there's nothing more I can do.
> 
> You are completely right. I have attached a layer, where the 'bitbake -c
> patch linux-yocto' does not patch the corresponding patch file.
> 
> The way to check if it was patch or now, is the following
> 

Not really. The way to see if the kernel change is applied, is to inspect
the git history :)

> $bitbake -c devshell linux-yocto
> $yctb05:~/poky/build-qemuarm/tmp/work-shared/qemuarm/kernel-source # grep
> -irn yocto drivers/misc/
> $yctb05:~/poky/build-qemuarm/tmp/work-shared/qemuarm/kernel-source #
Comment 22 Bruce Ashfield 2016-04-22 20:24:06 UTC
With the layer configured into my build, I have:

yow-bashfiel-d4 [/home/bruc...nel-source]> pwd
/home/bruce/poky/build/tmp/work-shared/myqemuarm/kernel-source

yow-bashfiel-d4 [/home/bruc...nel-source]> git branch
  master
* standard/arm-versatile-926ejs
  standard/base
  standard/beaglebone
  standard/edgerouter
  standard/fsl-mpc8315e-rdb
  standard/mti-malta32
  standard/mti-malta64
  standard/preempt-rt
  standard/qemuarm64
  standard/qemuppc
  standard/tiny/base
  standard/tiny/common-pc

yow-bashfiel-d4 [/home/bruc...nel-source]> git whatchanged 

commit 5fa96c86adfa81c2e33ec4543b0e9429b17a49c0
Author: invalid_git config <unknown@unknown>
Date:   Fri Apr 22 16:19:58 2016 -0400

:100644 100644 22892c701c63... be65df4897e2... M        drivers/misc/Kconfig
:100644 100644 537d7f3b78da... ff37ff70c0a0... M        drivers/misc/Makefile
:000000 100644 000000000000... 1e09d350bee5... A        drivers/misc/yocto-testmod.c
Comment 23 Bruce Ashfield 2016-04-22 20:25:41 UTC
Also, when running this test, make sure you are using my poky-contrib zedd/kernel
branch. It has the latest kern-tools changes as a commit.
Comment 24 Leonardo Sandoval Gonzalez 2016-04-22 20:50:43 UTC
(In reply to comment #23)
> Also, when running this test, make sure you are using my poky-contrib
> zedd/kernel
> branch. It has the latest kern-tools changes as a commit.

I see, that may be the reason. The git whatchanged cmd outputs different output compared to yours.

In the other hand, the HEAD of that branch looks quite old:

2012-05-09	kern-tools: fix do_patch errorszedd/kern-tools

but I believe is a consequence of commit's dates.
Comment 25 Bruce Ashfield 2016-04-22 20:53:38 UTC
(In reply to comment #24)
> (In reply to comment #23)
> > Also, when running this test, make sure you are using my poky-contrib
> > zedd/kernel
> > branch. It has the latest kern-tools changes as a commit.
> 
> I see, that may be the reason. The git whatchanged cmd outputs different
> output compared to yours.
> 
> In the other hand, the HEAD of that branch looks quite old:
> 
> 2012-05-09	kern-tools: fix do_patch errorszedd/kern-tools

What are you looking at for that ? Certainly not:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=zedd/kernel

Bruce

> 
> but I believe is a consequence of commit's dates.
Comment 26 Leonardo Sandoval Gonzalez 2016-04-22 21:02:03 UTC
(In reply to comment #25)

> What are you looking at for that ? Certainly not:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=zedd/kernel
> 
> Bruce

You mentioned 'It has the latest kern-tools changes as a commit.', so I look for that branch, but know I see you mean zedd/kernel.

Thanks, let me do the testing again!
Comment 27 Leonardo Sandoval Gonzalez 2016-04-25 16:23:44 UTC
(In reply to comment #23)
> Also, when running this test, make sure you are using my poky-contrib
> zedd/kernel
> branch. It has the latest kern-tools changes as a commit.

Bruce,

not sure what is going on again. I included the patch mentioned on comment 8, on top of your latest zedd/kernel:

  commit 16ec75e6eaa2de42a05fb1684e3b99da7d9265b7
  Author: Bruce Ashfield <bruce.ashfield@windriver.com>
  Date:   Fri Apr 22 15:03:24 2016 -0400

      linux-yocto/4.1: make ltsi content available


But patches are not being applied yet. Again, I did all my testing using yocto-bsp and yocto-kernel tools. In the other hand, after these changes on your contrib branch, is the patch on comment 8 really needed?

Thanks for your help on this issue.
Comment 28 Bruce Ashfield 2016-04-25 16:25:05 UTC
(In reply to comment #27)
> (In reply to comment #23)
> > Also, when running this test, make sure you are using my poky-contrib
> > zedd/kernel
> > branch. It has the latest kern-tools changes as a commit.
> 
> Bruce,
> 
> not sure what is going on again. I included the patch mentioned on comment
> 8, on top of your latest zedd/kernel:
> 
>   commit 16ec75e6eaa2de42a05fb1684e3b99da7d9265b7
>   Author: Bruce Ashfield <bruce.ashfield@windriver.com>
>   Date:   Fri Apr 22 15:03:24 2016 -0400
> 
>       linux-yocto/4.1: make ltsi content available
> 
> 
> But patches are not being applied yet. Again, I did all my testing using
> yocto-bsp and yocto-kernel tools. In the other hand, after these changes on
> your contrib branch, is the patch on comment 8 really needed?

Yep. You still need that patch.

I have no patching issues. So again, unless there's a machine that
I can log into and poke at this. I can't help any more.

Bruce

> 
> Thanks for your help on this issue.
Comment 29 Leonardo Sandoval Gonzalez 2016-04-25 17:02:46 UTC
(In reply to comment #28)
> I have no patching issues. So again, unless there's a machine that
> I can log into and poke at this. I can't help any more.
> 

There is no need to share the my bsp layer, you can create it yourself. 

Here are the instructions:

- git checkout -b test contrib/zedd/kernel 
- git am yocto-testmod.patch
- $ . oe-init-build-env build-qemuarm
- $ vim conf/local.conf and set MACHINE to "qemuarm"
- $ yocto-bsp create myqemuarm qem # default ALL (I tried it with 4.1 and 4.4, same result)
- $ yocto-kernel patch add myqemuarm ./yocto-testmod.patch
- $ bitbake -c patch linux-yocto
- $ bitbake -c devshell linux-yocto
- $ git status
- $ git whatchanged

In my case, the last git commands do show the patch been applied.
Comment 30 Bruce Ashfield 2016-04-25 17:03:48 UTC
(In reply to comment #29)
> (In reply to comment #28)
> > I have no patching issues. So again, unless there's a machine that
> > I can log into and poke at this. I can't help any more.
> > 
> 
> There is no need to share the my bsp layer, you can create it yourself. 
> 
> Here are the instructions:
> 
> - git checkout -b test contrib/zedd/kernel 
> - git am yocto-testmod.patch
> - $ . oe-init-build-env build-qemuarm
> - $ vim conf/local.conf and set MACHINE to "qemuarm"
> - $ yocto-bsp create myqemuarm qem # default ALL (I tried it with 4.1 and
> 4.4, same result)
> - $ yocto-kernel patch add myqemuarm ./yocto-testmod.patch
> - $ bitbake -c patch linux-yocto
> - $ bitbake -c devshell linux-yocto
> - $ git status
> - $ git whatchanged
> 
> In my case, the last git commands do show the patch been applied.

And I'm saying that this works fine for me, in all cases. So there's
nothing more I can do on my machines.
Comment 31 Leonardo Sandoval Gonzalez 2016-04-25 17:11:39 UTC
(In reply to comment #30)

> And I'm saying that this works fine for me, in all cases. So there's
> nothing more I can do on my machines.

That is fine. how do we proceed with this? the yocto-bsp patch (comment 8) should be ideally contained on your patch series from zedd/kernel, so you can either create the patch or I can send it to the ML (but it wont make much difference unless yours are applied).

Let me know who you think is best to proceed.
Comment 32 Bruce Ashfield 2016-04-25 17:14:24 UTC
(In reply to comment #31)
> (In reply to comment #30)
> 
> > And I'm saying that this works fine for me, in all cases. So there's
> > nothing more I can do on my machines.
> 
> That is fine. how do we proceed with this? the yocto-bsp patch (comment 8)
> should be ideally contained on your patch series from zedd/kernel, so you
> can either create the patch or I can send it to the ML (but it wont make
> much difference unless yours are applied).
> 
> Let me know who you think is best to proceed.

As I said ... "without access to a machine that is showing this problem", there's nothing more I can do.

So my way to proceed is to do nothing more.

I'm not planning to send any patches against yocto-bsp, since it isn't something that
I use and won't test in the future.

The kern-tools, and yocto-bsp patches are completely independent, so there's no need
for them to be in a single series.
Comment 33 Leonardo Sandoval Gonzalez 2016-04-25 19:03:58 UTC
(In reply to comment #28)


There are many missing and wrong steps in comment 28, hopefully this is the correct (and sorry for the spam)

- $ git checkout -b test contrib/zedd/kernel 
- $ git am 0001-scripts-lib-bsp-kernel.py-force-patching-when-branch.patch
- $ . oe-init-build-env build-qemuarm
- $ vim conf/local.conf and set MACHINE to "qemuarm"
- $ yocto-bsp create myqemuarm qem # Select ARM, the rest defaults (I tried it with 4.1 and 4.4, same result)
- $ vim conf/bblayers.conf and include the new layer
- $ yocto-kernel patch add myqemuarm ./yocto-testmod.patch
- $ bitbake -c patch linux-yocto
- $ bitbake -c devshell linux-yocto
- $ git status
- $ git whatchanged
Comment 34 Leonardo Sandoval Gonzalez 2016-04-25 19:05:42 UTC
Created attachment 3156 [details]
Patch sent to the ML
Comment 35 Leonardo Sandoval Gonzalez 2016-04-25 21:06:54 UTC
Patches are now applied with the attached patch.
Comment 36 Tom Zanussi 2016-04-27 23:49:21 UTC
With

0001-kernel-yocto-allow-branch-auditing-to-be-suspended.patch

I no longer see the warnings and forced branch switch.  So it seems to work as expected and I was able to finish updating the lab.

I also verified that 'yocto-kernel add patch' on an existing branch fails to add the patch without

0001-scripts-lib-bsp-kernel.py-force-patching-when-branch.patch

but works fine with it.

So afaict it all looks good now, thanks!
Comment 37 Ross Burton 2016-05-19 16:31:17 UTC
Fixed in oe-core 427f5473722e15e288cbce251a9ce18989c23548.
Comment 38 Yi Zhao 2016-08-04 06:17:33 UTC
(In reply to comment #37)
> Fixed in oe-core 427f5473722e15e288cbce251a9ce18989c23548.

Although the following 2 patches were backported to krogoth branch, I can still reproduce this issue in 2.1.1 RC2 build (git rev: krogoth/f5da2a5913319ad6ac2141438ba1aa17576326ab):


commit 720ae184031db33049f9be6326f9735224183492
Author: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Date:   Mon Apr 25 09:40:22 2016 +0000

    scripts/lib/bsp/kernel.py: force patching when branch is machine branch is re-use
    
    When a branch is re-used, the kernel tools turns off any patch pushing unless
    'mark patching' is explicitly set.
    
    [YOCTO #9120]
    


commit de9f4b6982696bed40cd1038adbb7f9fc88523cd
Author: Bruce Ashfield <bruce.ashfield@windriver.com>
Date:   Thu Apr 21 11:23:45 2016 -0400

    kernel-yocto: allow branch auditing to be suspended
    
    When working on the yocto-bsp and kernel-lab update for yocto 1.2
    we found it was impossible for a end-user BSP to isolate patches
    on a branch, since with the following commit:
    
      [kernel-yocto: enforce SRC_URI specified branch]
    
    Any new branch would be switched to whatever was specified on the
    SRC_URI and undoing the work that the yocto-bsp tool did to support
    board specific patches.
    
    To fix this, we'll keep the enforcing of branch consistency enabled
    by default, but introduce a variable "KMETA_AUDIT" that when not
    set will skip the check.
    
    There's no impact for existing users, and it is only something that
    other plumbing commands and tools will need to use (or care about).
    
    [YOCTO: #9120]
Comment 39 Yi Zhao 2016-11-15 07:26:57 UTC
This issue still happens in 2.1.2 RC1 build
Comment 40 Leonardo Sandoval Gonzalez 2016-11-15 19:50:16 UTC
(In reply to comment #39)
> This issue still happens in 2.1.2 RC1 build

I suggest filing a new bug because this bug entry was targeted for 2.2. Yi, can you help me with that? The solution may just consist of back-porting what we did for 2.2
Comment 41 Yi Zhao 2016-11-16 06:29:48 UTC
(In reply to comment #40)
> (In reply to comment #39)
> > This issue still happens in 2.1.2 RC1 build
> 
> I suggest filing a new bug because this bug entry was targeted for 2.2. Yi,
> can you help me with that? The solution may just consist of back-porting
> what we did for 2.2

A new bug 10674 was filed. And this bug can be verified.