Bug 11146

Exact steps:
commitish b1f09df0f664052e39d939b759a63b05d767b8ec
conf/local.conf:
MACHINE="qemux86_64"
IMAGE_FSTYPE += " wic "

$ bitbake core-image-minimal
$ bitbake -c menuconfig virtual/kernel
 -> uncheck /proc/config.gz setting to off.
 -> exit 
 -> say "yes" when it asks you to save
$ bitbake core-image-minimal
 -> bb says "2556 tasks of which 2556 didn't need to be run"
-------
If this is working for others, please add any additional commands you ran.
---

Thinking about this more, this is what I've always seen with this (then again, I only use menuconfig rarely).

Are you noticing this as a chance in behaviour, or just as a behaviour ?

Either way is fine, I'd just like to characterize the problem so I can look for a bug, or see about a new feature to taint the task when the .config is modified.

My question still stands if this is a change in behaviour .. but I do see that the compile taint did make it into the cml1.bbclass:

-------

    # FIXME this check can be removed when the minimum bitbake version has been bumped
    if hasattr(bb.build, 'write_taint'):
        try:
            newmtime = os.path.getmtime(".config")
        except OSError:
            newmtime = 0

        if newmtime > mtime:
            bb.note("Configuration changed, recompile will be forced")
            bb.build.write_taint('do_compile', d)

-------

Something in rss or the recent host tools changes make have broken it .. I'll see if I can debug it while I wait for input.

I just tried it on morty and saw the same behaviour.

The mtime is coming back the for the .config before and after the modification, so we are not tainting the task.

Trying out some fixes.

Unless I put a significant sleep after the oe_terminal call exits, we always get the same mtime on .config before and after.

I also tried doing a file compare (which IMHO is a better check, and one that I may use regardless), and without that same sleep .. it also returns that the files are the same when in fact .config and .config.orig do differ.

There seems to be a delay before .config actually lands on disk, and hence why the various calls are not picking up the delta.

I asked RP to comment, but if anyone else has the python foo to figure out what is happening here. I wouldn't object. :D

With the following change:

diff --git a/meta/classes/cml1.bbclass b/meta/classes/cml1.bbclass
index 38e6613c486d..c1e732fb6adf 100644
--- a/meta/classes/cml1.bbclass
+++ b/meta/classes/cml1.bbclass
@@ -19,6 +19,7 @@ TERMINFO = "${STAGING_DATADIR_NATIVE}/terminfo"
 KCONFIG_CONFIG_COMMAND ??= "menuconfig"
 python do_menuconfig() {
     import shutil
+    import filecmp
 
     try:
         mtime = os.path.getmtime(".config")
@@ -36,6 +37,19 @@ python do_menuconfig() {
         except OSError:
             newmtime = 0
 
+        bb.warn( "we are in: %s\n" % os.getcwd() )
+        #time.sleep(20)
+        bb.warn( "mtime: %s newtime: %s\n" % (mtime, newmtime))
+        cmd = d.expand("diff -u .config .config.orig")
+        bb.warn("running %s" % cmd)
+        (ret, result) = oe.utils.getstatusoutput("%s" % cmd)
+        bb.warn( "diff claims: %s" % result )
+
+        if filecmp.cmp('.config', '.config.orig'):
+            bb.error(".config and .config.old are the same")
+        else:
+            bb.error(".config and .config.old differ")
+
         if newmtime > mtime:
             bb.note("Configuration changed, recompile will be forced")
             bb.build.write_taint('do_compile', d)

-------------------------

I sometimes get:

WARNING: linux-yocto-4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0 do_menuconfig: we are in: /home/bruce/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0/linux-qemux86_64-standard-build

WARNING: linux-yocto-4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0 do_menuconfig: mtime: 1489533551.803492 newtime: 1489533551.803492

WARNING: linux-yocto-4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0 do_menuconfig: running diff -u .config .config.orig
WARNING: linux-yocto-4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0 do_menuconfig: diff claims: 

------------

i.e. the diff and filecmp don't seem to run, even when I do modify the file.

If I touch the python routine, and/or tweak the time, I get what I expect:

WARNING: linux-yocto-4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0 do_menuconfig: we are in: /home/bruce/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0/linux-qemux86_64-standard-build

WARNING: linux-yocto-4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0 do_menuconfig: mtime: 1489534099.2431152 newtime: 1489534099.2431152

WARNING: linux-yocto-4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0 do_menuconfig: running diff -u .config .config.orig
WARNING: linux-yocto-4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0 do_menuconfig: diff claims: --- .config 2017-03-14 19:31:21.921651923 -0400
+++ .config.orig        2017-03-14 19:31:14.325712074 -0400
@@ -132,8 +132,9 @@
 CONFIG_RCU_STALL_COMMON=y
 # CONFIG_TREE_RCU_TRACE is not set
 # CONFIG_RCU_EXPEDITE_BOOT is not set
-# CONFIG_BUILD_BIN2C is not set
-# CONFIG_IKCONFIG is not set
+CONFIG_BUILD_BIN2C=y
+CONFIG_IKCONFIG=y
+# CONFIG_IKCONFIG_PROC is not set
 CONFIG_LOG_BUF_SHIFT=17
 CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
 CONFIG_NMI_LOG_BUF_SHIFT=13
ERROR: linux-yocto-4.10.2+gitAUTOINC+69bfa87234_827a1164b1-r0 do_menuconfig: .config and .config.old differ

---------------------------------------------------

So there's clearly something that I do not understand about how bitbake is compiling/hashing that routine, and why we are not getting consistent results.

I emailed RP and added him to the bug .. I'm sure this is trivial for him, but over my head to figure out at the moment.

I had an email out to Richard on this, since I can't explain why I'm not seeing deterministic behaviour.

There are other high priority items for 2.3, and since this isn't an entirely new issue, I'm moving it to 2.4, and we can backport as required.

After looking at this again and was able to reproduce this, I think I tracked the issue down to TMUX terminal operations. Still not clear why this is happening, but as Bruce has pointed out without a significant delay (~12-15 seconds) the mtime difference is not detected correctly.

I tried to change to gnome, terminilogy and screen and they all worked and correctly detected a change in mtime and appropriately created the taint file.  I also tested with auto and an existing tmux session and that failed as well.

I further looked at the .config file (in a non-tmux session) right after exiting the menuconfig process in a tmux session and the file had been changed, the grep occurred within 10 seconds.  This leads me to believe that some how the TMUX session and maybe Python is caching something in strange way.

Basically, I just confirmed others findings, except that I went to older stable versions and all behave the same. It is 100% reproducible on master, pyro, morty and krogoth on a xterm terminal.

This is a diff of .config.orig and .config just after quiting the menuconfig, changing the recommended configuration. There is a delta of about 10 seconds between these, and somehow the engine is not detecting this change, thus no task is being tainted.

diff -Naur tmp/work/qemux86-poky-linux/linux-yocto/4.4.26+gitAUTOINC+3030330b06_ca6a08bd7f-r0/linux-qemux86-standard-build/.config.orig tmp/work/qemux86-poky-linux/linux-yocto/4.4.26+gitAUTOINC+3030330b06_ca6a08bd7f-r0/linux-qemux86-standard-build/.config
--- tmp/work/qemux86-poky-linux/linux-yocto/4.4.26+gitAUTOINC+3030330b06_ca6a08bd7f-r0/linux-qemux86-standard-build/.config.orig        2017-07-24 12:12:08.500933881 -0700
+++ tmp/work/qemux86-poky-linux/linux-yocto/4.4.26+gitAUTOINC+3030330b06_ca6a08bd7f-r0/linux-qemux86-standard-build/.config     2017-07-24 12:12:18.109169295 -0700
@@ -126,7 +126,7 @@
 # CONFIG_RCU_EXPEDITE_BOOT is not set
 CONFIG_BUILD_BIN2C=y
 CONFIG_IKCONFIG=y
-# CONFIG_IKCONFIG_PROC is not set
+CONFIG_IKCONFIG_PROC=y
 CONFIG_LOG_BUF_SHIFT=17
 CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
 CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
lsandov1@yctb05:~/poky-testing/poky-menuconfig-tmux/build-krogoth>

Nothing conclusive so far. Plan to tackle it again during M3.

One interesting fact is that the 'screen' terminal does not have this issue, so at least it can be used as a workaround, i.e. OE_TERMINAL = "screen" at local.conf

Created attachment 3888 [details]
strace log obtained when running bitbake -c menuconfig using tmux as terminal

cmd used:
$ strace -o strace.verbose.tmux -v -f -ttt bitbake -c menuconfig virtual/kernel -v

Pyro branch used as metadata.

Created attachment 3889 [details]
strace log obtained when running bitbake -c menuconfig using screen as terminal

cmd:
strace -o strace.verbose.screen -v -f -ttt bitbake -c menuconfig virtual/kernel -v

Leonardo,

It seems the GNOME Terminal fix can be applied here as well; it thus reminds me that something more generic must be done to avoid duplicating same code for TMUX handling.

Are you looking at it? otherwise I can pick this one.

(In reply to comment #14)
> Leonardo,
> 
> It seems the GNOME Terminal fix can be applied here as well; it thus reminds
> me that something more generic must be done to avoid duplicating same code
> for TMUX handling.
> 
> Are you looking at it? otherwise I can pick this one.

You are right, perhaps we need to generalized the fix for all terminals. Let me work on it.

Regarding the GNOME terminal fix that we are talking about, this is the related commit

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=c706bfbabbf9f7caf2cf509eb91381fb49aa44cb

Proposed patch sent to the mailing list:

http://lists.openembedded.org/pipermail/openembedded-core/2017-July/140187.html

Patch merged into oe-core, and present at poky

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d2ebee7c3a0c8ab9a09ebad12a2fda51f8d10954