8289 2015-09-11 15:07:25 +0000 kernel_configme step should use "oldnoconfig" instead of "allnoconfig" when silently upgrading user defconfig 2016-09-09 17:06:44 +0000 1 1 6 Yocto Project Subprojects Kernel kernel-tooling 1.8.1 All Multiple RESOLVED FIXED Medium normal 2.2 M3 1 rcalhoun-bugzilla bruce.ashfield sgw yp.kernel.watcher yp.watcher --- --- No (bug/feature does not impact docs) oldest_to_newest 53979 0 rcalhoun-bugzilla 2015-09-11 15:07:25 +0000 This bug was found by compiling linux 4.2 (mainline) using yocto fido 1.8.1 starting with a defconfig from linux-yocto-3.19. Automatic upgrade of the defconfig by the build system resulted in (new in 4.1) kernel option CONFIG_MULTIUSER being set to: # CONFIG_MULTIUSER is not set This builds fine, but a kernel that has CONFIG_MULTIUSER undefined causes numerous (and an potentially serious) runtime issues for embedded systems, including breaking remote login via dropbear and causing numerous issues with busybox. The busybox issues are tracked here: http://lists.busybox.net/pipermail/busybox/2015-May/082910.html I will file a separate ticket against dropbear. (Dropbear runs fine but fails during user login with "dropbear[3741]: Exit (root): Failed to set euid") The sense of CONFIG_MULTIUSER is by design (allnoconfig should generate de minimal kernel.) Since CONFIG_MULTIUSER is an advanced option that defaults to y, silent upgrades should define this kernel option (and others that default to Y) to avoid unexpected behavior. Yocto has two steps that modify the kernel configuration: do_kernel_configme (defined in classes/kernel-yocto.bbclass) do_configure (defined in classes/kernel.bbclass) kernel-yocto.bbclass uses allnoconfig; kernel.bbclass uses allnoconfig. Since do_kernel_configme runs before do_configure, user's defconfig is silently upgraded with to "CONFIG_MULTIUSER is not set" and kernel.bbclass's allnoconfig does not change it. kernel-yocto.bbclass should use "oldnoconfig" insted of "allnoconfig" when silently upgrading user's config to avoid unexpected behavior when upgrading kernel defconfigs. 53984 1 bruce.ashfield 2015-09-11 17:09:24 +0000 (In reply to comment #0) > This bug was found by compiling linux 4.2 (mainline) using yocto fido 1.8.1 > starting with a defconfig from linux-yocto-3.19. Automatic upgrade of the > defconfig by the build system resulted in (new in 4.1) kernel option > CONFIG_MULTIUSER being set to: > > # CONFIG_MULTIUSER is not set > > This builds fine, but a kernel that has CONFIG_MULTIUSER undefined causes > numerous (and an potentially serious) runtime issues for embedded systems, > including breaking remote login via dropbear and causing numerous issues > with busybox. > > The busybox issues are tracked here: > http://lists.busybox.net/pipermail/busybox/2015-May/082910.html > > I will file a separate ticket against dropbear. (Dropbear runs fine but > fails during user login with "dropbear[3741]: Exit (root): Failed to set > euid") > > The sense of CONFIG_MULTIUSER is by design (allnoconfig should generate de > minimal kernel.) Since CONFIG_MULTIUSER is an advanced option that defaults > to y, silent upgrades should define this kernel option (and others that > default to Y) to avoid unexpected behavior. > > Yocto has two steps that modify the kernel configuration: > > do_kernel_configme (defined in classes/kernel-yocto.bbclass) > do_configure (defined in classes/kernel.bbclass) > > kernel-yocto.bbclass uses allnoconfig; kernel.bbclass uses allnoconfig. > Since do_kernel_configme runs before do_configure, user's defconfig is > silently upgraded with to "CONFIG_MULTIUSER is not set" and kernel.bbclass's > allnoconfig does not change it. > > kernel-yocto.bbclass should use "oldnoconfig" insted of "allnoconfig" when > silently upgrading user's config to avoid unexpected behavior when upgrading > kernel defconfigs. The use of allnoconfig is on purpose, the processing of defconfigs is not intented as an upgrade path. The existing semantics of the defconfig processing in the yocto model (well, technically we'd rather they not be used at all .. but that is a different story) is that if you supply a deconfig, those are the settings that the use would explicitly like to set for the given kernel .. everything else should be turned off, to wipe the slate clean, and then the deconfig is applied. I won't break those existing semantics, but can patch the tools to allow a different mode to be passed via KCONFIG_MODE, and that should meet the need here. 53989 2 rcalhoun-bugzilla 2015-09-11 21:04:57 +0000 > The use of allnoconfig is on purpose, the processing of defconfigs is not intented as an upgrade path. Umm, so what is the correct upgrade path? It seems to me that there should be a kernel task defined to this, something comparable to the existing do_menuconfig. In the absence of a e.g. do_upgrade_kernel_config (which would call oldconfig?) one needs to upgrade the kernel configuration some other way. Doing it by hand within the yocto system (via -c devshell, make oldnoconfig) is not too easy, especially now that source directory and the build directory are different. I know how to do it now, but I did not know how to do it the first time I built the 4.2 kernel and got "unexpected results". Hence the bug report: yocto currently has two .config upgrade scripts defined (one in do_kernel_configme and one in do_configure), they are not the same, and the allnoconfig called in do_kernel_configme can lead one pretty badly into the weeds without any warning. (Yocto throws warnings about lesser matters than this!) > well, technically we'd rather they not be used at all .. but that is a different story Ah, configuration fragments! :-) I am currently doing board bring-up on a new machine using yocto. I am trying to use best practices with layers, patches, kernel recipes and so forth. I suppose I am just doing it all wrong, but it is pretty difficult even without configuration fragments. 54000 3 bruce.ashfield 2015-09-14 02:04:36 +0000 (In reply to comment #2) > > The use of allnoconfig is on purpose, the processing of defconfigs is not intented as an upgrade path. > > Umm, so what is the correct upgrade path? > > It seems to me that there should be a kernel task defined to this, something > comparable to the existing do_menuconfig. In the absence of a e.g. > do_upgrade_kernel_config (which would call oldconfig?) one needs to upgrade > the kernel configuration some other way. > > Doing it by hand within the yocto system (via -c devshell, make oldnoconfig) > is not too easy, especially now that source directory and the build > directory are different. I know how to do it now, but I did not know how to > do it the first time I built the 4.2 kernel and got "unexpected results". > Hence the bug report: yocto currently has two .config upgrade scripts > defined (one in do_kernel_configme and one in do_configure), they are not > the same, and the allnoconfig called in do_kernel_configme can lead one > pretty badly into the weeds without any warning. (Yocto throws warnings > about lesser matters than this!) For the most part, folks maintaining kernels and configuration blocks for those kernels, only use the build system at the very end of the cycle. It builds and generates and image, but as for maintenance of the tree, and the config .. they do that themselves, and when happy, codify it in the build system. Hence none of the tasks is really meant for upgrading a config, they are meant to apply the config and build it. The devtool work that is happening, is trying to address some of the most common development tasks, and can somewhat address this use case, but getting a config that you are happy with, is still in the hands of the developer. Note: If you are using fragments there is more auditing available, and new options are flagged when the base fragments are upgraded between revisions, so many options and unexpected results are caught when putting thee baseline in place. Having a dedicated task for upgrades, and extending configme to take oldnoconfig as an optional mode are both things that I'm willing to do. If there's a preference .. feel free to indicate it here. Bruce > > > well, technically we'd rather they not be used at all .. but that is a different story > > Ah, configuration fragments! :-) > > I am currently doing board bring-up on a new machine using yocto. I am > trying to use best practices with layers, patches, kernel recipes and so > forth. I suppose I am just doing it all wrong, but it is pretty difficult > even without configuration fragments. 54301 4 rcalhoun-bugzilla 2015-09-18 20:14:12 +0000 > Having a dedicated task for upgrades, and extending configme to take oldnoconfig as an optional mode are both things that I'm willing to do. If there's a preference .. feel free to indicate it here. Thanks for taking this on, Bruce. If it's not too late, I think adding a dedicated task for upgrades would be cleanest. Reasoning as follows: Josh Triplett (who worked on the CONFIG_MULTIUSER feature) recommended running "oldconfig" (vs "oldnoconfig") when upgrading a .config to a newer version of linux. This is good advice, but interactive scripts are the bane of build automation and thus "oldconfig" is not a realistic option for do_configure/do_kernel_configme. Yocto can always throw a warnings, however. I see no downside to throwing a warning whenever the (supplied / constructed) .config does not match the version of linux being built. The warning could recommend the user run the (new) upgrade task (do_oldconfig or do_upgrade_config or whatever) and build again. Like the current do_menuconfig task, the upgrade task can be interactive without breaking build automation. 54482 5 bruce.ashfield 2015-09-22 18:21:55 +0000 (In reply to comment #4) > > Having a dedicated task for upgrades, and extending configme to take > oldnoconfig as an optional mode are both things that I'm willing to > do. If there's a preference .. feel free to indicate it here. > > Thanks for taking this on, Bruce. > > If it's not too late, I think adding a dedicated task for upgrades would be > cleanest. We've missed the gate for the 2.0 release, but I do have some work in progress now. I can submit it to master and get a backport, since this is new functionality versus a behaviour change. > > Reasoning as follows: > > Josh Triplett (who worked on the CONFIG_MULTIUSER feature) recommended > running "oldconfig" (vs "oldnoconfig") when upgrading a .config to a newer > version of linux. This is good advice, but interactive scripts are the bane > of build automation and thus "oldconfig" is not a realistic option for > do_configure/do_kernel_configme. > > Yocto can always throw a warnings, however. I see no downside to throwing a > warning whenever the (supplied / constructed) .config does not match the > version of linux being built. The warning could recommend the user run the > (new) upgrade task (do_oldconfig or do_upgrade_config or whatever) and build > again. Like the current do_menuconfig task, the upgrade task can be > interactive without breaking build automation. Yes, there's a balance we can strike between warning (and not alarming) and not breaking the old behaviour. There's already an audit task, and it is easy enough to enhance it to pick up defconfig as well as fragment issues. 66014 6 bruce.ashfield 2016-09-09 17:06:44 +0000 It took a long time, but this has now been dealt with in the master branch of oe-core, which is the 2.2 release. The kernel-yocto configuration has been re-worked, and configme removed from the processing of fragments. This means that any fragments, defconfigs, etc, are processed by merge_config.sh directly. The mode that merge_config.sh uses is controlled by the KCONFIG_MODE variable. There's still no way to reliably detect if a defconfig is complete, a savedefconfig output, or an old defconfig so the control of what is to be done is left up to the recipe author / user. If KCONFIG_MODE is set to alldefconfig, merge_config will concatenate the provided configuration values (including a defconfig or only a defconfig) and then issue an alldefconfig configuration step. The end result is the same as an oldnoconfig/olddefconfig, in which any new symbols are filled in with their defaults. We would have to hack merge_config.sh, or do some extra detection if we really needed olddefconfig versus alldefconfig (with the delta really just being the "existing" versus "new config"). In that case, again, we could write a task that did an update and saved the new defconfig to a specified location which could be fed back into the build. New tasks/complexity in this area are to be avoided .. so trying the current implementation is the plan.