5090 2013-08-30 15:27:00 +0000 KMETA mis-used for both directories and branches 2014-10-14 07:49:30 +0000 1 1 6 Yocto Project Subprojects Kernel kernel-configuration 1.5 x86 Multiple RESOLVED FIXED Medium+ normal 1.7 M3 1 pab bruce.ashfield alexandru.c.georgescu dvhart sgw sjolley.yp.pm yp.kernel.watcher yp.watcher georgex.l.musat --- --- Don't know oldest_to_newest 36101 0 pab 2013-08-30 15:27:00 +0000 I'm starting with the current poky linux-yocto_3.10.bb recipe and attempting to add support for a new machine. I want to use standard features, but also need ones specific to the machine architecture. Per [1] I started trying to do this with recipe-space features, but this isn't going to fly: the list of features belongs in the scc file, and having to put them in KERNEL_FEATURES in the recipe instead is a maintenance headache. So instead I created branches in a local Linux kernel repository. I named the meta branch yocto-3.10/meta-gumstix instead of meta because I use this repository for other versions of Linux as well, am using yocto-3.10/meta as the local branch that tracks meta from the Yocto yocto-3.10 upstream repository, and want to distinguish it from a version that adds gumstix metadata. (If that all makes sense.) I then use: KMETA_overo = "yocto-3.10/gumstix-meta" in my bbappend, and provide a SRC_URI that fetches from my local repository. This breaks horribly, because the kgit scripts use KMETA for both the branch name and the name of the root directory within that branch. This is most obvious in kgit-checkpoint which ends with: mv $checkpoint_branch .$checkpoint_branch when creating the hidden .meta directory, and in kgit-meta which whines: meta_dir_name=$KMETA # intervening attempts to find $meta_dir_name in file system echo "ERROR cannot locate '$meta_dir_name' directory" I've given up and named the branch "meta" so the scripts behave and I can get on with my work, but I think the proper solution is to stop using KMETA and other machinations to identify a directory name, and instead require that metadata be in a constantly named source root directory "meta" (in branch ${KMETA}) or ".meta" (in branch ${KBRANCH} when ${KMETA} is empty). [1] http://article.gmane.org/gmane.comp.handhelds.openembedded.core/41298/ 36102 1 bruce.ashfield 2013-08-30 16:00:59 +0000 The actual problem is a lack of documentation around changing the meta branch, it was supposed to be done in 1.4, but I don't recall it making the release and post 3.10 kernel introduction, updated docs are pending since the workflow changed around in tree meta data. It has surprised several people, and surprises .. are not good. Docs will help, but also making things a bit smarter will help more. We can't fix the directory name, since there are users and use cases that actually do need several meta data branches and hence, non conflicting directories. In the distant past, special tags, hints or commits were all tried in the tree to indicate the directory / branch combinations, and they all failed as "too clunky" or "not obvious". The current approach is a compromise between usability and flexibility. That being said, I have the M5 milestone to see about making things a bit smarter using your use case as an example. 36104 2 pab 2013-08-30 16:27:20 +0000 Thanks. This is just the workaround. My use case is better served by the ability to support a PATH concept for locating features regardless of whether they come from a kernel source (KBRANCH) directory, an in-repository (KMETA) branch patching a directory into the kernel sources, or files provided in SRC_URI from recipe-space. Ideally all three should be permitted, with increasing priority. What I really want is a BSP scc that can live in recipe space and re-use the fragments from the meta branch through include directives rather than KERNEL_FEATURES. This got messy because I apparently couldn't have a bsp file recipe space that could locate metadata from the standard hierarchy: it seems that anything read from recipe space can't see the features that are elsewhere. For example, one thing I need to handle is that features/media/media-camera.cfg specifies CONFIG_MEDIA_SUBDRV_AUTOSELECT=y which disables visibility of the I2C camera I need to support. To work around this I need to replicate features/media/media-camera.scc and everything it recursively includes, instead of just overriding the one file that's wrong. Perhaps that should be a different bug/enhancement request. I'm also finding "in-tree" to be incomplete: it seems to cover both the presence of a kernel tree metadata directory in KBRANCH as well as a distinct KMETA branch which gets copied into the kernel tree during the patch process, which I think are conceptually different approaches. 36109 3 pab 2013-08-30 17:04:12 +0000 (In reply to comment #2) > This is just the workaround. My use case is better served by the ability to > support a PATH concept for locating features regardless of whether they come > from a kernel source (KBRANCH) directory, an in-repository (KMETA) branch > patching a directory into the kernel sources, or files provided in SRC_URI > from recipe-space. Ideally all three should be permitted, with increasing > priority. It would also be good, if backwards incompatible, to eliminate any automatic recursive searching for metadata files. It's confusing to have my bsp scc do: include features/media/v4l2.scc which in turn does: kconf v4l2.cfg which manages to find features/media/v4l2.cfg when there could be any number of specialized v4l2.cfg files that would be more appropriate. I think putting everything in recipe space is going to turn out to be easier until all this gets a little more orthogonal. 36110 4 dvhart 2013-08-30 17:09:43 +0000 Peter, lots of good ideas here. We really appreciate the feedback. I think I see at least 3 bugs here. The original, the recipe-space-descriptions-can't-see-kernel-space-meta-data, and the one in comment 3 about include/kconf paths. Some of this is likely just a doc fix to the kernel-dev manual. Others could likely be improved in the tooling. Would you please open a bug for each specific issue? Also, as a matter of process, it would have probably been best to have this discussion on the linux-yocto mailing list first, and then opened the bugs as they became well defined. Again, thank you for taking the time to report the issues. 36111 5 bruce.ashfield 2013-08-30 17:11:05 +0000 (In reply to comment #3) > (In reply to comment #2) > > This is just the workaround. My use case is better served by the ability to > > support a PATH concept for locating features regardless of whether they come > > from a kernel source (KBRANCH) directory, an in-repository (KMETA) branch > > patching a directory into the kernel sources, or files provided in SRC_URI > > from recipe-space. Ideally all three should be permitted, with increasing > > priority. > > It would also be good, if backwards incompatible, to eliminate any automatic > recursive searching for metadata files. It's confusing to have my bsp scc > do: > > include features/media/v4l2.scc > > which in turn does: > > kconf v4l2.cfg > > which manages to find features/media/v4l2.cfg when there could be any number > of specialized v4l2.cfg files that would be more appropriate. The searching is required for backwards compatibility, but non-relative path based configs and patches have been discouraged for some time .. some do slip in, but in general things are better. Note that "kconf v4l2.cfg", really isn't searching. The search path is the current directory and then the known kernel-cache directories. So in this case, it simply looks in the current dir, matches and will go no further. > > I think putting everything in recipe space is going to turn out to be easier > until all this gets a little more orthogonal. 36112 6 bruce.ashfield 2013-08-30 17:12:12 +0000 (In reply to comment #4) > Peter, lots of good ideas here. We really appreciate the feedback. I think I > see at least 3 bugs here. The original, the > recipe-space-descriptions-can't-see-kernel-space-meta-data, and the one in recipe space features are already tracked, and should work, I just haven't gotten to replying to it yet :) > comment 3 about include/kconf paths. Some of this is likely just a doc fix > to the kernel-dev manual. Others could likely be improved in the tooling. > > Would you please open a bug for each specific issue? > > Also, as a matter of process, it would have probably been best to have this > discussion on the linux-yocto mailing list first, and then opened the bugs > as they became well defined. Again, thank you for taking the time to report > the issues. 36113 7 bruce.ashfield 2013-08-30 17:19:39 +0000 (In reply to comment #2) > Thanks. > > This is just the workaround. My use case is better served by the ability to > support a PATH concept for locating features regardless of whether they come > from a kernel source (KBRANCH) directory, an in-repository (KMETA) branch > patching a directory into the kernel sources, or files provided in SRC_URI > from recipe-space. Ideally all three should be permitted, with increasing > priority. Most of this is already in place, since the yocto 1.4 release. Out of tree KERNEL_FEATURES are possible, and out of tree .scc files can use in tree features. You can even specify entire kernel-cache repositories on the SRC_URI and use them as KERNEL_FEATURES. There's a bitbake issue that needs to be resolved around directory handling to allow entire directories to be specified in userspace as feature dirs, but that's a minor issue at the moment, so it's on the back burner. > > What I really want is a BSP scc that can live in recipe space and re-use the > fragments from the meta branch through include directives rather than > KERNEL_FEATURES. This got messy because I apparently couldn't have a bsp > file recipe space that could locate metadata from the standard hierarchy: it > seems that anything read from recipe space can't see the features that are > elsewhere. This should work, the revs of the tools may have introduced a regression. Each kernel-cache, and recipe space feature is included on a search path. You can specify a .scc file relative to the root of any search dir and it should be found. Just like gcc or cpp do. If you have an example of a broken out of tree .scc file, attach it and I'll fix the issue. > > For example, one thing I need to handle is that > features/media/media-camera.cfg specifies CONFIG_MEDIA_SUBDRV_AUTOSELECT=y > which disables visibility of the I2C camera I need to support. To work > around this I need to replicate features/media/media-camera.scc and > everything it recursively includes, instead of just overriding the one file > that's wrong. Again, this should be working. You should be able to include the feature and override what you need. In 1.6, there will be some other options to include and override automatically, but I won't get into that now. > > Perhaps that should be a different bug/enhancement request. > > I'm also finding "in-tree" to be incomplete: it seems to cover both the > presence of a kernel tree metadata directory in KBRANCH as well as a > distinct KMETA branch which gets copied into the kernel tree during the > patch process, which I think are conceptually different approaches. I'm not following this. The meta branch is a convenience container for kernel meta data. Since it follows the kernel, you can do standalone builds without a build system and there is no branching/revision management issues. The contents of the meta branch are simply untracked in the BSP branches so they'll be globally visible to the tools, regardless of what branch they are on. So yes, a build is a combination of a board branch KBRANCH, and a meta data branch KMETA. 36114 8 pab 2013-08-30 17:23:23 +0000 (In reply to comment #4) > Also, as a matter of process, it would have probably been best to have this > discussion on the linux-yocto mailing list first, and then opened the bugs > as they became well defined. Again, thank you for taking the time to report > the issues. Originally this was just a bug that KMETA was being mis-used. It's blown up beyond that, and I agree the whole thing should be aired in linux-yocto. I'm writing up an intro mail and will move discussion there. 36115 9 bruce.ashfield 2013-08-30 17:28:51 +0000 (In reply to comment #8) > (In reply to comment #4) > > Also, as a matter of process, it would have probably been best to have this > > discussion on the linux-yocto mailing list first, and then opened the bugs > > as they became well defined. Again, thank you for taking the time to report > > the issues. > > Originally this was just a bug that KMETA was being mis-used. It's blown up > beyond that, and I agree the whole thing should be aired in linux-yocto. > I'm writing up an intro mail and will move discussion there. Take into account my recent updates please .. otherwise, I'll be forced to spend time writing up the same answers .. instead of working on the code :) 36116 10 pab 2013-08-30 19:34:03 +0000 (In reply to comment #4) > Peter, lots of good ideas here. We really appreciate the feedback. I think I > see at least 3 bugs here. The original, Yes: exactly how KMETA defines the directory name that's created in the kernel source tree should be documented. > the > recipe-space-descriptions-can't-see-kernel-space-meta-data, I've confirmed this seems fixed in master; I'd encountered it in dylan originally. > and the one in comment 3 about include/kconf paths. I'm not sure yet whether this is a bug; it's the focus of the linux-yocto email I've just sent. 37756 11 bruce.ashfield 2013-10-09 00:21:22 +0000 Moving this to 1.6, where I can address the flexibility and do documentation updates at the same time. 44661 12 sgw 2014-07-24 14:56:10 +0000 Bruce, can you update this this, or close if the changes are in 44728 13 bruce.ashfield 2014-07-25 19:00:13 +0000 Still pending. I updated the release milestone. 45847 14 bruce.ashfield 2014-09-25 15:11:53 +0000 This has been fixed the the tree. meta branches and the containing directories can now be different.