| Summary: | Can I customize the YP kernel headers? | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Robert Berger <pokylinux> |
| Component: | kernel | Assignee: | Bruce Ashfield <bruce.ashfield> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | randy.macleod, tom.zanussi |
| Version: | 3.1 | ||
| Target Milestone: | 3.3 M3 | ||
| Hardware: | All | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Yes (doc changes required) | |
I forgot another use case: Building kernel modules with the SDK without git cloning the kernel sources, but providing what's needed from the SDK. e.g. TOOLCHAIN_TARGET_TASK_append = " kernel-devsrc" (In reply to comment #0) > I think there is work in progress and probably done, but I wanted to sum my > take on kernel headers and OE/YP. Maybe this bug needs to be split into > several bugs. > > The UAPI between kernel and user space is pretty stable, but e.g. in case > someone hacks a kernel module e.g. adding an ioctl this is needed in user > space to access the kmod new functionality. > > kernel headers in user space > ============================ > > It it not quite clear to me how this is done properly. It might be just a > matter of documentation. I am happy to test and document as time permits. Yes, this is something that is already possible, and it involves the staging of your headers to a location that the user application can find. There's an open bug and a patch there for how to provide those headers in a slightly more streamlined way, but it already works now. > > The linux-libc-headers used to build the SDK do not change if you build > another kernel version or modify the kernel, which should be like this by > design, as far as I understand, since we don't want to rebuild the toolchain > each time something changes in the kernel. Yes, this is by design. We absolutely do not modify the libc-headers for each and every kernel that is built. It is only for gcc/glibc, something that needs the up to date headers, either does a custom export or should be looking at the staging directories. > > We have those use cases: > > ----------------------------------------------------------------------- > | | headers provided | SDK used but | > | cooker | by SDK | full sources avail | > --------------------|--------|-------------------|--------------------| > in tree kmod | 1a) | 1b) | works | > --------------------|--------|-------------------|--------------------| > in tree modif.kmod | 2a) | 2b) | works | > (e.g. added ioctl) | | | | > --------------------|--------|-------------------|--------------------| > out of tree kmod | 3a) | 3b) | works | > --------------------|--------|-------------------|--------------------| > > 1a) we use a newer kernel than what linux-libc-headers provides and would > like to use one of the new system calls/ioctls from user space You must export those yourself, or write your own linux-libc-headers recipe. But that recipe for the newer libc-headers is your responsibility and can be done in your own layers. > > 1b) as above, but instead of cooker mode we want to give an SDK to the > developer to use new system cals/iotcs from user space - I guess we'll need > to rebuild the SDK/toolchain with new linux-libc-headers here Yes. Or you include the exported, newer headers in the SDK. > > 2a) a patch modifies some kernel module and add an ioctl, we would like to > use this from a user space app in cooker mode > > something like that with magic?: > > do_configure[depends] += "virtual/kernel:do_compile_kernelmodules" You must go looking at the kernel staging sources for this. This is absolutely not a case where new headers should be exported. If this is being done via uapi, it is wrong. > > 2b) we would like to expose the new ioctl to user space without touching > linux-libc-headers headers and without rebuilding toolchain/SDK > > I guess we need some custom dir for the new headers. Yes, and there's already ways to do this, and bugs tracking it. > > 3a) we build an out of tree kmod with new ioctls and want to use this from > another user space app in cooker mode > something like that with magic?: > do_configure[depends] += "virtual/kernel:do_compile_kernelmodules" This is no different than if the new calls are in the main kernel tree, you need to export them to a known location, have the application that needs them depend on the recipe proving them, and build. > > 3b) We would like the new ioctl somewhere in the SDK without touching > linux-libc-headers headers and without rebuilding toolchain/SDK > > I guess we need some custom dir for the new headers. Yes. The answer to almost every case is: don't even consider libc-headers, just export the kernel header to a place where the application can find it, adjust the include paths of your application and build against it. There's not much that the core kernel infrastructure needs to do for this. > > (internal) kernel headers > ========================= > > can change any time (by design), but might be needed on target when eBPF/BPF > programs are compiled(dynamically)/run. Solution from 5.2 might be > CONFIG_KHEADERS, so they end up in /sys/kernel.kheaders.tar.xz, but this > might not work for cross-compiled stuff. This is already on and available for linux-yocto recipes, but it causes reproducibility issues so is only available via a KERNEL_FEATURE switch at the moment. OK I think I understood the general idea. Let me experiment a bit and see what I can come up with. I have a couple of patches for alternate kernel headers, and one for full kernel source as an option to curated headers. I'm updating them for master, but if something seems missing, we have a some things to try. I have another interesting use case here using the "classic" SDK Say I create with dunfell 3.1.1 a classic SDK. This SDK is built against 5.4 kernel headers by default. Let's say I build a 4.19.128 xenomai patched kernel and 3.1 xenomai user space (which seems to be the latest and greatest xeno). When I run tests now, I get: ./xeno-test ++ echo 0 ++ testdir=/usr/local/xenomai/bin ++ /usr/local/xenomai/bin/smokey --run random_alloc_rounds=64 pattern_check_rounds=64 arith OK ... posix_clock OK posix_cond OK posix_fork OK [ 466.998532] [Xenomai] bad syscall <0x197> posix_mutex OK posix_select OK ... 0x197 (407) is not avail in 4.19[1], but in 5.4[2] [1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm/tools/syscall.tbl?h=v4.19.131#n416 [2] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm/tools/syscall.tbl?h=v5.4.50#n424 I guess the proposed fix for this is *) to expose the 4.19 kernel headers on some non standard (alt) dir in the SDK *) and use those instead of the standard headers to build xenomai user land. Rethinking the above - I guess this might be the case where we'll need a "special SDK built against 4.19 headers". my take on it is: If the kernel you plan to use is the same or newer than the kernel headers which were used to build the toolchain/SDK all is good. If the kernel you plan to use is older there is potentially a serious problem, since system calls can be called from user space/glibc or whatever C lib which are not available in the kernel. which can lead to things like those: [ 251.080655] [Xenomai] bad syscall <0x197> although it's not quite obvious to me who exactly calls 407 common clock_nanosleep_time64 sys_clock_nanosleep it's getting more interesting: bitbake -s | grep headers linux-libc-headers :4.19-r0 nativesdk-linux-libc-headers :4.19-r0 root@multi-v7-ml:~# /usr/bin/smokey --run=16 [ 448.996933] [Xenomai] bad syscall <0x197> posix_mutex OK If you find that the callsite is coming through glibc, then we have to look into why the syscall wrappers aren't picking up the variant and making the proper fallback. If they are direct syscalls, that's a different question and one we can't do much about. about [Xenomai] bad syscall <0x197>: This does not come from the kernel, but from Xenomai/Cobalt! I was digging a bit deeper and figured that a call to __real_usleep()[1] causes the bad syscall. So glibc/linux are happy as far as I can tell. It seems to be a problem between xenomai and a "recent" glibc. [1] https://gitlab.denx.de/Xenomai/xenomai/-/blob/v3.1/testsuite/smokey/posix-mutex/posix-mutex.c#L933 You are off the hook on this one ;) ... But I guess you already smelled something like that coming ;) The ball is now in Xenomai land for this one. I think this can be closed, since the system is behaving as designed. We can re-open it for specific cases as they pop up. I guess I get the idea and kind of agree with Bruce, but I think I miss a huge chunk of documentation which describes how it's supposed to work and how it works. Is there somewhere documentation how the use cases described are supposed to work? I mean working examples and so on? If not I am happy to work (with Bruce?) on the documentation as time permits. As I already indicated in the first post it might just be a matter of documenting things. Just for completeness I add this from the chat here: (16:13:19) derRichard: when i have a custom kernel, with a custom uapi, shouldn't yocto use these kernel headers as glibc kernel-headers? (16:13:36) derRichard: or another way asked, where does the glibc recipe take the kernel uapi headers from? (16:18:03) qschulz: linux-kernel-headers recipe (16:18:16) qschulz: derRichard: kernels are machine specific recipes (16:18:29) derRichard: ahhhh, linux-kernel-headers (16:18:30) derRichard: thx! (16:18:41) derRichard: this is why i miss stuff (16:19:01) qschulz: so making glibc dependent on kernel headers is basically making any package non-reusable by other machines (16:19:13) qschulz: which is extremely inefficient (16:19:59) qschulz: if you want to add an entry to the uapi, you need to create a recipe that install a header file into the sysroot which will be used by the kernel, kernel modules and userspace applications that requires it (16:21:07) derRichard: the problem is less complicated. my kernel is 5.4 but linux-kernel-headers seems to use 5.2 (16:21:14) derRichard: so, i miss new stuff (16:25:28) qschulz: derRichard: makes sense to upgrade linux-kernel-headers recipe AND USE IT FOR ALL MACHINES (16:26:16) qschulz: derRichard: since 5.4 is an LTS, we probably have/had a recipe for that in oe-core/poky (16:27:00) derRichard: thx, this is what i was about to do :) (16:27:51) fray: the Linux-kernel-headers are ONLY used by userspace. The versions and content do NOT have to match the running ekrenl.. (16:28:01) fray: I would NOT upgrade it, as you can break glibc/musl, etc.. (16:28:29) fray: as long as the linux-kernel-headers are the same or older then the running kerenl, it will work fine (16:29:30) fray: people need to stop thinking there is something magic in the linux-kernel-headers.. there isn't, it's just defining the APIs used by the system libc. The Linux kernel folks have promised that they will remain backwards compatible in newer versions of the kernel -- so there is no reason to upgrade it (as a user) even if you are rolling foward your kernel version. (16:29:58) fray: Additionally, programs that are trying to talk to module interfaces should NOT be using it as a foundation for ioctl and similar. As ioctls are not part of the defined glibc mechanisms (16:31:42) fray: (best way to handle this, think of the system as userspace and kernel space. linux-kernel-headers, libc, etc are all userspace. Linux _kernel_ and modules are all kernel space.. When building, one should space should not refer to another. (16:33:46) qschulz: fray: well now the question is... what derRichard Is missing in 5.2 kernel headers (16:34:06) derRichard: since socketcan folks broke the uabi, i need to use 5.4 heards on both sides (16:34:15) derRichard: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f5223e9eee651e005c0f6d6d078909087601b7e9 (16:34:37) derRichard: struct socketaddr_can now has differenct sizes :( (16:34:44) derRichard: (this needs to be fixed anyway) (16:34:50) derRichard: but for now i need to work around that (16:35:36) fray: Any userspace driver that needs specific knowledge of a kernel interface (magic numbers, including ioctl) should carry it's own knowledge. Often this is done with a copy of a specific chunk of the driver's (kernel) header in the userspace driver.. (16:35:50) fray: For well defined interfaces, glibc handles the interactions/translations for you (16:37:20) derRichard: not if due to a kernel bug the uabi was broken (16:37:24) derRichard: ...which is here the case (16:37:29) fray: in this case, I would verify that glibc has no references to can.h (I'm pretty sure it doesn't, as it has no specific knowledge of can inteface, unlike sockets/pipes/tcp/ip.. (16:38:09) fray: Assuming glibc does NOT have direct knowledge of the can.h interfaces, then you can carry a temporary copy of it -- or patch linux-kernel-headers with the one file. Either will work fine.. (16:38:10) ***derRichard checked (16:38:31) fray: This is a bit of an unusual situation as it should be a "more" stable interface.. (16:39:01) fray: where I usually see this is ioctls and other 'magic numbers' that userspace programs are trying to get access. In those cases, the right answer is always to include a copy of the magic numbers within the userspce component.. (16:39:09) JPEW: Ya, that one is a little annoying because it's a change in the socket address type (16:39:36) derRichard: i found that an hour ago, will talk later to socket can folks. IMHO this is a bad bug (16:39:53) derRichard: but for now i need to get my application stack in shape again (16:39:56) fray: since they placed it at the end of the structure, theoretically it's compatible.. but ya.. iffy.. (16:40:13) fray: I'd almost consider defining a new structure specifically to resolve this issue.. i.e. can_new.. can_2.. etc etc etc (16:40:20) fray: breaking an existing interface is not a good idea (16:40:27) JPEW: derRichard: Is the problem that the program is receiving the address into a `struct sockaddr_can` struct? (16:40:29) fray: (even for a less used interface like CAN) (16:40:56) derRichard: JPEW: yes. i use recvfrom() on AF_CAN (16:41:30) JPEW: Ya, the recieve buffer should be something else... "struct sockaddr" maybe? That ensures it's big enough and you typecast it to the specific type (I think) (16:41:56) fray: (for a VERY long time, CAN required a local to the app copy of the can headers.. just as an FYI..) (16:42:57) derRichard: JPEW: let me check that the application really does (it was not written by me) (16:43:04) derRichard: *what the (16:47:57) JPEW: derRichard: Ah, how unfortunate: it looks like the struct sockaddr only reserves 14 bytes for sa_data, but the sockaddr_can has 17 data bytes, so even that wouldn't fix it :( (16:48:46) JPEW: derRichard: Oh, right, use "struct sockaddr_storage". It's big enough :) (16:50:31) alexlarsson left the room (quit: Remote host closed the connection). (16:53:00) derRichard: qschulz: JPEW: should i add a recipes-kernel/linux-libc-headers/ recipe to my layer? (16:53:18) derRichard: i'm not sure that the best way is (16:53:33) derRichard: i don't really want to copy linux-libc-headers.inc into my layer (16:54:40) JPEW: derRichard: No, fray is correct. If your application needs specific kernel API, it should have it's own copy of the header... a lot of libraries that wrap kernel functionality do this (e.g. libdrm) (16:55:45) derRichard: JPEW: and what about can-utils? they use can too with recvfrom() (16:55:47) derRichard: it will break too (16:56:35) JPEW: Ya, that's a bug in can-utils (16:56:49) JPEW: They should be using struct sockaddr_storage (16:58:00) derRichard: just checked, they have their own copy of can.h. so, they are safe (16:58:20) JPEW: derRichard: They still should be using struct sockaddr_storage (16:58:33) JPEW: Otherwise, they have to be kept in lock-step with the kernel (16:59:24) JPEW: Well, maybe not since they pass their own size I guess (16:59:58) derRichard: and why is poky even upgrading thir recipes-kernel/linux-libc-headers/? (17:00:21) derRichard: you say every application has to ship their own headers, which makes only kind of sense (17:01:27) RP: derRichard: linux-libc-headers is the copy of the headers for libc (17:02:14) derRichard: RP: yes, these headers will then be available in sdks, right? (17:02:28) RP: derRichard: since the sdk contains libc, yes (17:02:42) derRichard: i'm using kernel 5.4 and want the headers from 5.4 in my sdk, and not 5.2. :) (17:02:56) derRichard: IIUC i need to move linux-libc-headers to 5.4 then (17:03:51) RP: derRichard: correct. But just use a plain upstream kernel for linux-libc-headers, not some machine hacked up one (17:04:01) derRichard: of course, yes (17:04:07) derRichard: in my case a plain kernel will do it (17:04:27) RP: derRichard: if a plain kernel wouldn't do it, there would be an issue somewhere (17:05:02) derRichard: agreed (17:05:31) derRichard: so i'll have my own linux-libc-headers in my layer with 5.4 upstream such that it matches my upstream 5.4 kerel (17:05:34) derRichard: *kernel (17:11:34) rber@freenode: @derRichard: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13955 |
I think there is work in progress and probably done, but I wanted to sum my take on kernel headers and OE/YP. Maybe this bug needs to be split into several bugs. The UAPI between kernel and user space is pretty stable, but e.g. in case someone hacks a kernel module e.g. adding an ioctl this is needed in user space to access the kmod new functionality. kernel headers in user space ============================ It it not quite clear to me how this is done properly. It might be just a matter of documentation. I am happy to test and document as time permits. The linux-libc-headers used to build the SDK do not change if you build another kernel version or modify the kernel, which should be like this by design, as far as I understand, since we don't want to rebuild the toolchain each time something changes in the kernel. We have those use cases: ----------------------------------------------------------------------- | | headers provided | SDK used but | | cooker | by SDK | full sources avail | --------------------|--------|-------------------|--------------------| in tree kmod | 1a) | 1b) | works | --------------------|--------|-------------------|--------------------| in tree modif.kmod | 2a) | 2b) | works | (e.g. added ioctl) | | | | --------------------|--------|-------------------|--------------------| out of tree kmod | 3a) | 3b) | works | --------------------|--------|-------------------|--------------------| 1a) we use a newer kernel than what linux-libc-headers provides and would like to use one of the new system calls/ioctls from user space 1b) as above, but instead of cooker mode we want to give an SDK to the developer to use new system cals/iotcs from user space - I guess we'll need to rebuild the SDK/toolchain with new linux-libc-headers here 2a) a patch modifies some kernel module and add an ioctl, we would like to use this from a user space app in cooker mode something like that with magic?: do_configure[depends] += "virtual/kernel:do_compile_kernelmodules" 2b) we would like to expose the new ioctl to user space without touching linux-libc-headers headers and without rebuilding toolchain/SDK I guess we need some custom dir for the new headers. 3a) we build an out of tree kmod with new ioctls and want to use this from another user space app in cooker mode something like that with magic?: do_configure[depends] += "virtual/kernel:do_compile_kernelmodules" 3b) We would like the new ioctl somewhere in the SDK without touching linux-libc-headers headers and without rebuilding toolchain/SDK I guess we need some custom dir for the new headers. (internal) kernel headers ========================= can change any time (by design), but might be needed on target when eBPF/BPF programs are compiled(dynamically)/run. Solution from 5.2 might be CONFIG_KHEADERS, so they end up in /sys/kernel.kheaders.tar.xz, but this might not work for cross-compiled stuff.