Bug 13360

current status: rebasing patches to master, and performing multi-arch tests. Some symlinks, etc, need to be moved to post install scripts as they conflict and prevent some packages from being installed at the same time.

FYI: The new reference headers and source packages are following the debian/ubuntu naming strategy of: linux-headers-<version> and linux-source-<version>

The only real difference between linux-headers-<version> and kernel-devsrc is that there are extra headers. So it is essential a super-set of kernel-devsrc. This is due to the fact that to actually make the headers useful, you'd want to build against them. kernel-devsrc already has minimal headers + everything we need to do this. The request for linux-headers is to grab more "what if" header files. So it is a larger package and works similarly to the debian version.

The details are:

The linux-source-<version> package install to /usr/src/linux-source-<version>
The kernel-headers-<version> package install to /usr/src/linux-headers-<version>

It is also possible that we can make a postinst convenience symlink from /usr/src/linux-source to the currently installed linux-source, and likewise for the linux-headers package.

Hello Bruce,

It seems that my bugzilla entry 13361 is duplicate of 13360. I apologize for that. I did not spot this one, although I searched the entries. My bad.

@13360:

It is simple impossible what you have wrote here. If you use minimalistic kernel with all source code in it, it is impossible that the same header tree would be 2x bigger.

I did explicit investigation about my latest host (HP Elite Book 840-G1) where I have dual-boot system (WIN10 and Fedora 29) F29 kernel, and results are speaking for themselves (obviously opposing your statements).

Here is what my F29 5.0.17 kernel headers are in size (installed):
[user@fedora29-ssd 5.0.17-200.fc29.x86_64]$ pwd
/usr/src/kernels/5.0.17-200.fc29.x86_64
[user@fedora29-ssd 5.0.17-200.fc29.x86_64]$ du --summarize 
102124	.
[user@fedora29-ssd 5.0.17-200.fc29.x86_64]$

And here is what the size of the whole kernel, compressed is:
https://rpmfind.net/linux/RPM/fedora/updates/29/x86_64/Packages/k/kernel-5.0.17-200.fc29.x86_64.html

You can download it, unpack, and see how big it is!
Here is just the basic kernel source in .xz format (I did not include patches and support shell files, after unpacking):
[vuser@fedora29-ssd kernel.fedora]$ pwd
/home/vuser/projects/kernel.fedora
[vuser@fedora29-ssd kernel.fedora]$ ls -al linux-5.0.tar.xz
-rw-rw-r--. 1 vuser vboxusers 105235152 Mar  4 15:57 linux-5.0.tar.xz
[vuser@fedora29-ssd kernel.fedora]$

I unpacked it, to see/experience for myself:
[vuser@fedora29-ssd linux-5.0]$ pwd
/home/vuser/projects/kernel.fedora/linux-5.0
[vuser@fedora29-ssd linux-5.0]$ du --summarize 
955756	.
[vuser@fedora29-ssd linux-5.0]$

I hope this clears the confusion.

Zoran

https://bugzilla.yoctoproject.org/show_bug.cgi?id=13360#c3

Zoran

(In reply to comment #3)
> Hello Bruce,
> 
> It seems that my bugzilla entry 13361 is duplicate of 13360. I apologize for
> that. I did not spot this one, although I searched the entries. My bad.
> 
> @13360:
> 
> It is simple impossible what you have wrote here. If you use minimalistic
> kernel with all source code in it, it is impossible that the same header
> tree would be 2x bigger.

This is *not* what I am saying. The existing kernel-devsrc package is not the entire kernel source tree. It is already a minimal set of headers + build infrastructure for building out of three modules. You cannot get any smaller than it via a headers package.

As such, there's nothing really to do on that front. kernel-devsrc is already the smallest.

This bug will provide two new packages for those that want them: linux-headers (which is slightly larger than kernel-devsrc) and linux-source, which is a compressed capture of the entire kernel source tree (which is large, but is only intended for reference).

Bruce

> 
> I did explicit investigation about my latest host (HP Elite Book 840-G1)
> where I have dual-boot system (WIN10 and Fedora 29) F29 kernel, and results
> are speaking for themselves (obviously opposing your statements).
> 
> Here is what my F29 5.0.17 kernel headers are in size (installed):
> [user@fedora29-ssd 5.0.17-200.fc29.x86_64]$ pwd
> /usr/src/kernels/5.0.17-200.fc29.x86_64
> [user@fedora29-ssd 5.0.17-200.fc29.x86_64]$ du --summarize 
> 102124	.
> [user@fedora29-ssd 5.0.17-200.fc29.x86_64]$
> 
> And here is what the size of the whole kernel, compressed is:
> https://rpmfind.net/linux/RPM/fedora/updates/29/x86_64/Packages/k/kernel-5.0.
> 17-200.fc29.x86_64.html
> 
> You can download it, unpack, and see how big it is!
> Here is just the basic kernel source in .xz format (I did not include
> patches and support shell files, after unpacking):
> [vuser@fedora29-ssd kernel.fedora]$ pwd
> /home/vuser/projects/kernel.fedora
> [vuser@fedora29-ssd kernel.fedora]$ ls -al linux-5.0.tar.xz
> -rw-rw-r--. 1 vuser vboxusers 105235152 Mar  4 15:57 linux-5.0.tar.xz
> [vuser@fedora29-ssd kernel.fedora]$
> 
> I unpacked it, to see/experience for myself:
> [vuser@fedora29-ssd linux-5.0]$ pwd
> /home/vuser/projects/kernel.fedora/linux-5.0
> [vuser@fedora29-ssd linux-5.0]$ du --summarize 
> 955756	.
> [vuser@fedora29-ssd linux-5.0]$
> 
> I hope this clears the confusion.
> 
> Zoran

Logging the latest numbers for reference:

root@qemux86-64:/usr/src# ls -alF
total 16
drwxr-xr-x  4 root root 4096 May 28 15:52 ./
drwxr-xr-x 10 root root 4096 May 28 13:38 ../
lrwxrwxrwx  1 root root   46 May 28 13:38 kernel -> ../../lib/modules/5.0.17-yocto-standard/source/
lrwxrwxrwx  1 root root   35 May 28 15:51 linux-headers -> linux-headers-5.0.17-yocto-standard/
drwxr-xr-x 23 root root 4096 May 28 15:49 linux-headers-5.0.17-yocto-standard/
lrwxrwxrwx  1 root root   34 May 28 15:52 linux-source -> linux-source-5.0.17-yocto-standard/
drwxr-xr-x  2 root root 4096 May 28 15:49 linux-source-5.0.17-yocto-standard/
root@qemux86-64:/usr/src# du -shL kernel
89M     kernel
root@qemux86-64:/usr/src# du -shL linux-headers
114M    linux-headers
root@qemux86-64:/usr/src# du -shL linux-source
123M    linux-source
root@qemux86-64:/usr/src# ls -alF linux-source/
total 125764
drwxr-xr-x 2 root root      4096 May 28 15:49 ./
drwxr-xr-x 4 root root      4096 May 28 15:52 ../
-rw-r--r-- 1 root root 128771907 May 28 13:41 linux-source-5.0.17-yocto-standard.tar.bz2

root@qemux86-64:/usr/src# rpm -qa | grep linux-headers
linux-headers-5.0.17-yocto-standard-1.0-r0.qemux86_64
root@qemux86-64:/usr/src# rpm -qa | grep linux-source 
linux-source-5.0.17-yocto-standard-1.0-r0.qemux86_64

*** Bug 13361 has been marked as a duplicate of this bug. ***

> root@qemux86-64:/usr/src# du -shL kernel
> 89M     kernel
> root@qemux86-64:/usr/src# du -shL linux-headers
> 114M    linux-headers
> root@qemux86-64:/usr/src# du -shL linux-source
> 123M    linux-source
> root@qemux86-64:/usr/src# ls -alF linux-source/
> total 125764
> drwxr-xr-x 2 root root      4096 May 28 15:49 ./
> drwxr-xr-x 4 root root      4096 May 28 15:52 ../
> -rw-r--r-- 1 root root 128771907 May 28 13:41 linux-source-5.0.17-yocto-standard.tar.bz2

Here I assume that the package linux-headers is made out of linux-source (linux-source-5.0.17-yocto-standard.tar.bz2 size 123M), NOT out of minimalistic kernel (how you called it). In this case, everything follows the right pattern.

To be politically correct, you should make another package linux-headers out of your minimalistic kernel, and this (my best guess) will follow the rules, known for so long.

Zoran

Implementation is done for this. 

The patches work, but there's an overhead to do the extra copying in devsrc. I can make the patches available, but really, this should probably be split to avoid the copying unless required. It may not make the release due to that last issue.

Do we have any plan to get this merged to Yocto? Thanks.

(In reply to comment #10)
> Do we have any plan to get this merged to Yocto? Thanks.

I may propose it in the next release cycle, but there are still some changes required.

But it is still a bad idea. Your comment implies that you think you have a valid use case for it, so please, share that here.

Before https://git.openembedded.org/openembedded-core/commit/?id=007ef1f468110b2698a27ea9a6d43fed5a0a9fc2
We were able to build kernel from full source on target and our customers have been using that for further development since.

(In reply to comment #12)
> Before
> https://git.openembedded.org/openembedded-core/commit/
> ?id=007ef1f468110b2698a27ea9a6d43fed5a0a9fc2
> We were able to build kernel from full source on target and our customers
> have been using that for further development since.

That's a very old commit at this point, so what did, or didn't work before isn't really relevant when it wasn't an issue for the past two plus years.

The intention of this package is not to support a full kernel rebuild, it is simply to make the sources available as a reference.

That being said, I'll complete work on this and taint it properly such that it can't be used instead of the devsrc, but instead be a variant of devsrc and it should be acceptable.

> The intention of this package is not to support
> a full kernel rebuild, it is simply to make the
> sources available as a reference.

Wrong reason, if you, Bruce, allow me. The main reason is to allow OOT module support/development on the target platforms, without full kernel source tree support (saving lot of mmc's target space).

REFERENCES:
       Debian: linux-headers packages
       Fedora: kernel-devel packages.

Zoran
_______

(In reply to comment #14)
> > The intention of this package is not to support
> > a full kernel rebuild, it is simply to make the
> > sources available as a reference.
> 
> Wrong reason, if you, Bruce, allow me. The main reason is to allow OOT
> module support/development on the target platforms, without full kernel
> source tree support (saving lot of mmc's target space).
> 
> REFERENCES:
>        Debian: linux-headers packages
>        Fedora: kernel-devel packages.
> 
> Zoran
> _______


No, I meant what I typed. kernel-devsrc already provides that capability. This variant was only added as a potential reference to the entire source tree.

> kernel-devsrc already provides that capability.

So, all other distros (RHEL, Fedora, SuSE, Debian, Ubuntu...) are wrong, taking for the target OOT drivers' compilation ONLY kernel headers approach, NOT the entire kernel source.

This is what you are trying to tell to/convince us, or ... ???

Z

(In reply to comment #16)
> > kernel-devsrc already provides that capability.
> 
> So, all other distros (RHEL, Fedora, SuSE, Debian, Ubuntu...) are wrong,
> taking for the target OOT drivers' compilation ONLY kernel headers approach,
> NOT the entire kernel source.
> 
> This is what you are trying to tell to/convince us, or ... ???
> 
> Z

kernel-devsrc IS the minimal subset of the kernel source for building, so Im not sure what you are trying to say above.

That's not what I'm saying, and I'm aware of what is, or isn't done in other distros, and am not trying to validate or invalidate what they do, nor are we trying to copy what they do.

We HAD full kernel source, it caused problems with I/O, we are using a curated kernel source, similar to what RHEL does (in fact, the basis of the package are file selections from RHEL).

The points are:

  - The history and other practices are understood.
  - We already fully support out of tree module builds
  - I've said that I have a way to tweak the existing package to have a variant and not double the support, and I'll revisit it in the next dev cycle.

> I've said that I have a way to tweak the
> existing package to have a variant and
> not double the support, and I'll revisit
> it in the next dev cycle.

Argument accepted. But before revisiting, please, read the following important remark (Debian build for armv7 A8 on target and tested - prepared on target)

Kernel >= 5.10.0: to prepare out-of-tree device driver compilation in /usr/src/$(uname -r)/

debian@arm:/usr/src/$(uname -r)# make modules_prepare

root@arm:/usr/src/5.10.14-bone22# make modules_prepare
	CALL	scripts/checksyscalls.sh
	CALL	scripts/atomic/check-atomics.sh
	LDS	scripts/module.lds		<<======= NEW ADDITION =======

Please, do note the difference from previous kernels and 5.10.0 onward: LDS scripts/module.lds

The 5.10.0 patch changing the behavior is the following:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/Makefile?id=746b25b1aa0f5736d585728ded70a8141da91edb

Just mentioning, since this change might take you of the guard! ;-)

Zoran

Bulk move of medium importance bugs from 5.1 to 5.2 on behalf of Bruce.

Parked in 6.99