Linux kernel upstream is a single git tree with multiple releases. Even stable tree with updates to various major releases is a single git tree. Yet in yocto we have yocto-linux git trees for each major kernel release. A single Linux kernel git tree size is nowadays around one gigabyte. Please consider hosting all yocto-linux kernels from a single git tree. Do releases from different branches etc. and refer to them in various recipes. This avoids filling up yocto download mirror with multiple gigabyte size kernel source tar balls: http://downloads.yoctoproject.org/mirror/sources/?C=S&O=D git2_git.pokylinux.org.linux-yocto-dev.git.tar.gz 2G 2014-May-22 19:34 git2_github.com.Freescale.linux-fslc.git.tar.gz 1G 2016-Apr-12 10:35 git2_git.yoctoproject.org.linux-yocto-4.1.git.tar.gz 1G 2017-Jul-20 15:11 git2_git.yoctoproject.org.linux-yocto-4.4.git.tar.gz 1G 2017-May-25 23:27 git2_git.freescale.com.imx.linux-2.6-imx.git.tar.gz 1018M 2015-Apr-09 21:10 git2_github.com.Freescale.linux-mainline.git.tar.gz 942M 2014-Jul-09 22:16 git2_git.yoctoproject.org.linux-yocto-3.19.git.tar.gz 933M 2016-Mar-13 19:26 git2_git.yoctoproject.org.linux-yocto-dev.git.tar.gz 892M 2014-Dec-22 20:56 git2_git.yoctoproject.org.linux-yocto-3.17.git.tar.gz 878M 2015-Feb-07 13:38 git2_github.com.Timesys.linux-timesys.git.tar.gz 851M 2015-Dec-16 22:28 git2_git.yoctoproject.org.linux-yocto-3.14.git.tar.gz 835M 2016-Feb-05 00:12 git2_git.yoctoproject.org.linux-yocto-3.10.git.tar.gz 801M 2015-Feb-07 13:54 git2_git.yoctoproject.org.linux-yocto-3.10.tar.gz 799M 2015-Jan-27 04:51 git2_git.freescale.com.ppc.sdk.linux.git.tar.gz 762M 2016-Jan-04 21:02 git2_sourceware.org.git.binutils-gdb.git.tar.gz 692M 2017-Aug-15 12:49 git2_git.yoctoproject.org.linux-yocto-3.4.git.tar.gz 669M 2014-Apr-17 14:41 git2_git.yoctoproject.org.linux-yocto-3.8.tar.gz 650M 2013-Aug-07 00:47 git2_git.yoctoproject.org.linux-yocto-3.8.git.tar.gz 645M 2014-Apr-17 14:49 git2_git.yoctoproject.org.linux-yocto-3.0.tar.gz 627M 2012-Nov-21 17:58 git2_git.yoctoproject.org.linux-yocto-3.2.git.tar.gz 620M 2012-Aug-26 15:04 git2_git.yoctoproject.org.linux-yocto-3.0.tar.gz.1 619M 2012-May-17 16:50 git2_git.yoctoproject.org.linux-yocto-3.4.tar.gz 605M 2012-Dec-19 08:10 git2_git.yoctoproject.org.linux-yocto-3.2.tar.gz 589M 2012-Aug-28 00:15 git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz 564M 2012-May-08 18:09 git2_git.yoctoproject.org.linux-yocto-3.0-1.1.x.git.tar.gz 561M 2012-May-15 23:21 git2_git.yoctoproject.org.linux-yocto-3.0.git.tar.gz 538M 2012-Aug-26 07:38 And users should not need to download them from scratch every time yocto updates to a new major kernel release. Actually, the tar ball caching is not at all efficient for git trees if users already have the previous version, but this is I guess another bug.
This has already been proposed and offered on the mailing list by myself. And has been mentioned more than once in the past. The decided direction was to use shallow clones versus a single tree, since the tree is maintained/managed differently than other trees. Alternatively, having the project host a single pristine korg tree and all other kernels are reference clones against that kernel tree to reduce their size. A few deltas in the process for generating it are as follows: - The tree is rebased and re-created, since we carry content that cannot be easily updated in a fast forward manner. Hence history is re-created on top of each major release for clarity. - The kernel-cache repository tracks the patch history over time, not the actual kernel tree itself. - We won't use quilt or directories full of patches to manage the actual changes to the tree, that was rejected long before the first Yocto release. - We have topic branches for the various invasive features and kernel types (and also to represent BSPs), hence any versioned tracked trees would quickly have 50 or 60 branches and then continue to grow from there. So the actual fix is within bitbake and the options for git fetching, which are being looked at last I heard. Once a reference clone, or shallow clone option are reliably available, I'll adjust the reference kernel recipes to leverage them
I believe shallow clone functionality is in bitbake already thanks for patches from Chris. I did ask for documentation patches, I don't think I ever got any :(. http://git.yoctoproject.org/cgit.cgi/poky/log/?qt=grep&q=shallow shows the patches in question. The other option would be a "clone by reference" setup where we have a common repo for kernels which we then clone the individuals from. That functionality obviously doesn't exist.
Ah yes. I did find the old conversations on this topic, buried in a gcc6 upgrade thread! And one where I offered some options during the release of the 4.8 tree. We talked about shallow clones and references (actually my preference) in those threads. We've had a "linux-yocto" tree on git.yoctoproject.org for nearly a year now, which I asked to be created to work on this in the background. Getting reference clones is obviously more challenging since the code doesn't exist, but during the next release, I'll look at leveraging the shallow clone support and then see what would really be needed for reference clones. (In reply to comment #2) > I believe shallow clone functionality is in bitbake already thanks for > patches from Chris. I did ask for documentation patches, I don't think I > ever got any :(. > > http://git.yoctoproject.org/cgit.cgi/poky/log/?qt=grep&q=shallow shows the > patches in question. > > The other option would be a "clone by reference" setup where we have a > common repo for kernels which we then clone the individuals from. That > functionality obviously doesn't exist.
Please provide an update on the status of this and let me know if it will be ready for YP 2.5 M3.
This has been in place since the 4.8 yocto kernel. I've picked it up for 4.14 and 4.15 as part of the upcoming release.