13727 2020-01-09 00:01:23 +0000 unable to run dockerd on MACHINE=intel-corei7-64 yet works on MACHINE=genericx86-64 2020-06-30 16:05:20 +0000 1 1 7 Build System, Metadata & Runtime BSPs bsps-meta-intel unspecified x86 x86_64 RESOLVED FIXED Medium+ normal 3.2 M2 1 ksaye ksaye bruce.ashfield randy.macleod tim.orling --- --- No (bug/feature does not impact docs) oldest_to_newest 85989 0 ksaye 2020-01-09 00:01:23 +0000 Yocto version: warrior build OS: Ubuntu 18.04 lts This issue is discussed here: https://github.com/Azure/meta-iotedge/issues/26#issuecomment-572270282 Using the meta-intel layer and the machine=intel-corei7-64, there are features missing in the kernel that limit the ability to start docker. Docker gives the error: pickfirstBalancer: HandleSubConnStateChange: 0xc0007c6b80, CONNECTING module=grpc failed to start daemon: Error initializing network controller: Error creating default "bridge" network: Failed to program NAT chain: Failed to inject DOCKER in PREROUTING chain: iptables failed: iptables --wait -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER: iptables: No chain/target/match by that name. Running the docker check-config.sh script (https://github.com/moby/moby/blob/master/contrib/check-config.sh) it shows several kernel features missing, most specifically: CONFIG_VETH and CONFIG_NETFILTER_XT_MATCH_ADDRTYPE. If I change the machine to genericx86-64, it works fine Here is my full build script: https://github.com/ksaye/IoTDemonstrations/blob/master/batch/run.sh 86017 1 tim.orling 2020-01-10 06:40:53 +0000 This issue is that the kernel recipe is not being bbappended to include ${BPN}_virtualiation.inc See: http://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/tree/recipes-kernel/linux?h=warrior Each linux kernel recipe needs a bbappend and a named/symlinked ${BPN}_virtualization.inc, e.g. linux-intel_birtualization.inc and linux-intel_%.bbappend You can add these to e.g. dynamic-layers/virtualization-layer/recipes-kernel/linux In other words, meta-virtualization is hard coded to only work with linux-yocto at the moment, so it doesn't understand linux-intel which is set by intel-corei7-64 machine. 86020 2 ksaye 2020-01-10 18:42:14 +0000 Tim, Thanks for the investigation. I did a little testing and the following copy commands do resolve the issue, meaing that your assessment is correct. cp meta-virtualization/recipes-kernel/linux/linux-yocto_4.19.bbappend meta-virtualization/recipes-kernel/linux/linux-intel_4.19.bbappend cp meta-virtualization/recipes-kernel/linux/linux-yocto_virtualization.inc meta-virtualization/recipes-kernel/linux/linux-intel_virtualization.inc While this works for me and unblocks me today, what do you recommend for a larger fix? Kevin 86022 3 tim.orling 2020-01-10 21:20:47 +0000 We are coincidentally working on this internally, hence the solution was already known (but not public yet). I see two possible paths forward: (1) meta-intel adds the necessary bbappends and incs to dynamic-layers (this is up to Anuj to decide as he is the final word on meta-intel). {2) meta-virtualization adds the necessary linux-intel files (this is up to Bruce Ashfield as he is the final word on meta-virtualization). There is probably a third better option than makes the meta-virtualization bbappends more universally applicable. We should start discussion on the meta-virtualization mailing list [1]. There might even be an opportunity to move this to oe-core, but regardless the discussion belongs on the meta-virtualization mailing list. Backporting to "warrior" might be contentious, so just a heads up that you may need to carry technical debt until you move to 3.1 release (currently "master"). I will assume you have a business need to be on "warrior", rather than ask you why you can't move to a newer release. [1] https://lists.yoctoproject.org/g/meta-virtualization 86023 4 bruce.ashfield 2020-01-10 21:29:09 +0000 Since the linux-intel recipes are good citizens and support fragments, we can make the bappends more generic. If other users of meta-virt have kernel recipes that match, and don't have fragment support .. they won't be harmed either (since they'll just be ignored). So something like linux-%_<version>.bbappend in meta-virt should meet the need (I *think* the wildcard will work like that in the middle, otherwise, even linux-%.bbappend could be considered). (In reply to comment #3) > We are coincidentally working on this internally, hence the solution was > already known (but not public yet). > > I see two possible paths forward: > > (1) meta-intel adds the necessary bbappends and incs to dynamic-layers (this > is up to Anuj to decide as he is the final word on meta-intel). > {2) meta-virtualization adds the necessary linux-intel files (this is up to > Bruce Ashfield as he is the final word on meta-virtualization). > > There is probably a third better option than makes the meta-virtualization > bbappends more universally applicable. We should start discussion on the > meta-virtualization mailing list [1]. There might even be an opportunity to > move this to oe-core, but regardless the discussion belongs on the > meta-virtualization mailing list. > > Backporting to "warrior" might be contentious, so just a heads up that you > may need to carry technical debt until you move to 3.1 release (currently > "master"). I will assume you have a business need to be on "warrior", rather > than ask you why you can't move to a newer release. > > [1] https://lists.yoctoproject.org/g/meta-virtualization 86602 5 bruce.ashfield 2020-02-26 22:04:53 +0000 I've pushed a patch to meta-virtualization that should resolve this. If anyone is available to quickly test it, that would be helpful. I'd rather not close this until we get confirmation. 86616 6 ksaye 2020-02-28 00:01:28 +0000 I am testing the patch now. 86617 7 ksaye 2020-02-28 00:47:06 +0000 It still seems to fail on me. My source is here: https://github.com/ksaye/IoTDemonstrations/blob/master/batch/inteltest.sh If I comment out lines 56 and 57, it still fails. Am i missing something? K 86619 8 bruce.ashfield 2020-02-28 04:15:07 +0000 I'll have a look at linux-intel and run some tests. 86626 9 bruce.ashfield 2020-02-28 21:22:16 +0000 I can confirm that linux-intel does now use the fragments from meta-virt: bitbake -e shows: SRC_URI=" git://github.com/intel/linux-intel-lts.git;protocol=https;name=machine;branch=5.4/yocto; git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=kernel-meta file://perf-fix-build-with-binutils.patch file://xt-checksum.scc file://ebtables.scc file://vswitch.scc file://lxc.scc file://docker.scc file://0001-menuconfig-mconf-cfg-Allow-specification-of-ncurses-.patch" If someone can let me know hot to boot the image in qemu (I'm failing on systemd startup), I can also do a docker runtime test. 86628 10 bruce.ashfield 2020-02-28 21:34:06 +0000 Cancel that. I hacked the fstab and was able to boot: root@intel-corei7-64:~# docker pull busybox Using default tag: latest latest: Pulling from library/busybox bdbbaa22dec6: Pull complete Digest: sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a Status: Downloaded newer image for busybox:latest docker.io/library/busybox:latest root@intel-corei7-64:~# docker run -it busybox /bin/sh docker0: port 1(veth1a3a75a) entered blocking state docker0: port 1(veth1a3a75a) entered disabled state device veth1a3a75a entered promiscuous mode eth0: renamed from vethd917b53 IPv6: ADDRCONF(NETDEV_CHANGE): veth1a3a75a: link becomes ready docker0: port 1(veth1a3a75a) entered blocking state docker0: port 1(veth1a3a75a) entered forwarding state IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready / # docker0: port 1(veth1a3a75a) entered disabled state vethd917b53: renamed from eth0 docker0: port 1(veth1a3a75a) entered disabled state device veth1a3a75a left promiscuous mode docker0: port 1(veth1a3a75a) entered disabled state root@intel-corei7-64:~# uname -a Linux intel-corei7-64 5.4.15-intel-pk-standard #1 SMP PREEMPT Thu Feb 6 03:40:33 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux root@intel-corei7-64:~# 86643 11 ksaye 2020-03-02 21:04:28 +0000 Bruce, You seem to be on the 5.4.15 kernel where I am on: Linux intel-corei7-64 4.19.80-intel-pk-standard #1 SMP PREEMPT Fri Feb 28 00:34:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Note my work around copy commands copy to linux-intel_4.19.bbappend Kevin 86644 12 bruce.ashfield 2020-03-02 21:16:18 +0000 I realize I'm on 5.4, I'm only working on master for these changes. Testing other branches and requesting a backport needs to be done by someone with the relevant test capabilities. 86654 13 chee.yang.lee 2020-03-04 06:58:58 +0000 (In reply to comment #11) > Bruce, > You seem to be on the 5.4.15 kernel where I am on: > > Linux intel-corei7-64 4.19.80-intel-pk-standard #1 SMP PREEMPT Fri Feb 28 > 00:34:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux > > Note my work around copy commands copy to linux-intel_4.19.bbappend > > Kevin meta-intel master just added bbappend for linux-intel 4.19 http://git.yoctoproject.org/cgit/cgit.cgi/meta-intel/commit/?id=108c6938a9b8a71b94d2e8d368c2f39e6ec20530 can you try that? 86658 14 ksaye 2020-03-04 17:35:02 +0000 I have tried that and it did not resolve it. 86815 15 chee.yang.lee 2020-03-26 04:47:13 +0000 (In reply to comment #14) > I have tried that and it did not resolve it. i build it this way and it works. can you try that ? https://github.com/cheeyanglee/IoTDemonstrations/commit/d9aef05e9772471e2766236a36753e24c2f6ea6e#diff-f5dc6c0071ef0304136c91697342c0ef 87542 16 tim.orling 2020-06-22 18:30:01 +0000 Kevin, it looks like we are waiting for confirmation from you of the fix. Can you please test and leave comments on whether it meets your needs? 87632 17 ksaye 2020-06-26 13:01:11 +0000 (In reply to comment #16) > Kevin, it looks like we are waiting for confirmation from you of the fix. > Can you please test and leave comments on whether it meets your needs? Thanks for the reminder, should have this tested by Tuesday 6/30. 87664 18 ksaye 2020-06-30 16:05:20 +0000 All, I just verified that the fix works as planned. Thanks all for the resolution! Kevin