We noticed this selftest failure: http://autobuilder.yocto.io:8010/builders/nightly-oe-selftest/builds/35/steps/Running%20oe-selftest/logs/stdio ====================================================================== FAIL: test_devtool_add_git_local (oeqa.selftest.devtool.DevtoolTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/decorators.py", line 109, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/devtool.py", line 224, in test_devtool_add_git_local result = runCmd('devtool add %s' % srcdir) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py", line 114, in runCmd raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, result.output)) AssertionError: Command 'devtool add /tmp/devtoolqaf_idqkoz/noname' returned non-zero exit status 1: NOTE: Creating workspace layer in /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/workspace NOTE: Enabling workspace layer in bblayers.conf ERROR: Command 'recipetool --color=auto create -o /tmp/devtoolhnpfmrpf "/tmp/devtoolqaf_idqkoz/noname" ' failed: Traceback (most recent call last): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/fetch2/__init__.py", line 1483, in latest_revision return revs[key] File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/persist_data.py", line 87, in __getitem__ raise KeyError(key) KeyError: 'git:git.yoctoproject.org.git.dbus-waitmaster-defaultpkgname' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/recipetool", line 120, in <module> ret = main() File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/recipetool", line 109, in main ret = args.func(args) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/recipetool/create.py", line 577, in create_recipe if srcuri and supports_srcrev(srcuri): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/recipetool/create.py", line 317, in supports_srcrev fetcher = bb.fetch2.Fetch([uri], localdata) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/fetch2/__init__.py", line 1516, in __init__ self.ud[url] = FetchData(url, d, localonly) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/fetch2/__init__.py", line 1201, in __init__ self.method.urldata_init(self, d) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/fetch2/git.py", line 189, in urldata_init ud.setup_revisons(d) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/fetch2/__init__.py", line 1231, in setup_revisons self.revisions[name] = srcrev_internal_helper(self, d, name) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/fetch2/__init__.py", line 1096, in srcrev_internal_helper srcrev = ud.method.latest_revision(ud, d, name) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/fetch2/__init__.py", line 1485, in latest_revision revs[key] = rev = self._latest_revision(ud, d, name) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/fetch2/git.py", line 397, in _latest_revision output = self._lsremote(ud, d, "") File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/fetch2/git.py", line 388, in _lsremote output = runfetchcmd(cmd, d, True) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/fetch2/__init__.py", line 845, in runfetchcmd raise FetchError(error_message) bb.fetch2.FetchError: Fetcher failure: Fetch command export PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots/x86_64-linux/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots/qemux86-64/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots/x86_64-linux/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots/x86_64-linux/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-autobuilder/bin:/home/pokybuild/yocto-autobuilder/yocto-autobuilder/scripts:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pokybuild/.local/bin:/home/pokybuild/bin"; export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0 ls-remote https://git.yoctoproject.org/git/dbus-wait failed with exit code 128, output: fatal: unable to access 'https://git.yoctoproject.org/git/dbus-wait/': A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. ---------------------------------------------------------------------- Ran 200 tests in 15243.410s FAILED (failures=1) which comes from a failure of the "git -c core.fsyncobjectfiles=0 ls-remote https://git.yoctoproject.org/git/dbus-wait" command. This is reproducible outside the autobuilder itself on the fedora24 and feora23 workers: [pokybuild@fedora24 build]$ while true; do git -c core.fsyncobjectfiles=0 ls-remote https://git.yoctoproject.org/git/dbus-wait | grep -v 6cc6; done ^[[Bfatal: unable to access 'https://git.yoctoproject.org/git/dbus-wait/': A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. fatal: unable to access 'https://git.yoctoproject.org/git/dbus-wait/': A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. fatal: unable to access 'https://git.yoctoproject.org/git/dbus-wait/': A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. fatal: unable to access 'https://git.yoctoproject.org/git/dbus-wait/': A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. fatal: unable to access 'https://git.yoctoproject.org/git/dbus-wait/': A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. fatal: unable to access 'https://git.yoctoproject.org/git/dbus-wait/': A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. so it seems there is an intermittent certificate/network glitch specific to fedora. I couldn't reproduce this on the opensuse132, centos7, ubuntu1604 or debian8 workers, only fedora23 and fedora24. Paul Eggleton reproduced this on his remote fedora24 system which rules out autobuilder networking so its looking like a fedora specific problem.
https://bugzilla.redhat.com/show_bug.cgi?id=1317691 seems potentially related. The export variable mentioned there doesn't help.
(In reply to comment #1) > https://bugzilla.redhat.com/show_bug.cgi?id=1317691 seems potentially > related. The export variable mentioned there doesn't help. Just to confirm RP observation, the export mentioned at [1] did not help: $ export NSS_STRICT_NOFORK=DISABLED $ while true; do git -c core.fsyncobjectfiles=0 ls-remote https://git.yoctoproject.org/git/dbus-wait | grep -v 6cc6; done fatal: unable to access 'https://git.yoctoproject.org/git/dbus-wait/': A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. This is on a F24 machine. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1317691#c2
(In reply to comment #1) > https://bugzilla.redhat.com/show_bug.cgi?id=1317691 seems potentially > related. The export variable mentioned there doesn't help. Richard, this seems to be a problem with the distro where building is done, in this case Fedora. There is a proposed fix [1], but in case this solves the problem (which does not seem to fix all scenarios [2], except the sample case) on host, how do we handle this case? we can patch the packages we control but of course we can not patch distros. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1317691#c13 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1317691#c16
The proposed patch (https://bugzilla.redhat.com/attachment.cgi?id=1144900&action=diff) did not solve the issue on the AB machine so moving the status to WaitForUpstream. As mentioned, the bug has already being filed on the redhat bugzilla tracker and has not been fixed.
The simplified scenario in our case was reported in the main bug tracker page, giving another simple way to reproduce it. https://bugzilla.redhat.com/show_bug.cgi?id=1317691#c29
Last change from bugzilla's RedHat: https://bugzilla.redhat.com/show_bug.cgi?id=1317691 errata-xmlrpc <errata-xmlrpc@redhat.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|VERIFIED |RELEASE_PENDING
Michael, assigning this bug to you because Red Hat have provided a set of updates that fixes the issue on the fedora(s) builder(s) Please follow the recommendations from: https://bugzilla.redhat.com/show_bug.cgi?id=1317691#c34
Reading the advisories it appears that this issue should be solved with the newest set of Fedora packages. They have been installed so I think this is resolved.
Verified on local autobuilders and failure was not observed Dec 08 14:31 dae00fbb913c... Dec 08 08:38 18e4688bd52e... Dec 07 18:03 e38775a1d82e... Dec 07 12:16 e38775a1d82e... Dec 07 10:04 adb34b8ddcd8... Dec 06 18:18 9e63f81c78e2... Dec 06 12:24 9e63f81c78e2...