Bug 11973

Summary: BB_SERVER_TIMEOUT causes testsdkext failures
Product: [Yocto Project Subprojects] eSDK Reporter: Richard Purdie <richard.purdie>
Component: eSDKAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact: Francisco Pedraza <francisco.j.pedraza.gonzalez>
Severity: normal    
Priority: Medium CC: oscarx.lopez.arandas, stephano, wmills
Version: unspecified   
Target Milestone: Future   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
full_log none

Description Richard Purdie 2017-08-24 12:08:18 UTC
If you set BB_SERVER_TIMEOUT=10, build an eSDK (bitbake core-image-minimal -c populate_sdk_ext), then test it (bitbake core-image-minimal -c testsdkext) then it fails. Locally I saw:

$ bitbake core-image-minimal -c testsdkext
NOTE: Reconnecting to bitbake server...
Loading cache: 100% |##########################################################################################################################################| Time: 0:00:00
Loaded 1290 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION        = "1.35.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "universal"
TARGET_SYS        = "mips-poky-linux"
MACHINE           = "qemumips"
DISTRO            = "poky"
DISTRO_VERSION    = "2.3"
TUNE_FEATURES     = "o32 bigendian fpu-hard mips32r2"
TARGET_FPU        = ""
meta              
meta-poky         
meta-yocto-bsp    = "wip:362d6e0490b2ab9a0e9eecbc6c5eb63f2eb83533"
meta-mingw        = "master:0679d2a91575bd7f3efd0973aa99d7ee681e9395"
meta-selftest     = "wip:362d6e0490b2ab9a0e9eecbc6c5eb63f2eb83533"

Initialising tasks: 100% |#####################################################################################################################################| Time: 0:00:02
NOTE: Executing RunQueue Tasks
Extensible SDK testing environment: mips32r2-poky-linux
THREAD: 139990289868544
----------------------------------------------------------------------

Running tests...
----------------------------------------------------------------------

----------------------------------------------------------------------
Ran 0 tests in 0.001s

OK (skipped=1)

Generating XML reports...

THREAD: 139990527887104
----------------------------------------------------------------------

Running tests...
----------------------------------------------------------------------

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK (skipped=1)

Generating XML reports...

THREAD: 139990298261248
----------------------------------------------------------------------

Running tests...
----------------------------------------------------------------------
  test_perl_exists (perl.PerlTest) ... OK (0.014s)
  test_perl_works (perl.PerlTest) ... OK (0.017s)

----------------------------------------------------------------------
Ran 2 tests in 0.034s

OK

Generating XML reports...

THREAD: 139990511101696
----------------------------------------------------------------------

Running tests...
----------------------------------------------------------------------
  test_gcc_compile (gcc.GccCompileTest) ... OK (0.064s)
  test_gpp2_compile (gcc.GccCompileTest) ... OK (0.071s)
  test_gpp_compile (gcc.GccCompileTest) ... OK (0.099s)
  test_make (gcc.GccCompileTest) ... OK (0.059s)

----------------------------------------------------------------------
Ran 4 tests in 0.305s

OK

Generating XML reports...

THREAD: 139990519494400
----------------------------------------------------------------------

Running tests...
----------------------------------------------------------------------
  test_lzip (buildlzip.BuildLzipTest) ... OK (4.291s)

----------------------------------------------------------------------
Ran 1 test in 4.330s

OK

Generating XML reports...

THREAD: 139990767052544
----------------------------------------------------------------------

Running tests...
----------------------------------------------------------------------
  test_cpio (buildcpio.BuildCpioTest) ... OK (52.939s)

----------------------------------------------------------------------
Ran 1 test in 53.465s

OK

Generating XML reports...

THREAD: 139990281475840
----------------------------------------------------------------------

Running tests...
----------------------------------------------------------------------
  test_devtool_location (devtool.DevtoolTest) ... OK (0.008s)
  test_devtool_add_reset (devtool.DevtoolTest) ... OK (7.104s)
  test_devtool_build_cmake (devtool.DevtoolTest) ... OK (30.998s)
  test_devtool_build_esdk_package (devtool.DevtoolTest) ... OK (15.581s)
  test_devtool_build_make (devtool.DevtoolTest) ... OK (16.802s)
  test_devtool_kernelmodule (devtool.DevtoolTest) ... OK (121.021s)
  test_extend_autotools_recipe_creation (devtool.DevtoolTest) ... OK (43.555s)
  test_recipes_for_nodejs (devtool.DevtoolTest) ... OK (4.555s)
  test_sdk_update_http (devtool.SdkUpdateTest) ... ERROR (15.812s)

======================================================================
ERROR [15.812s]: test_sdk_update_http (devtool.SdkUpdateTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/media/build1/poky/meta/lib/oeqa/sdkext/cases/devtool.py", line 124, in test_sdk_update_http
    output = self._run("devtool sdk-update \"%s\"" % self.http_url)
  File "/media/build1/poky/meta/lib/oeqa/sdkext/case.py", line 21, in _run
    shell=True, env=env, universal_newlines=True)
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
oeqa.utils.subprocesstweak.OETestCalledProcessError: Command '. /media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/environment-setup-mips32r2-poky-linux > /dev/null; devtool sdk-update "http://127.0.0.1:41569";' returned non-zero exit status 255
Standard Output: NOTE: Starting bitbake server...
NOTE: Reconnecting to bitbake server...
Cloning into 'layers'...
NOTE: Preparing build system... (This may take some time.)
ERROR: Preparation failed:
NOTE: Reconnecting to bitbake server...
Loading cache...done.
Loaded 1290 entries from dependency cache.
Parsing recipes...done.
Parsing of 835 .bb files complete (0 cached, 835 parsed). 1290 targets, 74 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Initialising tasks...ERROR: An uncaught exception occurred in runqueue
Traceback (most recent call last):
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1375, in execute_runqueue
    return self._execute_runqueue()
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1315, in _execute_runqueue
    self.start_worker()
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1169, in start_worker
    self.worker[mc] = self._start_worker(mc)
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1115, in _start_worker
    worker = subprocess.Popen(["bitbake-worker", magic], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'bitbake-worker'
Traceback (most recent call last):
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1315, in RunQueue._execute_runqueue():
                     self.rqdata.init_progress_reporter.next_stage()
    >                self.start_worker()
                     self.rqdata.init_progress_reporter.next_stage()
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1169, in RunQueue.start_worker():
             for mc in self.rqdata.dataCaches:
    >            self.worker[mc] = self._start_worker(mc)
     
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1115, in RunQueue._start_worker(mc='', fakeroot=False, rqexec=None):
             else:
    >            worker = subprocess.Popen(["bitbake-worker", magic], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
             bb.utils.nonblockingfd(worker.stdout)
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 676, in Popen.__init__(args=['bitbake-worker', 'decafbad'], bufsize=-1, executable=None, stdin=-1, stdout=-1, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False, pass_fds=()):
                                     errread, errwrite,
    >                                restore_signals, start_new_session)
             except:
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 1282, in Popen._execute_child(args=['bitbake-worker', 'decafbad'], executable=b'bitbake-worker', preexec_fn=None, close_fds=True, pass_fds=(), cwd=None, env=None, startupinfo=None, creationflags=0, shell=False, p2cread=114, p2cwrite=115, c2pread=116, c2pwrite=117, errread=-1, errwrite=-1, restore_signals=True, start_new_session=False):
                                     err_msg += ': ' + repr(orig_executable)
    >                    raise child_exception_type(errno_num, err_msg)
                     raise child_exception_type(err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'bitbake-worker'

ERROR: Running idle function
Traceback (most recent call last):
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/server/process.py", line 270, in idle_commands
    retval = function(self, data, False)
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/cooker.py", line 1374, in buildTargetsIdle
    retval = rq.execute_runqueue()
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1375, in execute_runqueue
    return self._execute_runqueue()
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1315, in _execute_runqueue
    self.start_worker()
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1169, in start_worker
    self.worker[mc] = self._start_worker(mc)
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1115, in _start_worker
    worker = subprocess.Popen(["bitbake-worker", magic], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'bitbake-worker'
Traceback (most recent call last):
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1315, in RunQueue._execute_runqueue():
                     self.rqdata.init_progress_reporter.next_stage()
    >                self.start_worker()
                     self.rqdata.init_progress_reporter.next_stage()
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1169, in RunQueue.start_worker():
             for mc in self.rqdata.dataCaches:
    >            self.worker[mc] = self._start_worker(mc)
     
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/poky/bitbake/lib/bb/runqueue.py", line 1115, in RunQueue._start_worker(mc='', fakeroot=False, rqexec=None):
             else:
    >            worker = subprocess.Popen(["bitbake-worker", magic], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
             bb.utils.nonblockingfd(worker.stdout)
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 676, in Popen.__init__(args=['bitbake-worker', 'decafbad'], bufsize=-1, executable=None, stdin=-1, stdout=-1, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False, pass_fds=()):
                                     errread, errwrite,
    >                                restore_signals, start_new_session)
             except:
  File "/media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/subprocess.py", line 1282, in Popen._execute_child(args=['bitbake-worker', 'decafbad'], executable=b'bitbake-worker', preexec_fn=None, close_fds=True, pass_fds=(), cwd=None, env=None, startupinfo=None, creationflags=0, shell=False, p2cread=114, p2cwrite=115, c2pread=116, c2pwrite=117, errread=-1, errwrite=-1, restore_signals=True, start_new_session=False):
                                     err_msg += ': ' + repr(orig_executable)
    >                    raise child_exception_type(errno_num, err_msg)
                     raise child_exception_type(err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'bitbake-worker'


Summary: There were 2 ERROR messages shown, returning a non-zero exit code.



----------------------------------------------------------------------
Ran 9 tests in 286.471s

FAILED (errors=1)

Generating XML reports...

RESULTS:
RESULTS - buildcpio.BuildCpioTest.test_cpio - Testcase -1: PASSED
RESULTS - buildgalculator.GalculatorTest.test_galculator - Testcase -1: SKIPPED
RESULTS - buildlzip.BuildLzipTest.test_lzip - Testcase -1: PASSED
RESULTS - gcc.GccCompileTest.test_gcc_compile - Testcase -1: PASSED
RESULTS - gcc.GccCompileTest.test_gpp2_compile - Testcase -1: PASSED
RESULTS - gcc.GccCompileTest.test_gpp_compile - Testcase -1: PASSED
RESULTS - gcc.GccCompileTest.test_make - Testcase -1: PASSED
RESULTS - perl.PerlTest.test_perl_exists - Testcase -1: PASSED
RESULTS - perl.PerlTest.test_perl_works - Testcase -1: PASSED
RESULTS - python.PythonTest.test_python_exists - Testcase -1: SKIPPED
RESULTS - python.PythonTest.test_python_stdout - Testcase -1: SKIPPED
RESULTS - python.PythonTest.test_python_testfile - Testcase -1: SKIPPED
RESULTS - devtool.DevtoolTest.test_devtool_add_reset - Testcase -1: PASSED
RESULTS - devtool.DevtoolTest.test_devtool_build_cmake - Testcase 1607: PASSED
RESULTS - devtool.DevtoolTest.test_devtool_build_esdk_package - Testcase 1606: PASSED
RESULTS - devtool.DevtoolTest.test_devtool_build_make - Testcase 1605: PASSED
RESULTS - devtool.DevtoolTest.test_devtool_kernelmodule - Testcase 1609: PASSED
RESULTS - devtool.DevtoolTest.test_devtool_location - Testcase -1: PASSED
RESULTS - devtool.DevtoolTest.test_extend_autotools_recipe_creation - Testcase 1608: PASSED
RESULTS - devtool.DevtoolTest.test_recipes_for_nodejs - Testcase 1610: PASSED
RESULTS - devtool.SdkUpdateTest.test_sdk_update_http - Testcase -1: ERROR
SUMMARY:
core-image-minimal esdk (poky-glibc-x86_64-core-image-minimal-mips32r2-toolchain-ext-2.3.sh:environment-setup-mips32r2-poky-linux) - Ran 21 tests in 286.484s
core-image-minimal esdk - FAIL - Required tests failed
ERROR: core-image-minimal-1.0-r0 do_testsdkext: core-image-minimal - FAILED - check the task log and the commands log
ERROR: core-image-minimal-1.0-r0 do_testsdkext: Function failed: do_testsdkext
ERROR: Logfile of failure stored in: /media/build1/poky/build/tmp/work/qemumips-poky-linux/core-image-minimal/1.0-r0/temp/log.do_testsdkext.65719
ERROR: Task (/media/build1/poky/meta/recipes-core/images/core-image-minimal.bb:do_testsdkext) failed with exit code '1'
NOTE: Tasks Summary: Attempted 1 tasks of which 0 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /media/build1/poky/meta/recipes-core/images/core-image-minimal.bb:do_testsdkext
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

There is also a report of:

2017-08-23 05:44:05 [post-build-test] NOTE: Tasks Summary: Attempted 82 tasks of which 26 didn't need to be rerun and all succeeded.
2017-08-23 05:44:05 [post-build-test] NOTE: Successfully installed meta-extsdk-toolchain
2017-08-23 05:44:05 [post-build-test] NOTE: Reconnecting to bitbake server...
2017-08-23 05:44:05 [post-build-test] Loading cache...done.
2017-08-23 05:44:05 [post-build-test] Loaded 3652 entries from dependency cache.
2017-08-23 05:44:05 [post-build-test] Parsing recipes...done.
2017-08-23 05:44:05 [post-build-test] Parsing of 2733 .bb files complete (2729 cached, 4 parsed). 3656 targets, 475 skipped, 1 masked, 0 errors.
2017-08-23 05:44:05 [post-build-test] NOTE: Resolving any missing task queue dependencies
2017-08-23 05:44:05 [post-build-test] Initialising tasks...NOTE: Checking active recipes
2017-08-23 05:44:05 [post-build-test] ERROR: An uncaught exception occurred in runqueue
2017-08-23 05:44:05 [post-build-test] Traceback (most recent call last):
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/runqueue.py", line 1375, in RunQueue.execute_runqueue():
2017-08-23 05:44:05 [post-build-test]              try:
2017-08-23 05:44:05 [post-build-test]     >            return self._execute_runqueue()
2017-08-23 05:44:05 [post-build-test]              except bb.runqueue.TaskFailure:
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/runqueue.py", line 1351, in RunQueue._execute_runqueue():
2017-08-23 05:44:05 [post-build-test]                  self.teardown_workers()
2017-08-23 05:44:05 [post-build-test]     >            if self.rqexe.stats.failed:
2017-08-23 05:44:05 [post-build-test]                      logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and %d failed.", self.rqexe.stats.completed + self.rqexe.stats.failed, self.rqexe.stats.skipped, self.rqexe.stats.failed)
2017-08-23 05:44:05 [post-build-test] AttributeError: 'RunQueueExecuteScenequeue' object has no attribute 'stats'
2017-08-23 05:44:05 [post-build-test] 
2017-08-23 05:44:05 [post-build-test] ERROR: Running idle function
2017-08-23 05:44:05 [post-build-test] Traceback (most recent call last):
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/server/process.py", line 252, in ProcessServer.idle_commands(delay=0.10000000000000001, fds=[<socket.socket fd=7, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>, <socket.socket fd=13, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>, <bb.server.process.ConnectionReader object at 0x7fd77907feb8>]):
2017-08-23 05:44:05 [post-build-test]                  try:
2017-08-23 05:44:05 [post-build-test]     >                retval = function(self, data, False)
2017-08-23 05:44:05 [post-build-test]                      if retval is False:
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/cooker.py", line 1364, in buildTargetsIdle(server=<ProcessServer(ProcessServer-1, started)>, rq=<bb.runqueue.RunQueue object at 0x7fd77cc6c278>, abort=False):
2017-08-23 05:44:05 [post-build-test]                  try:
2017-08-23 05:44:05 [post-build-test]     >                retval = rq.execute_runqueue()
2017-08-23 05:44:05 [post-build-test]                  except runqueue.TaskFailure as exc:
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/runqueue.py", line 1375, in RunQueue.execute_runqueue():
2017-08-23 05:44:05 [post-build-test]              try:
2017-08-23 05:44:05 [post-build-test]     >            return self._execute_runqueue()
2017-08-23 05:44:05 [post-build-test]              except bb.runqueue.TaskFailure:
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/runqueue.py", line 1351, in RunQueue._execute_runqueue():
2017-08-23 05:44:05 [post-build-test]                  self.teardown_workers()
2017-08-23 05:44:05 [post-build-test]     >            if self.rqexe.stats.failed:
2017-08-23 05:44:05 [post-build-test]                      logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and %d failed.", self.rqexe.stats.completed + self.rqexe.stats.failed, self.rqexe.stats.skipped, self.rqexe.stats.failed)
2017-08-23 05:44:05 [post-build-test] AttributeError: 'RunQueueExecuteScenequeue' object has no attribute 'stats'
2017-08-23 05:44:05 [post-build-test] 
2017-08-23 05:44:05 [post-build-test] 
2017-08-23 05:44:05 [post-build-test] Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
2017-08-23 05:44:05 [post-build-test] ERROR: Failed to bitbake build-sysroots:
2017-08-23 05:44:05 [post-build-test] Execution of '. /srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/oe-init-build-env /srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext > /dev/null && bitbake build-sysroots' failed with exit code 1:
2017-08-23 05:44:05 [post-build-test] NOTE: Reconnecting to bitbake server...
2017-08-23 05:44:05 [post-build-test] Loading cache...done.
2017-08-23 05:44:05 [post-build-test] Loaded 3652 entries from dependency cache.
2017-08-23 05:44:05 [post-build-test] Parsing recipes...done.
2017-08-23 05:44:05 [post-build-test] Parsing of 2733 .bb files complete (2729 cached, 4 parsed). 3656 targets, 475 skipped, 1 masked, 0 errors.
2017-08-23 05:44:05 [post-build-test] NOTE: Resolving any missing task queue dependencies
2017-08-23 05:44:05 [post-build-test] Initialising tasks...NOTE: Checking active recipes
2017-08-23 05:44:05 [post-build-test] ERROR: An uncaught exception occurred in runqueue
2017-08-23 05:44:05 [post-build-test] Traceback (most recent call last):
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/runqueue.py", line 1375, in RunQueue.execute_runqueue():
2017-08-23 05:44:05 [post-build-test]              try:
2017-08-23 05:44:05 [post-build-test]     >            return self._execute_runqueue()
2017-08-23 05:44:05 [post-build-test]              except bb.runqueue.TaskFailure:
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/runqueue.py", line 1351, in RunQueue._execute_runqueue():
2017-08-23 05:44:05 [post-build-test]                  self.teardown_workers()
2017-08-23 05:44:05 [post-build-test]     >            if self.rqexe.stats.failed:
2017-08-23 05:44:05 [post-build-test]                      logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and %d failed.", self.rqexe.stats.completed + self.rqexe.stats.failed, self.rqexe.stats.skipped, self.rqexe.stats.failed)
2017-08-23 05:44:05 [post-build-test] AttributeError: 'RunQueueExecuteScenequeue' object has no attribute 'stats'
2017-08-23 05:44:05 [post-build-test] 
2017-08-23 05:44:05 [post-build-test] ERROR: Running idle function
2017-08-23 05:44:05 [post-build-test] Traceback (most recent call last):
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/server/process.py", line 252, in ProcessServer.idle_commands(delay=0.10000000000000001, fds=[<socket.socket fd=7, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>, <socket.socket fd=13, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>, <bb.server.process.ConnectionReader object at 0x7fd77907feb8>]):
2017-08-23 05:44:05 [post-build-test]                  try:
2017-08-23 05:44:05 [post-build-test]     >                retval = function(self, data, False)
2017-08-23 05:44:05 [post-build-test]                      if retval is False:
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/cooker.py", line 1364, in buildTargetsIdle(server=<ProcessServer(ProcessServer-1, started)>, rq=<bb.runqueue.RunQueue object at 0x7fd77cc6c278>, abort=False):
2017-08-23 05:44:05 [post-build-test]                  try:
2017-08-23 05:44:05 [post-build-test]     >                retval = rq.execute_runqueue()
2017-08-23 05:44:05 [post-build-test]                  except runqueue.TaskFailure as exc:
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/runqueue.py", line 1375, in RunQueue.execute_runqueue():
2017-08-23 05:44:05 [post-build-test]              try:
2017-08-23 05:44:05 [post-build-test]     >            return self._execute_runqueue()
2017-08-23 05:44:05 [post-build-test]              except bb.runqueue.TaskFailure:
2017-08-23 05:44:05 [post-build-test]   File "/srv/jenkins/workspace/ci-2017-08-23_04-27-13-build-431/build/tmp-headless-glibc/work/intel_corei7_64-refkit-linux/refkit-image-gateway/1.0-r0/testsdkext/layers/openembedded-core/bitbake/lib/bb/runqueue.py", line 1351, in RunQueue._execute_runqueue():
2017-08-23 05:44:05 [post-build-test]                  self.teardown_workers()
2017-08-23 05:44:05 [post-build-test]     >            if self.rqexe.stats.failed:
2017-08-23 05:44:05 [post-build-test]                      logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and %d failed.", self.rqexe.stats.completed + self.rqexe.stats.failed, self.rqexe.stats.skipped, self.rqexe.stats.failed)
2017-08-23 05:44:05 [post-build-test] AttributeError: 'RunQueueExecuteScenequeue' object has no attribute 'stats'
2017-08-23 05:44:05 [post-build-test] 
2017-08-23 05:44:05 [post-build-test] 
2017-08-23 05:44:05 [post-build-test] Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

which I suspect is some kind of similar issue.
Comment 1 Richard Purdie 2017-08-24 12:09:21 UTC
Adding BB_SERVER_TIMEOUT to SDK_LOCAL_CONF_BLACKLIST worked around the first failure, second not as yet tested.
Comment 2 Richard Purdie 2017-08-30 13:56:19 UTC
Confirmed as fixing the second too. Merged the fix as a workaround to master whilst we look into and fix this properly.
Comment 3 oscar lopez arandas 2017-10-24 16:54:49 UTC

    This bug is still present on  in  GDC autobuilder  ERROR: core-image-minimal-1.0-r0 do_testsdkext: Couldn't install the extensible SDK: but in diferent BBVERSION and DISTRO_VERSION.

Build Configuration:
BB_VERSION        = "1.32.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "universal"
TARGET_SYS        = "i586-poky-linux"
MACHINE           = "qemux86"
DISTRO            = "poky"
DISTRO_VERSION    = "2.2.2"
TUNE_FEATURES     = "m32 i586"        
meta-yocto-bsp    = "heads/morty:1d92cb1a20135cfffff9f94a6633ec0840518738"
==============================================================================
ERROR: core-image-minimal-1.0-r0 do_testsdkext: Function failed: do_testsdkext
ERROR: Logfile of failure stored in: /home/ab/autobuilder/yocto-worker/nightly-x86/build/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_testsdkext.22423
NOTE: recipe core-image-minimal-1.0-r0: task do_testsdkext: Failed
ERROR: Task (/home/ab/autobuilder/yocto-worker/nightly-x86/build/meta/recipes-core/images/core-image-minimal.bb:do_testsdkext) failed with exit code '1'
ERROR: core-image-sato-1.0-r0 do_testsdkext: Couldn't install the extensible SDK:
Poky (Yocto Project Reference Distro) Extensible SDK installer version 2.2.2
============================================================================
full-log in attachment
Comment 4 oscar lopez arandas 2017-10-24 16:55:29 UTC
Created attachment 4095 [details]
full_log
Comment 5 Richard Purdie 2020-08-25 11:14:52 UTC
The issue was the sys.argv references in runqueue used for BB_SETSCENE_ENFORCE. http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=8ce3e9a76dd34be067486c014b9a9a1fc52854bc is the fix.
Comment 6 Richard Purdie 2021-09-22 17:31:25 UTC
These issues have been resolved now. I was seeing local failures but tracked that to an unrelated issue which I will sent a patch for separately.