Bug 13243

Summary:	devtool test failure in python scanning code
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Paul Eggleton <bluelightning>
Component:	Scripts and Tools	Assignee:	New Comer Bugs <newcomer>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	open.source, randy.macleod, ross.burton
Version:	5.99
Target Milestone:	2.8 M2
Hardware:	x86
OS:	Multiple
Whiteboard:	NEWCOMER
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Paul Eggleton 2019-03-26 23:56:09 UTC

Failures in several of the devtool tests involving python scanning have started showing up; they appear to be the same underlying issue:

======================================================================
2019-03-24 23:59:31,514 - oe-selftest - INFO - FAIL: devtool.DevtoolAddTests.test_devtool_add_fetch (subunit.RemotedTestCase)
2019-03-24 23:59:31,514 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-03-24 23:59:31,514 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 405, in test_devtool_add_fetch
    result = runCmd('devtool add %s %s -f %s' % (testrecipe, srcdir, url))
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 194, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'devtool add python-markupsafe /tmp/devtoolqa0qtw___7/python-markupsafe -f https://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.23.tar.gz' returned non-zero exit status 1:
NOTE: Starting bitbake server...
NOTE: Creating workspace layer in /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-10944/workspace
WARNING: -f/--fetch option is deprecated - you can now simply specify the URL to fetch as a positional argument instead
NOTE: Starting bitbake server...
NOTE: Fetching https://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.23.tar.gz...
Loading cache...done.
Loaded 1285 entries from dependency cache.
Parsing recipes...done.
Parsing of 812 .bb files complete (811 cached, 1 parsed). 1286 targets, 43 skipped, 0 masked, 0 errors.
Removing 1 recipes from the core2-64 sysroot...done.
Removing 1 recipes from the qemux86_64 sysroot...done.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION           = "1.40.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "2.6.1"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "thud-next:d8402a9143617d4cdbd2826a87e15665fe6fd46e"
meta-selftest        = "master:3cf8e1ea39cfb78b1302a6f7e86fa03c8ccf5531"
workspace            = "thud-next:d8402a9143617d4cdbd2826a87e15665fe6fd46e"
Initialising tasks...done.
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Scanning paths for packages & dependencies: markupsafe
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/recipetool", line 121, in <module>
    ret = main()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/recipetool", line 110, in main
    ret = args.func(args)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create.py", line 757, in create_recipe
    handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create_buildsys_python.py", line 280, in process
    mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create_buildsys_python.py", line 498, in scan_setup_python_deps
    scanned_deps = self.scan_python_dependencies([os.path.join(srctree, p) for p in to_scan])
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create_buildsys_python.py", line 519, in scan_python_dependencies
    dep, filename = line.split('\t', 1)
ValueError: not enough values to unpack (expected 2, got 1)
ERROR: Command 'recipetool --color=auto create --devtool -o /tmp/devtoolcm1p6mp6 'https://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.23.tar.gz'  -x /tmp/devtoolqa0qtw___7/python-markupsafe -N python-markupsafe' failed
======================================================================
2019-03-24 23:59:31,514 - oe-selftest - INFO - FAIL: recipetool.RecipetoolTests.test_recipetool_create_github (subunit.RemotedTestCase)
2019-03-24 23:59:31,514 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-03-24 23:59:31,514 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 455, in test_recipetool_create_github
    result = runCmd(['recipetool', 'create', '-o', temprecipe, srcuri])
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 194, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command '['recipetool', 'create', '-o', '/tmp/recipetoolqaaa43n8sa/recipe', 'https://github.com/mesonbuild/meson;rev=0.32.0']' returned non-zero exit status 1:
NOTE: Starting bitbake server...
NOTE: Fetching git://github.com/mesonbuild/meson;rev=0.32.0;protocol=https;nobranch=1...
Loading cache...done.
Loaded 1286 entries from dependency cache.
Parsing recipes...done.
Parsing of 812 .bb files complete (811 cached, 1 parsed). 1286 targets, 43 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION           = "1.40.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "2.6.1"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "thud-next:d8402a9143617d4cdbd2826a87e15665fe6fd46e"
meta-selftest        = "master:3cf8e1ea39cfb78b1302a6f7e86fa03c8ccf5531"
recipetoolqa5e7cur6s = "<unknown>:<unknown>"
Initialising tasks...done.
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Scanning paths for packages & dependencies: mesonbuild, mesonbuild/modules, mesonbuild/scripts, mesonbuild/backend, mesonbuild/wrap, meson.py, mesonconf.py, mesonintrospect.py, wraptool.py
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/recipetool", line 121, in <module>
    ret = main()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/recipetool", line 110, in main
    ret = args.func(args)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create.py", line 757, in create_recipe
    handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create_buildsys_python.py", line 280, in process
    mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create_buildsys_python.py", line 498, in scan_setup_python_deps
    scanned_deps = self.scan_python_dependencies([os.path.join(srctree, p) for p in to_scan])
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create_buildsys_python.py", line 519, in scan_python_dependencies
    dep, filename = line.split('\t', 1)
ValueError: not enough values to unpack (expected 2, got 1)
======================================================================
2019-03-24 23:59:31,514 - oe-selftest - INFO - FAIL: recipetool.RecipetoolTests.test_recipetool_create_github_tarball (subunit.RemotedTestCase)
2019-03-24 23:59:31,515 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-03-24 23:59:31,515 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 471, in test_recipetool_create_github_tarball
    result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 194, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'recipetool create -o /tmp/recipetoolqaj43o9p87/recipe https://github.com/mesonbuild/meson/releases/download/0.32.0/meson-0.32.0.tar.gz' returned non-zero exit status 1:
NOTE: Starting bitbake server...
NOTE: Fetching https://github.com/mesonbuild/meson/releases/download/0.32.0/meson-0.32.0.tar.gz...
Loading cache...done.
Loaded 1287 entries from dependency cache.
Parsing recipes...done.
Parsing of 812 .bb files complete (811 cached, 1 parsed). 1286 targets, 43 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION           = "1.40.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "2.6.1"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "thud-next:d8402a9143617d4cdbd2826a87e15665fe6fd46e"
meta-selftest        = "master:3cf8e1ea39cfb78b1302a6f7e86fa03c8ccf5531"
recipetoolqa5e7cur6s = "<unknown>:<unknown>"
Initialising tasks...done.
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Scanning paths for packages & dependencies: mesonbuild, mesonbuild/modules, mesonbuild/scripts, mesonbuild/backend, mesonbuild/wrap, meson.py, mesonconf.py, mesonintrospect.py, wraptool.py
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/recipetool", line 121, in <module>
    ret = main()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/recipetool", line 110, in main
    ret = args.func(args)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create.py", line 757, in create_recipe
    handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create_buildsys_python.py", line 280, in process
    mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create_buildsys_python.py", line 498, in scan_setup_python_deps
    scanned_deps = self.scan_python_dependencies([os.path.join(srctree, p) for p in to_scan])
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/scripts/lib/recipetool/create_buildsys_python.py", line 519, in scan_python_dependencies
    dep, filename = line.split('\t', 1)
ValueError: not enough values to unpack (expected 2, got 1)
======================================================================

Comment 1 Paul Eggleton 2019-03-26 23:56:27 UTC

Build link:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/115/steps/7/logs/step2d

Comment 2 Oleksandr Kravchuk 2019-06-15 14:20:23 UTC

Fixed by ca9c738053090b41.