Bug 12655

Summary: [2.5 M3 rc1][OE Core:Debian9] layerappend.LayerAppendTests.test_layer_appends - Testcase 1196: ERROR
Product: [QA/Testing] Build Testing Reporter: Aaron Chan <aaron.chun.yew.chan>
Component: generalAssignee: Paul Eggleton <bluelightning>
Status: RESOLVED FIXED QA Contact: Aaron Chan <aaron.chun.yew.chan>
Severity: major    
Priority: High CC: stephano
Version: 2.5   
Target Milestone: 2.5 M4   
Hardware: Other   
OS: x86   
Whiteboard:
OS type for building Yocto: --- Type of Regression: Regression (Found by QA test)
Verified: Documentation change: Don't know
Attachments:
Description Flags
[2.5 M3 rc1] Build 49 - Debian9 - Oe-selftest - PASSED none

Description Aaron Chan 2018-04-04 19:05:19 UTC
===============================================================================
Build Configuration:
===============================================================================
BB_VERSION           = "1.37.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "debian-9"
TARGET_SYS           = "i586-poky-linux"
MACHINE              = "qemux86"
DISTRO               = "poky"
DISTRO_VERSION       = "2.4+snapshot-20180402"
TUNE_FEATURES        = "m32 i586"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "heads/master:f49ee61422c867516db252054e993df29f775136"

===============================================================================
Stdout: 
===============================================================================
2018-04-02 08:02:03,981 - oe-selftest - INFO -   test_layer_appends (layerappend.LayerAppendTests)
2018-04-02 08:02:25,777 - oe-selftest - INFO -  ... ERROR (21.796s)
...
...
2018-04-02 11:03:14,773 - oe-selftest - INFO - ======================================================================
2018-04-02 11:03:14,774 - oe-selftest - INFO - ERROR [21.796s]: test_layer_appends (layerappend.LayerAppendTests)
2018-04-02 11:03:14,774 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-04-02 11:03:14,774 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/selftest/cases/layerappend.py", line 85, in test_layer_appends
    data = ftools.read_file(stagingdir + "/appendtest.txt")
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/utils/ftools.py", line 23, in read_file
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp/work/i586-poky-linux/layerappendtest/1.0-r0/sysroot-destdir/appendtest.txt'


===============================================================================
Summary:
===============================================================================
Based on the workspace (local filesystem), the file structure 1.0-r0/sysroot-destdir/appendtest.txt seems to be missing.

http://centaurus01.png.intel.com:8080/workspace/debian9/nightly-oe-selftest-debian9/build/build/tmp/work/i586-poky-linux/layerappendtest/


Due to this setup failure, it causes a chain reaction in resulting in other failures. Categorized as:

2018-04-02 11:03:15,082 - oe-selftest - INFO - RESULTS - runtime_test.Postinst.test_failing_postinst - Testcase -1: PASSED
2018-04-02 11:03:14,776 - oe-selftest - INFO - ======================================================================
2018-04-02 11:03:14,776 - oe-selftest - INFO - ERROR [450.924s]: test_failing_postinst (runtime_test.Postinst)
2018-04-02 11:03:14,776 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-04-02 11:03:14,776 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 257, in test_failing_postinst
    "rootfs-before-failure file was not created")
AssertionError: False is not true : rootfs-before-failure file was not created

2018-04-02 11:03:15,082 - oe-selftest - INFO - RESULTS - runtime_test.Postinst.test_postinst_rootfs_and_boot - Testcase 1540: PASSED
2018-04-02 11:03:14,779 - oe-selftest - INFO - ======================================================================
2018-04-02 11:03:14,779 - oe-selftest - INFO - ERROR [2393.749s]: test_postinst_rootfs_and_boot (runtime_test.Postinst)
2018-04-02 11:03:14,779 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-04-02 11:03:14,779 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 216, in test_postinst_rootfs_and_boot
    "rootfs state file was not created")
AssertionError: False is not true : rootfs state file was not created

2018-04-02 11:03:15,022 - oe-selftest - INFO - RESULTS - buildoptions.ImageOptionsTests.test_ccache_tool - Testcase 286: FAILED
2018-04-02 11:03:14,780 - oe-selftest - INFO - ======================================================================
2018-04-02 11:03:14,780 - oe-selftest - INFO - FAIL [20.331s]: test_ccache_tool (buildoptions.ImageOptionsTests)
2018-04-02 11:03:14,780 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-04-02 11:03:14,780 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/selftest/cases/buildoptions.py", line 38, in test_ccache_tool
    self.assertTrue(os.path.isfile(p), msg = "No ccache found (%s)" % p)
AssertionError: False is not true : No ccache found (/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp/work/x86_64-linux/ccache-native/3.3.5-r0/sysroot-destdir/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp/work/x86_64-linux/ccache-native/3.3.5-r0/recipe-sysroot-native/usr/bin/ccache)


2018-04-02 11:03:15,044 - oe-selftest - INFO - RESULTS - liboe.LibOE.test_copy_tree_xattr - Testcase 1636: FAILED
2018-04-02 11:03:14,780 - oe-selftest - INFO - ======================================================================
2018-04-02 11:03:14,780 - oe-selftest - INFO - FAIL [7.529s]: test_copy_tree_xattr (liboe.LibOE)
2018-04-02 11:03:14,780 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-04-02 11:03:14,781 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/selftest/cases/liboe.py", line 66, in test_copy_tree_xattr
    runCmd('%s/setfattr -n user.oetest -v "testing liboe" %s' % (bindir, oe.path.join(src, testfilename)))
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/utils/commands.py", line 191, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command '/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp/work/x86_64-linux/attr-native/2.4.47-r0/sysroot-destdir/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp/work/x86_64-linux/attr-native/2.4.47-r0/recipe-sysroot-native/usr/bin/setfattr -n user.oetest -v "testing liboe" /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp/liboetests/src/testxattr' returned non-zero exit status 127:
/bin/sh: 1: /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp/work/x86_64-linux/attr-native/2.4.47-r0/sysroot-destdir/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp/work/x86_64-linux/attr-native/2.4.47-r0/recipe-sysroot-native/usr/bin/setfattr: not found


2018-04-02 11:03:15,083 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_dnf - Testcase 1883: FAILED
2018-04-02 11:03:15,083 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_install - Testcase 1644: FAILED
2018-04-02 11:03:15,087 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_allarch_samesigs - Testcase 1368: FAILED
2018-04-02 11:03:15,087 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_allarch_samesigs_multilib - Testcase 1645: FAILED
2018-04-02 11:03:15,091 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_sametune_samesigs - Testcase 1369: FAILED
2018-04-02 11:03:15,095 - oe-selftest - INFO - RESULTS - wic.Wic.test_bmap_long - Testcase 1655: FAILED
2018-04-02 11:03:15,095 - oe-selftest - INFO - RESULTS - wic.Wic.test_bmap_short - Testcase 1496: FAILED
2018-04-02 11:03:15,095 - oe-selftest - INFO - RESULTS - wic.Wic.test_bootloader_config - Testcase 1385: FAILED
2018-04-02 11:03:15,095 - oe-selftest - INFO - RESULTS - wic.Wic.test_build_artifacts - Testcase 1212: FAILED
2018-04-02 11:03:15,095 - oe-selftest - INFO - RESULTS - wic.Wic.test_build_image_name - Testcase 1211: FAILED
2018-04-02 11:03:15,095 - oe-selftest - INFO - RESULTS - wic.Wic.test_build_rootfs_long - Testcase 1656: FAILED
2018-04-02 11:03:15,098 - oe-selftest - INFO - RESULTS - wic.Wic.test_default_output_dir - Testcase 1562: FAILED
2018-04-02 11:03:15,098 - oe-selftest - INFO - RESULTS - wic.Wic.test_exclude_path - Testcase 1661: FAILED
2018-04-02 11:03:15,099 - oe-selftest - INFO - RESULTS - wic.Wic.test_fixed_size - Testcase 1847: FAILED
2018-04-02 11:03:15,100 - oe-selftest - INFO - RESULTS - wic.Wic.test_fs_types - Testcase 1849: FAILED
2018-04-02 11:03:15,100 - oe-selftest - INFO - RESULTS - wic.Wic.test_gpt_image - Testcase 1157: FAILED
2018-04-02 11:03:15,102 - oe-selftest - INFO - RESULTS - wic.Wic.test_image_bootpart_globbed - Testcase 1850: FAILED
2018-04-02 11:03:15,103 - oe-selftest - INFO - RESULTS - wic.Wic.test_iso_image - Testcase 1346: FAILED
2018-04-02 11:03:15,103 - oe-selftest - INFO - RESULTS - wic.Wic.test_kickstart_parser - Testcase 1851: FAILED
2018-04-02 11:03:15,105 - oe-selftest - INFO - RESULTS - wic.Wic.test_mkefidisk - Testcase 1350: FAILED
2018-04-02 11:03:15,105 - oe-selftest - INFO - RESULTS - wic.Wic.test_mkfs_extraopts - Testcase 1922: FAILED
2018-04-02 11:03:15,106 - oe-selftest - INFO - RESULTS - wic.Wic.test_qemux86_directdisk - Testcase 1348: FAILED
2018-04-02 11:03:15,107 - oe-selftest - INFO - RESULTS - wic.Wic.test_rawcopy_plugin - Testcase 1853: FAILED
2018-04-02 11:03:15,107 - oe-selftest - INFO - RESULTS - wic.Wic.test_rootfs_artifacts - Testcase 1269: FAILED
2018-04-02 11:03:15,107 - oe-selftest - INFO - RESULTS - wic.Wic.test_rootfs_indirect_recipes - Testcase 1268: FAILED
2018-04-02 11:03:15,108 - oe-selftest - INFO - RESULTS - wic.Wic.test_sdimage_bootpart - Testcase 1561: FAILED
2018-04-02 11:03:15,108 - oe-selftest - INFO - RESULTS - wic.Wic.test_skip_build_check_long - Testcase 1671: FAILED
2018-04-02 11:03:15,108 - oe-selftest - INFO - RESULTS - wic.Wic.test_skip_build_check_short - Testcase 1563: FAILED
2018-04-02 11:03:15,109 - oe-selftest - INFO - RESULTS - wic.Wic.test_systemd_bootdisk - Testcase 1560: FAILED
2018-04-02 11:03:15,110 - oe-selftest - INFO - RESULTS - wic.Wic.test_wic_cp - Testcase 1856: FAILED
2018-04-02 11:03:15,110 - oe-selftest - INFO - RESULTS - wic.Wic.test_wic_cp_ext - Testcase -1: FAILED
2018-04-02 11:03:15,111 - oe-selftest - INFO - RESULTS - wic.Wic.test_wic_ls - Testcase 1857: FAILED
2018-04-02 11:03:15,111 - oe-selftest - INFO - RESULTS - wic.Wic.test_wic_ls_ext - Testcase -1: FAILED
2018-04-02 11:03:15,111 - oe-selftest - INFO - RESULTS - wic.Wic.test_wic_rm - Testcase 1858: FAILED
2018-04-02 11:03:15,111 - oe-selftest - INFO - RESULTS - wic.Wic.test_wic_rm_ext - Testcase -1: FAILED
2018-04-02 11:03:15,022 - oe-selftest - INFO - RESULTS - buildoptions.ImageOptionsTests.test_ccache_tool - Testcase 286: FAILED
2018-04-02 11:03:15,044 - oe-selftest - INFO - RESULTS - liboe.LibOE.test_copy_tree_xattr - Testcase 1636: FAILED
Comment 1 Paul Eggleton 2018-04-16 21:23:33 UTC
Aaron and I looked into this. The bug is reproducible every time, and it appears that the issue is that rm_work was enabled, and only on his debian9 worker. He has disabled rm_work and is re-running the build to confirm that this fixes all of the failures.
Comment 2 Aaron Chan 2018-04-16 22:12:16 UTC
===============================================================================
REPRODUCE ERRORS
===============================================================================
The issue was found to be reproducible with the INHERIT += "rm_work" set on the build/conf/auto.conf file.

{'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'x86_64',
                    'distro': 'poky', 'nosstate': True,
                    'packages': 'rpm', 'buildhistory' : False,
                    'prserv': False,
                    'atextappend': 'RPM_GPG_SIGN_CHUNK = "1"\nINHERIT += "rm_work"\n'}},

STEPS:
$ cd ~/yocto-worker/nightly-oe-selftest-debian9/build
$ source oe-init-build-env
$ oe-selftest -r layerappend.LayerAppendTests.test_layer_appends

===============================================================================
ISSUE RESOLVED
===============================================================================

Omit the INHERIT += "rm_work" from the build/conf/auto.conf. Refer to link below:

http://centaurus01.png.intel.com:8080/workspace/ab-controller/buildset-config/nightly-oe-selftest-debian9.conf

Rerun the same steps as above to manually verify the issues goes away.

===============================================================================
DEBIAN-9 RERUN ON 2.5 MR RC1 below:
===============================================================================

http://centaurus01.png.intel.com:8010/builders/nightly-oe-selftest-debian9/builds/48
Comment 3 Aaron Chan 2018-04-17 03:32:55 UTC
From the logs http://centaurus01.png.intel.com:8010/builders/nightly-oe-selftest-debian9/builds/48/steps/Running%20oe-selftest/logs/stdio

it appears the test_layer_appends (layerappend.LayerAppendTests) seems to be passing and issue was resolved after INHERIT += "rm_work" has been
removed during AutoConf Buildstep generated by Autobuilder.

2018-04-17 11:28:24,925 - oe-selftest - INFO -  ... OK (218.790s)
2018-04-17 11:28:24,928 - oe-selftest - INFO -   test_layer_appends (layerappend.LayerAppendTests)
2018-04-17 11:29:04,068 - oe-selftest - INFO -  ... OK (39.140s)
2018-04-17 11:29:05,574 - oe-selftest - INFO -   test_copy_hardlink_tree_count (liboe.LibOE)
2018-04-17 11:29:05,608 - oe-selftest - INFO -  ... OK (0.034s)
Comment 4 Aaron Chan 2018-04-17 04:36:52 UTC
Since this issue has been quickly resolved. Set the ticket status to RESOLVED FIXED.
Comment 5 Aaron Chan 2018-04-17 18:42:52 UTC
Rerun again and all the issues on Debian9 2.5 M3 rc1 is now resolved.

http://centaurus01.png.intel.com:8010/builders/nightly-oe-selftest-debian9/builds/49
Comment 6 Aaron Chan 2018-04-17 18:43:48 UTC
Created attachment 4272 [details]
[2.5 M3 rc1] Build 49 - Debian9 - Oe-selftest - PASSED