Bug 12587 - [2.4.2 rc2] test_sstate_sametune_samesigs (sstatetests.SStateTests)
Summary: [2.4.2 rc2] test_sstate_sametune_samesigs (sstatetests.SStateTests)
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 2.4.2
Hardware: x86 x86_64
: High normal
Target Milestone: 2.4.2
Deadline: 2018-03-30
Assignee: Ross Burton
QA Contact: Aaron Chan
URL: http://centaurus01.png.intel.com:8010
Whiteboard:
Depends on:
Blocks:
 
Reported: 2018-03-06 19:04 UTC by Aaron Chan
Modified: 2018-03-12 22:48 UTC (History)
10 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: Regression (Found by QA test)
Verified:
Documentation change: Don't know

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Aaron Chan 2018-03-06 19:04:48 UTC 
In 2.4.2 rc2 QA autobuilder testruns discover a Oe-selftest failure common on 
Debian 9 and OpenSuse 42.3 Linux Distribution and all other testcases were found to be passing. Here are the details in breakdown.


OpenSuse42.3
============

Reference Link: http://centaurus01.png.intel.com:8010/builders/nightly-oe-selftest-opensuse423/builds/13/steps/Running%20oe-selftest/logs/stdio

Stdout Log:
2018-02-22 05:39:45,628 - oe-selftest - INFO -   test_sstate_sametune_samesigs (sstatetests.SStateTests)
2018-02-22 05:42:08,592 - oe-selftest - INFO -  ... FAIL (142.964s)
...
...
2018-02-22 06:24:32,669 - oe-selftest - INFO - ======================================================================
2018-02-22 06:24:32,669 - oe-selftest - INFO - FAIL [142.964s]: test_sstate_sametune_samesigs (sstatetests.SStateTests)
2018-02-22 06:24:32,669 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-02-22 06:24:32,670 - oe-selftest - INFO - Traceback (most recent call last):
  File "~/autobuilder/yocto-worker/nightly-oe-selftest-opensuse423/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "~/autobuilder/yocto-worker/nightly-oe-selftest-opensuse423/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 413, in test_sstate_sametune_samesigs
    self.assertCountEqual(files1, files2)
AssertionError: Element counts were not equal:
First has 1, Second has 0:  '~/autobuilder/yocto-worker/nightly-oe-selftest-opensuse423/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-sysvinit/2.88dsf-r14.do_package_write_rpm.sigdata.1160409047766e564611af437ca8e15a'
First has 1, Second has 0:  '~/autobuilder/yocto-worker/nightly-oe-selftest-opensuse423/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-sysvinit/2.88dsf-r14.do_package_qa.sigdata.44d610245db83c87411fe6f2157d7ecb'
First has 1, Second has 0:  '~/autobuilder/yocto-worker/nightly-oe-selftest-opensuse423/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-initscripts/1.0-r155.do_package_qa.sigdata.9ed59a3ff3168c119e9e2a1b84920eed'
First has 1, Second has 0:  '~/autobuilder/yocto-worker/nightly-oe-selftest-opensuse423/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-initscripts/1.0-r155.do_package_write_rpm.sigdata.098579aa48494e34ffa8e3131491e9a5'
First has 0, Second has 1:  '~/autobuilder/yocto-worker/nightly-oe-selftest-opensuse423/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-sysvinit/2.88dsf-r14.do_package_write_rpm.sigdata.fa6830a1f6474dde350997cc84a47aa6'
First has 0, Second has 1:  '~/autobuilder/yocto-worker/nightly-oe-selftest-opensuse423/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-sysvinit/2.88dsf-r14.do_package_qa.sigdata.1cb239634f81d9861b1fba44febae2e2'
First has 0, Second has 1:  '~/autobuilder/yocto-worker/nightly-oe-selftest-opensuse423/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-initscripts/1.0-r155.do_package_write_rpm.sigdata.e5417bbe33e6a226be69cdfc854583ae'
First has 0, Second has 1:  '~/autobuilder/yocto-worker/nightly-oe-selftest-opensuse423/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-initscripts/1.0-r155.do_package_qa.sigdata.117b713e2fc93f4fd96bcd812515d537'

2018-02-22 06:24:32,670 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-02-22 06:24:32,670 - oe-selftest - INFO - Ran 317 tests in 33429.483s
2018-02-22 06:24:32,670 - oe-selftest - INFO - FAILED
2018-02-22 06:24:32,671 - oe-selftest - INFO -  (failures=1, skipped=1)

Debian 9 (Stretch)
=================
Reference Link: http://centaurus01.png.intel.com:8010/builders/nightly-oe-selftest-debian9/builds/18/steps/Running%20oe-selftest/logs/stdio

Stdout Log: 
2018-02-21 16:33:00,750 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_sametune_samesigs - Testcase 1369: FAILED
...
...
2018-02-21 16:33:00,563 - oe-selftest - INFO - ======================================================================
2018-02-21 16:33:00,563 - oe-selftest - INFO - FAIL [160.937s]: test_sstate_sametune_samesigs (sstatetests.SStateTests)
2018-02-21 16:33:00,563 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-02-21 16:33:00,563 - oe-selftest - INFO - Traceback (most recent call last):
  File "~/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "~/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 413, in test_sstate_sametune_samesigs
    self.assertCountEqual(files1, files2)
AssertionError: Element counts were not equal:
First has 1, Second has 0:  '~/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-initscripts/1.0-r155.do_package_qa.sigdata.117b713e2fc93f4fd96bcd812515d537'
First has 1, Second has 0:  '~/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-initscripts/1.0-r155.do_package_write_rpm.sigdata.e5417bbe33e6a226be69cdfc854583ae'
First has 1, Second has 0:  '~/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-sysvinit/2.88dsf-r14.do_package_write_rpm.sigdata.fa6830a1f6474dde350997cc84a47aa6'
First has 1, Second has 0:  '~/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-sysvinit/2.88dsf-r14.do_package_qa.sigdata.1cb239634f81d9861b1fba44febae2e2'
First has 0, Second has 1:  '~/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-initscripts/1.0-r155.do_package_qa.sigdata.9ed59a3ff3168c119e9e2a1b84920eed'
First has 0, Second has 1:  '~/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-initscripts/1.0-r155.do_package_write_rpm.sigdata.098579aa48494e34ffa8e3131491e9a5'
First has 0, Second has 1:  '~/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-sysvinit/2.88dsf-r14.do_package_write_rpm.sigdata.1160409047766e564611af437ca8e15a'
First has 0, Second has 1:  '~/autobuilder/yocto-worker/nightly-oe-selftest-debian9/build/build/tmp-sstatesamehash/stamps/x86-pokymllib32-linux/lib32-sysvinit/2.88dsf-r14.do_package_qa.sigdata.44d610245db83c87411fe6f2157d7ecb'

2018-02-21 16:33:00,564 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-02-21 16:33:00,564 - oe-selftest - INFO - Ran 317 tests in 34716.151s
2018-02-21 16:33:00,564 - oe-selftest - INFO - FAILED
2018-02-21 16:33:00,564 - oe-selftest - INFO -  (failures=1, skipped=1)

Summary:
The test_sstate_sametune_samesigs() test[1] is trying to ensure that the sstate checksums generated for two identical configurations are the same. It does so by generating the two lists and using Python unittest’s assertCountEqual() method[2] to compare the two and ensure that they have the same elements (regardless of order).

The AssertionError we’re seeing is because the two generated lists *don’t* contain the same elements, and therefore the test failed. The output (as produced by unittest) attempts to show how the two lists differ. I’d guess a change in either multilib or the sysvinit & initscripts recipes are the cause of the failure, although it’s strange we didn’t see this when building the release.

Joshua

1.	http://git.yoctoproject.org/clean/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py?h=rocko#n371
2.	https://docs.python.org/3.2/library/unittest.html#unittest.TestCase.assertCountEqual
Comment 1 Ross Burton 2018-03-08 04:51:33 UTC 
Does the QA autobuilder enable rm_work?  I see these errors with rm_work enabled but not if I disable it.
Comment 2 Aaron Chan 2018-03-08 18:27:22 UTC 
Hi Ross,

I'm not aware if there is such a feature in autobuilder and try to search for the keyword "rm_work" and it did not return me any results. Could you elaborate more and explain to me what "rm_work" does?
Comment 3 Ross Burton 2018-03-09 02:19:42 UTC 
You'd have to have added INHERIT += "rm_work" to the local.conf.  If you didn't do this then I can't replicate these failures...

Would you be able to re-run selftest on that autobuilder with the same SHA to see if they're reproducible?
Comment 4 Ross Burton 2018-03-09 06:20:09 UTC 
Chased this and Richard's encyclopaedic knowledge of previous commits remembered that we'd hit this in master.

Now backported to poky:rocko fe2c3c59e3c7c98d1683a0aa81868b8696b1116b.
Comment 5 Aaron Chan 2018-03-12 22:48:53 UTC 
Hi Ross Burton,

The following changes has been verified in our PG local autobuilder as:

http://centaurus01.png.intel.com:8010/builders/nightly-oe-selftest-debian9/builds/26/steps/Running%20oe-selftest/logs/stdio

On the ~/build/build/conf/auto.conf

INHERIT += "report-error"
DISTRO = "poky"
PACKAGE_CLASSES = "package_rpm"
BB_NUMBER_THREADS = "16"
PARALLEL_MAKE = "-j 16"
SDKMACHINE ?= "x86_64"
IMAGE_FSTYPES_append = "  tar.gz"
BB_TASK_NICE_LEVEL = "5"
BB_TASK_NICE_LEVEL_task-testimage = "0"
BB_TASK_IONICE_LEVEL = "2.7"
BB_TASK_IONICE_LEVEL_task-testimage = "2.1"
DL_DIR = "/srv/data/sources"
MACHINE = "qemux86-64"
PREMIRRORS = ""
ADTREPO = "http://adtrepo.yoctoproject.org//2.4.2"
QEMU_USE_KVM = "True"
SANITY_TESTED_DISTROS = ""
RPM_GPG_SIGN_CHUNK = "1"
INHERIT += "rm_work"