Bug 13815 - Reproducibility test failure occurs for deb even if package_deb is not set
Summary: Reproducibility test failure occurs for deb even if package_deb is not set
Status: RESOLVED FIXED
Alias: None
Product: Functional (self) Testing
Classification: QA/Testing
Component: oe-selftest (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 3.2 M2
Assignee: Trevor Gamblin
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2020-02-25 20:45 UTC by Trevor Gamblin
Modified: 2020-06-04 12:10 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Trevor Gamblin 2020-02-25 20:45:56 UTC 
When running reproducible build tests with "oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds" and PACKAGE_CLASSES ?= "package_ipk" as set by default in local.conf, the reproducibility test fails while looking for a path for .deb files:

2020-02-24 11:38:22,777 - oe-selftest - INFO - Adding layer libraries:
2020-02-24 11:38:22,778 - oe-selftest - INFO -  /ala-lpggp31/tgamblin/yocto/oe-core.git/meta/lib
2020-02-24 11:38:22,778 - oe-selftest - INFO -  /ala-lpggp31/tgamblin/yocto/oe-core.git/meta-selftest/lib
2020-02-24 11:38:22,778 - oe-selftest - INFO -  /ala-lpggp31/tgamblin/yocto/meta-oe.git/meta-oe/lib
2020-02-24 11:38:22,779 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2020-02-24 11:38:26,090 - oe-selftest - INFO - test_reproducible_builds (reproducible.ReproducibleTests)
2020-02-24 12:09:11,043 - oe-selftest - INFO - Reproducibility summary for ipk: same=7883 different=0 missing=0 total=7883
2020-02-24 12:09:11,072 - oe-selftest - INFO -  ... ======================================================================
2020-02-24 12:09:11,072 - oe-selftest - INFO - ERROR: test_reproducible_builds (reproducible.ReproducibleTests) (package_class='deb')
2020-02-24 12:09:11,072 - oe-selftest - INFO - ----------------------------------------------------------------------
2020-02-24 12:09:11,072 - oe-selftest - INFO - Traceback (most recent call last):
  File "/ala-lpggp31/tgamblin/yocto/oe-core.git/meta/lib/oeqa/selftest/cases/reproducible.py", line 204, in test_reproducible_builds
    result = self.compare_packages(deploy_A, deploy_B, diffutils_sysroot)
  File "/ala-lpggp31/tgamblin/yocto/oe-core.git/meta/lib/oeqa/selftest/cases/reproducible.py", line 115, in compare_packages
    os.chdir(test_dir)
FileNotFoundError: [Errno 2] No such file or directory: '/ala-lpggp31/tgamblin/yocto/build-st/tmp-glibc/deploy/deb'

2020-02-24 12:09:11,072 - oe-selftest - INFO - ----------------------------------------------------------------------
2020-02-24 12:09:11,073 - oe-selftest - INFO - Ran 1 test in 1844.983s
2020-02-24 12:09:11,073 - oe-selftest - INFO - FAILED
2020-02-24 12:09:11,073 - oe-selftest - INFO -  (errors=1)
2020-02-24 12:09:13,964 - oe-selftest - INFO - RESULTS:
2020-02-24 12:09:13,966 - oe-selftest - INFO - RESULTS - reproducible.ReproducibleTests.test_reproducible_builds: UNKNOWN (1844.98s)
2020-02-24 12:09:13,967 - oe-selftest - INFO - SUMMARY:
2020-02-24 12:09:13,967 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 1846.735s
2020-02-24 12:09:13,968 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=0, errors=1)
tgamblin@ala-lpggp3$

This looks like a problem with the way package_classes is defined in meta/lib/oeqa/selftest/cases/reproducible.py, line 81:

package_classes = ['deb', 'ipk']

This doesn't fail for the autobuilder as it adds package_deb to its builds, and I can confirm that if I manually change PACKAGE_CLASSES before running oe-selftest, then the reproducibility test passes without issue.

What should the appropriate behavior be here? It's easy enough to change that line, but is that what we want? It doesn't seem like we'd want to change how the default local.conf looks, so I'm assuming that we maybe want a check to see which package types are enabled before the test runs?
Comment 1 Trevor Gamblin 2020-03-05 15:30:25 UTC 
Test behaves as expected if I use "oe-selftest -r reproducible -j 1" instead of "oe-selftest -r reproducible". 

Likely tied to https://git.openembedded.org/openembedded-core/commit/?id=50238b0717b04e0a1fa69d618e8c8aa8445a80b0 as previously discussed.

Would be good if some others could try reproducing the same failures on master.
Comment 2 Trevor Gamblin 2020-03-05 16:51:18 UTC 
Note that when encountering this issue, I have the oe-core and bitbake layers, as well as the build directory specified to oe-init-build-env, at the same level.
Comment 3 Trevor Gamblin 2020-03-05 17:42:54 UTC 
Reproducibility works as expected for me if using *poky* - it's only the use of oe-core + bitbake layers independently where I'm seeing the problem.
Comment 4 Paul Barker 2020-06-03 13:09:58 UTC 
Could you see if this patch fixes this issue? https://lists.openembedded.org/g/openembedded-core/message/139195
Comment 5 Trevor Gamblin 2020-06-04 04:37:41 UTC 
Ran a build last night. It appears to work with that patch. Nice!
Comment 6 Trevor Gamblin 2020-06-04 12:10:06 UTC 
Now that the patch is in master and it's been tested successfully, this can be closed. Thanks Paul.