Bug 15269

Summary: patchtest test_lic_files_chksum_modified_not_mentioned is inconsistent between local and automated runs
Product: [Yocto Project Subprojects] Patchwork/Patchtest Reporter: Trevor Gamblin <tgamblin>
Component: Patchtest TestsuiteAssignee: Trevor Gamblin <tgamblin>
Status: NEW --- QA Contact:
Severity: normal    
Priority: Medium+ CC: randy.macleod
Version: 5.0   
Target Milestone: 6.2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
oe-core patch that was tested by patchtest.yocto.io none

Description Trevor Gamblin 2023-10-27 21:05:52 UTC
Created attachment 4982 [details]
oe-core patch that was tested by patchtest.yocto.io

test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned is providing different results between host and guest modes, for certain patches. Attached is an example from the mailing list that reported FAIL in the automated system (see attached .testresult file) but consistently reports PASS when tested with a local setup of patchtest.
Comment 1 Trevor Gamblin 2023-10-27 21:07:00 UTC
Failing result because I couldn't get the .testresult file to attach properly:

Testing patch /home/patchtest/share/mboxes/kirkstone-1-2-linux-firmware-create-separate-package-for-cirrus-and-cnm-firmwares.patch

FAIL: test lic files chksum modified not mentioned: LIC_FILES_CHKSUM changed without "License-Update:" tag and description in commit message (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)

PASS: pretest lic files chksum modified not mentioned (test_metadata.TestMetadata.pretest_lic_files_chksum_modified_not_mentioned)
PASS: pretest src uri left files (test_metadata.TestMetadata.pretest_src_uri_left_files)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test src uri left files (test_metadata.TestMetadata.test_src_uri_left_files)

SKIP: pretest pylint: No python related patches, skipping test (test_python_pylint.PyLint.pretest_pylint)
SKIP: test CVE presence in commit message: No new patches introduced (test_mbox.TestMbox.test_cve_presence_in_commit_message)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test pylint: No python related patches, skipping test (test_python_pylint.PyLint.test_pylint)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
SKIP: test target mailing list: Series merged, no reason to check other mailing lists (test_mbox.TestMbox.test_target_mailing_list)
Comment 2 Trevor Gamblin 2023-10-27 21:11:47 UTC
Tested with a virtualenv to make sure there were no extra modules polluting my Python setup on my build machine, but that didn't change anything - the test still reported PASS on the attached patch.

Only immediate delta between the Yocto environment and my host that I can think of is the Python version itself (Yocto -> 3.11.5, F38 -> 3.11.6)...