Bug 4078

In addition to a parse check, I see us needing to have a slightly broader patch checking mechanism. Some other common mistakes we ought to check for with this:
* Tabs in python code
* Missing LIC_FILES_CHKSUM
* LIC_FILES_CHKSUM changed but no mention in commit message
* Files removed from SRC_URI but left around (of course, we'll need to check if other recipes refer to these)
* Missing Signed-off-by or Upstream-Status in patches
* Missing Signed-off-by in the overall commit message
* Short DESCRIPTION set instead of SUMMARY
* Recipe upgraded where SRC_URI points to something other than an SCM, but SRC_URI checksums not updated

We can probably think of other things but this would be a good start.

(In reply to comment #1)
> In addition to a parse check, I see us needing to have a slightly broader
> patch checking mechanism. Some other common mistakes we ought to check for
> with this:
> * Tabs in python code
> * Missing LIC_FILES_CHKSUM
> * LIC_FILES_CHKSUM changed but no mention in commit message
> * Files removed from SRC_URI but left around (of course, we'll need to check
> if other recipes refer to these)
> * Missing Signed-off-by or Upstream-Status in patches
> * Missing Signed-off-by in the overall commit message
> * Short DESCRIPTION set instead of SUMMARY
> * Recipe upgraded where SRC_URI points to something other than an SCM, but
> SRC_URI checksums not updated
> 
> We can probably think of other things but this would be a good start.

I recently (re)discovered oelint.bbclass which performs some similar tasks.

Indeed; one challenge though is that this needs to operate on the change being made as opposed to the current state. (Of course you could get the former from the latter by doing some kind of diff.)

This needs to work both standalone as well as in conjunction with what is developed for 8648

Implementing now, have a shell script to do the patch applying and parse check at http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/?h=franco%2Fprecheck, working together with Leo...

Have been collaborating with Leo, still not completely determined how some of the tests will be flowing but we have some common ground and basic tests that are nearly interchangeable across our scripts (the one for this bug, and the one for bug 8648 which is concerned with automated patch testing, integrated with patchwork). We still have more integration to do but are working closely.

Summary of tests we need, according to previous posts and considerations:

*Considerations* for different sets of tests:

a. **oe-selftest** tests
  - Required for these tests:
    - repo dir location
  - Tests:
    1. Bitbake parse (done ✓)

b. **Patch** file checks
  - Required for these tests:
    - list of mbox-formatted patches
  - Tests:
    1. Signed-off-by is present (done ✓)
    2. Upstream-Status is present (done ✓)
    3. Summary length is 70 chars or less (done ✓)
    4. Description is present
    5. Patches apply (checked by framework)

c. **Code** checks
  - Required for these tests:
    - list of mbox-formatted patches
    - list of files modified per patch
  - Tests:
    1. Tabs/whitespace in python files modified by patch

d. If **recipe** files modified by patch, recipe checks
  - Required for these tests:
    - list of mbox-formatted patches
    - list of files modified per patch
    - bitbake environment: use tinfoil? bitbake -e? new oe-selftest test?

  - Tests:
    1. SRC_URI - removed files, but files still around
    2. SRC_URI - points to something other than an SCM, but checksums not changed
    3. LIC_FILES_CHKSUM - missing
    4. LIC_FILES_CHKSUM - changed but commit message doesn't include "lic"

[1] link is another source of possible checks


[1] http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#CVE_Patches

Bug 9244 opened to address:

d. If **recipe** files modified by patch, recipe checks
  - Required for these tests:
    - list of mbox-formatted patches
    - list of files modified per patch
    - bitbake environment: use tinfoil? bitbake -e? new oe-selftest test?

  - Tests:
    1. SRC_URI - removed files, but files still around
    2. SRC_URI - points to something other than an SCM, but checksums not changed
    3. LIC_FILES_CHKSUM - missing
    4. LIC_FILES_CHKSUM - changed but commit message doesn't include "lic"

Made this bug depend on 8648 as it is being addressed by pt-suites which depends on standalone patchtest readiness (this bug doesn't really need for the testing to be automated, just for it to be available)

The tests called "pt-suites" along with the "patchtest" software cover the first part of this bug. The second part is covered by bug 8648.

I am setting this to NEEDINFO since I need to ask whether pt-suites should also follow a PDT process along with patchtest, and thus determine the resolution of this bug.

Alex,

When you set something to NEEDINFO, please change the owner of the bug to who you expect to supply the information.  It is not clear from your question who you expect to answer the question.

It was determined that this won't require a special process, will soon be available on the Yocto project git repo... But it still needs patchtest readiness, which in turn is depending on a new patchwork instance. Setting this to NEEDINFO and assigning to Michael as he's working on that instance (although technically this bug is for a simple manual testing, the same test suite developed here is also used through patchtest for automated testing)

Blocked on Patchwork upgrade in process.

This bug and 9244 are currently being developed, so taking ownership of it again because it does not depend on the new patchwork instance (9130) until is is completed.

This work is expected to be finished early 2.2 M2.

Moving back to M1. Any change on milestone must be discussed previously with the CCB.

All targeted tests are now implemented but not fully tested on 'live' patchwork series events. The reason to move it to M2 is to have this window to test incoming patches but results will be send to certain people (myself) instead of patchwork. This way, we can do changes into the suite and no spamming done to the real developer. Test suite is still not public.

Current status: pending checks are being merged (full list can be done by a search by 'patchtest-oe' with 2.2 M2 milestone) and tested. Several people involved on this effort and I am acting as a integrator. At the same time, new series from patchwork.yp.org are being fetched and tested. Bug is on track and expected to be resolved/fixed before indicated milestone.

+ This is an umbrella bug covering most of the checks mentioned (listed below)
+ Current pre-checks=4 (done before patch merge), current checks=18 (done after patch merge). Failed checks are the ones to be reported to the user
+ Checks are divided in four main areas: bitbake, patch, mbox and python.
+ The test suite is still not public
+ In order to split this bug, several bugs have been filed and most of it have been resolved (search for 'patchtest-oe')

Current pre-checks:

bitbake.parse.test_parse.BitbakeParse.pretest_bitbake_environment
bitbake.parse.test_parse.BitbakeParse.pretest_bitbake_environment_on_target
bitbake.parse.test_parse.BitbakeParse.pretest_bitbake_parse
python.lint.test_pylint.PyLint.pretest_pylint

Current checks:

patch.signed_off_by.test_signed_off_by.PatchSignedOffBy.test_signed_off_by_format
patch.signed_off_by.test_signed_off_by.PatchSignedOffBy.test_signed_off_by_presence
patch.upstream_status.test_upstream_status.PatchUpstreamStatus.test_upstream_status
patch.cve.test_cve.CVE.test_cve_presence_on_subject
patch.cve.test_cve.CVE.test_cve_tag_format
bitbake.parse.test_parse.BitbakeParse.test_bitbake_environment
bitbake.parse.test_parse.BitbakeParse.test_bitbake_environment_on_target
bitbake.parse.test_parse.BitbakeParse.test_bitbake_parse
mbox.signed_off_by.test_signed_off_by.SignedOffBy.test_signed_off_by_format
mbox.signed_off_by.test_signed_off_by.SignedOffBy.test_signed_off_by_presence
mbox.subject.test_subject.Subject.test_subject_format
mbox.subject.test_subject.Subject.test_subject_length
mbox.subject.test_subject.Subject.test_subject_presence
mbox.bugzilla.test_bugzilla.Bugzilla.test_bugzilla_entry_format
mbox.description.test_description.Description.test_description_presence
python.lint.test_pylint.PyLint.test_pylint
metadata.lic_files_chksum.test_lic_files_chksum.LicFilesChkSum.test_lic_files_chksum_modified_not_mentioned
metadata.lic_files_chksum.test_lic_files_chksum.LicFilesChkSum.test_lic_files_chksum_presence

I note above that it says this test suite is not public.

Has this changed, if so where will it be uploaded to?

(In reply to comment #20)
> I note above that it says this test suite is not public.
> 
> Has this changed, if so where will it be uploaded to?

During 2.3M4 should be opened. I am just finishing the last actions required internally so we can open source it. If you want to see the latest on this area, CC on this one https://bugzilla.yoctoproject.org/show_bug.cgi?id=9457

(In reply to comment #21)
> (In reply to comment #20)
> > I note above that it says this test suite is not public.
> > 
> > Has this changed, if so where will it be uploaded to?
> 
> During 2.3M4 should be opened. I am just finishing the last actions required
> internally so we can open source it. If you want to see the latest on this
> area, CC on this one https://bugzilla.yoctoproject.org/show_bug.cgi?id=9457

I meant **2.2 M4**, not 2.3. (thanks JoseP for noticing it)