Bug 15400

Summary: patchtest: check that the dependency of the modified recipe do exist (even when using BBCLASSEXTEND)
Product: [Yocto Project Subprojects] Patchwork/Patchtest Reporter: Yoann Congal <yoann.congal>
Component: PatchtestAssignee: Simone Weiß <simone.p.weiss>
Status: RESOLVED NOTABUG QA Contact:
Severity: enhancement    
Priority: Medium+ CC: randy.macleod, simone.p.weiss, tgamblin, tim.orling
Version: unspecified   
Target Milestone: 5.0 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)

Description Yoann Congal 2024-02-13 10:49:14 UTC
Contributors often forget to check that the BBCLASSEXTENDed recipes build when patching. For example:
* https://lists.openembedded.org/g/openembedded-devel/topic/meta_python_patch/104328324?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,0,104328324,previd%3D1707815904911115207,nextid%3D1707526768688914271&previd=1707815904911115207&nextid=1707526768688914271
* https://lists.openembedded.org/g/openembedded-core/message/195225

This can be "easily" checked without building with just:
  bitbake --dry-run recipe recipe-native nativesdk-recipe
Comment 1 Randy MacLeod 2024-02-15 15:41:17 UTC
We need to check what the overhead of this would be and see if the patchtest  schedule has enough time to complete this useful check as currently scheduled.
Comment 2 Yoann Congal 2024-02-15 16:39:24 UTC
As a datapoint, without any sstate, a dry-run of a random recipe (~3000 tasks) takes a minute on my machine.

But, when I do this test manually, I can cancel it earlier : either the recipe is buildable and the dry-run starts or it is not and the dry-run don't start.

I've tried "bitbake --parse-only" but this succeeds even on unbuildable recipes.

Maybe we need a new bitbake option for "build task-graph then exit" ?
Comment 3 Simone Weiß 2024-02-15 18:50:58 UTC
Yes, this is of course a fact, maybe we could do this for a patch: If it adds a new recipe (also bbappends, incs, inherited new classes), or if it modifies its dependencies or BBCLASSEXTEND. That might not catch all cases like dependencies, though. I checked to get a rough idea how often this happens, the last 100 commits on a branch, it was around 5%. So we could most likely skip the test most of the time.

Besides that, I do not know how much overhead is OK. What is the current schedule?
Comment 4 Yoann Congal 2024-02-19 08:27:15 UTC
Note: A recent bitbake series might be helpful for this enhancement:
https://lists.openembedded.org/g/bitbake-devel/message/15912 :
> This will specifically request the bitbake dependecy data and do a quick
> sanity test of the results. This is the most vulnerable part of the 
> integration over releases.

If it can't be used directly, maybe it can inspire a solution?
Comment 5 Simone Weiß 2024-02-23 19:15:04 UTC
After thinking about this further:
1) The goal of patchtest is to ensure:
       - patch is well-formed
       - has appropriate metadata
       - can be merged
2) This is related to checking if a patch introduces build issues. If we start here with this, where to draw the line? This is out of scope for patchtest.
3)  Also the time for running bitbake is not all the testcase would add also the setup of the environment etc. The environment where the patchtests are run is also constraint resource wise (See: https://git.yoctoproject.org/meta-patchtest/tree/scripts/patchtest_periodic.run) We are already in a situation where we are unable to test all patches that are scheduled in the patchtests runs from time to time.

All in all: I would not add this to patchtest. This should be checked elsewhere is checked by building already.
I would hence propose to a) either close this issue or move it to another component
Comment 6 Yoann Congal 2024-02-23 22:13:08 UTC
(In reply to Simone Weiß from comment #5)
> All in all: I would not add this to patchtest. This should be checked
> elsewhere is checked by building already.
> I would hence propose to a) either close this issue or move it to another
> component

Agreed. This should not be put into patchtest and not now. Thanks for your analysis :)

But, I still think testing patch for existing dependency would be useful (it's hard to see in review, but easy to test) and quite fast (no actual build needed just parsing the whole recipe tree)
Comment 7 Simone Weiß 2024-02-23 23:26:40 UTC
What about adding it as a recommendation in the documentation for contributions? Do ppl use some scripts for patch-review? If so, if there are some common, could one semi-automate this there?
Comment 8 Simone Weiß 2024-02-24 00:25:28 UTC
Will add some hint in the docs on testing changes, before submitting, this won't hurt anyway.
Comment 9 Trevor Gamblin 2024-02-25 14:52:51 UTC
I definitely agree that adding it to the documentation as a recommendation is (In reply to Simone Weiß from comment #7)
> What about adding it as a recommendation in the documentation for
> contributions? Do ppl use some scripts for patch-review? If so, if there are
> some common, could one semi-automate this there?

Good idea. This also makes me wonder if we shouldn't have a little disclaimer in the patchtest replies as well, something short that hints at what patchtest doesn't check (e.g. successful build, ptests) which the author should test before submission?
Comment 10 Simone Weiß 2024-02-25 18:07:44 UTC
Created https://bugzilla.yoctoproject.org/show_bug.cgi?id=15412 to follow up on the documentation ideas discussed here.