The checkpkg selftest has to maintain a separate list of recipes that can't pass checkpkg's upgrade testing. This isn't scalable to other layers as there is only a single exception list. The checkpkg logic should have a variable that recipes can set to exclude themselves from the upgrade check. This will speed up checkpkg, and allow tooling to tell between "couldn't tell if there is an upgrade" and "we cannot tell if there is an upgrade" (where the former could be fixed or be transient network failure).
I think there is a need for two per-recipe variables actually: UPSTREAM_CHECK_UNKNOWN - set if the upstream check fails reliably, e.g. absent git tags, or weird versions used on our or on upstream side. The upstream check test will treat this the same way it treats the exception list now - mismatches both ways will be reported as failures. UPSTREAM_CHECK_UNRELIABLE - set if the upstream check cannot be reliably performed due to transient network failures, or server behaving weirdly. This one should be used sparingly, as it completely excludes a recipe from upstream checking. The upstream check test will never fail. Agree?
That seems reasonable, yes.