Bug 14707 - do_unpack may access network in FetchMethod.latest_revision
Summary: do_unpack may access network in FetchMethod.latest_revision
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 5.99
Hardware: x86 Multiple
: High normal
Target Milestone: 4.0 M3
Assignee: Pavel Zhukov
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2022-01-28 15:48 UTC by Peter Hoyes
Modified: 2022-02-22 17:26 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Hoyes 2022-01-28 15:48:17 UTC
With the latest master (where network access is disabled in do_unpack):

Reproduction steps:
 * Create a recipe whose SRCREV is *not* a SHA (e.g. a Git tag)
 * Build the recipe
 * Make a change to the recipe, such that do_unpack is re-triggered but do_fetch is not
 * Build the recipe again. do_unpack fails as it tries to run `git ls-remote` to resolve the tag.

The offending function call is at https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/git.py#n256

I understand that it is best practice to use a SHA in the SRCREV so that no `ls-remote` is necessary, but this does not seem to be mandatory at the moment.

Could do_unpack fetch the latest revision from the clone in the downloads directory instead?
Comment 1 Randy MacLeod 2022-02-03 15:40:08 UTC
Pavel will take this once this he gets a BZ account.
Comment 2 Pavel Zhukov 2022-02-05 09:08:51 UTC
Hello,

Thank you for reporting this issue. 
Bitbake caches srcrev information in PERSISTENT_DIR and it should persist between builds (according to https://www.yoctoproject.org/docs/2.0.3/bitbake-user-manual/bitbake-user-manual.html ). In fact this behavior is controlled by BB_SRCREV_POLICY and the default policy is "clean" that means the cache is being cleaned each time (https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/__init__.py#n511)

To mitigate the issue BB_SRCREV_POLICY can be set to "cache" in local.conf. In that case bitbake uses URI_HEADREVS store to resolve revisions and the issue doesn't occur. 

latest_revisions() for git fetcher can be improved a bit to use rev-parse in case if network is not available but this will not work in all cases (rebasesable repos for example).
Comment 3 Pavel Zhukov 2022-02-22 17:26:18 UTC
This is fixed in https://git.openembedded.org/bitbake/commit/?id=4b5eed1626709ef3dc06b32fd55d40a2a6edd179
Closing