Bug 15202

Summary:	"Replicating a Build Offline" does not work as documented
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Stephan Mühlstrasser <stephan.muehlstrasser>
Component:	configuration	Assignee:	Unassigned <unassigned>
Status:	RESOLVED INVALID	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	alanp, hongxu.jia, liezhi.yang, poky.bs.watcher, poky.watcher, randy.macleod, yoann.congal
Version:	4.2.2
Target Milestone:	5.0 M1
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know

Description Stephan Mühlstrasser 2023-08-28 06:45:49 UTC

The Yocto documentation describes a procedure to "Replicate a Build Offline":

https://docs.yoctoproject.org/4.2.2/dev-manual/building.html?highlight=bb_no_network#replicating-a-build-offline

The documentation says that it is sufficient to keep the tar files creates by setting 'BB_GENERATE_MIRROR_TARBALLS = "1"' and that it is possible to delete any Git or other SCM repositories from the downloads directory, but this is not working. Regarding Git the documentation specifically says that it is possible to delete all ${DL_DIR}/git2/* directories.

When the removal of the ${DL_DIR}/git2/* directories is skipped in the steps below, the offline build succeeds.

How to reproduce:

~/src$ git clone -b mickledore-4.2.2 https://git.yoctoproject.org/poky
Cloning into 'poky'...
...
~/src$ cd poky/
~/src/poky$ . oe-init-build-env

# Now edited conf/local.conf and added: BB_GENERATE_MIRROR_TARBALLS = "1".
# Tarballs are created as expected. Git directories are removed from the download directory.

~/src/poky/build$ bitbake core-image-minimal --runonly=fetch
...
~/src/poky/build$ rm -rf downloads/git2/*

# Now I edit conf/local.conf again and add the following according to the documentation
# SOURCE_MIRROR_URL ?= "file://${TOPDIR}/downloads/"
# INHERIT += "own-mirrors"
# BB_NO_NETWORK = "1"

~/src/poky/build$ tail -10 conf/local.conf
# CONF_VERSION is increased each time build/conf/ changes incompatibly and is used to
# track the version of this file when it was generated. This can safely be ignored if
# this doesn't mean anything to you.
CONF_VERSION = "2"

BB_GENERATE_MIRROR_TARBALLS = "1"
SOURCE_MIRROR_URL ?= "file://${TOPDIR}/downloads/"
INHERIT += "own-mirrors"
BB_NO_NETWORK = "1"

# An attempt to run the build fails early with these error messages:

~/src/poky/build$ bitbake core-image-minimal
Loading cache: 100% |                                                                                   | ETA:  --:--:--
Loaded 0 entries from dependency cache.
Parsing recipes: 100% |##################################################################################| Time: 0:00:07
Parsing of 899 .bb files complete (0 cached, 899 parsed). 1799 targets, 49 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
 
Build Configuration:
BB_VERSION           = "2.4.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "4.2.2"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta
meta-poky
meta-yocto-bsp       = "HEAD:6e17b3e644ca15b8b4afd071ccaa6f172a0e681a"
 
Initialising tasks: 100% |###############################################################################| Time: 0:00:01
Sstate summary: Wanted 1074 Local 0 Mirrors 0 Missed 1074 Current 0 (0% match, 0% complete)
NOTE: Executing Tasks
ERROR: gnu-config-native-20230216+gitAUTOINC+63acb96f92-r0 do_unpack: Bitbake Fetcher Error: UnpackError('No up to date source found: clone directory not available or not up to date: /home/stm/src/poky/build/downloads/git2/git.savannah.gnu.org.git.config.git; shallow clone not enabled', 'git://git.savannah.gnu.org/git/config.git;protocol=https;branch=master')
ERROR: Logfile of failure stored in: /home/stm/src/poky/build/tmp/work/x86_64-linux/gnu-config-native/20230216+gitAUTOINC+63acb96f92-r0/temp/log.do_unpack.19635
ERROR: Task (virtual:native:/home/stm/src/poky/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_unpack) failed with exit code '1'
ERROR: pkgconfig-native-0.29.2+gitAUTOINC+d97db4fae4-r0 do_unpack: Bitbake Fetcher Error: UnpackError('No up to date source found: clone directory not available or not up to date: /home/stm/src/poky/build/downloads/git2/gitlab.freedesktop.org.pkg-config.pkg-config.git; shallow clone not enabled', 'git://gitlab.freedesktop.org/pkg-config/pkg-config.git;branch=master;protocol=https')
ERROR: Logfile of failure stored in: /home/stm/src/poky/build/tmp/work/x86_64-linux/pkgconfig-native/0.29.2+gitAUTOINC+d97db4fae4-r0/temp/log.do_unpack.19647
ERROR: Task (virtual:native:/home/stm/src/poky/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb:do_unpack) failed with exit code '1'
NOTE: Tasks Summary: Attempted 44 tasks of which 22 didn't need to be rerun and 2 failed.
 
Summary: 2 tasks failed:
  virtual:native:/home/stm/src/poky/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_unpack
  virtual:native:/home/stm/src/poky/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb:do_unpack
Summary: There were 2 ERROR messages, returning a non-zero exit code.

Comment 1 Randy MacLeod 2023-08-31 15:25:06 UTC

Stephan,
Thanks for the bug report.
Are you able to test this on master?

We don't have anyone available to work on the issue right not but
the Robert and/or Hongxu may have some insight into the issue so I've CCed them.

Comment 2 Stephan Mühlstrasser 2023-09-01 06:48:06 UTC

(In reply to Randy MacLeod from comment #1)
> Stephan,
> Thanks for the bug report.

Thanks for looking into this.

> Are you able to test this on master?

I repeated the steps from the description, with the only difference that I checked out master:

~/src$ git clone -b master https://git.yoctoproject.org/poky

This is the commit that I got:

commit bdc74e5ff3c16fed24829fd494e5860754c08778 (HEAD -> master, origin/master, origin/HEAD)
Author: Etienne Cordonnier <ecordonnier@snap.com>
Date:   Tue Aug 29 13:45:47 2023 +0200

    migration-guides: system-conf -> systemd-conf

The error is reproducible in the same manner as from the mickledore branch.

Comment 3 Randy MacLeod 2023-10-30 15:29:15 UTC

Bulk move to 5.0 M1. -- Randy

Comment 4 Alan Perry 2023-11-08 23:09:31 UTC

I was able to reproduce the symptoms using the steps indicated.

However, this repro omits a step from the cited Yocto documentation, specifically:

"2. Start With a Clean Build: You can start with a clean build by removing the ${TMPDIR} directory or using a new Build Directory."

After I removed the ${TMPDIR} directory and re-ran bitbake, the problem did not occur. However bitbake later failed in do_package_write_rpm for openssl.

Comment 5 Stephan Mühlstrasser 2023-11-10 13:51:39 UTC

(In reply to Alan Perry from comment #4)
> However, this repro omits a step from the cited Yocto documentation,
> specifically:
> 
> "2. Start With a Clean Build: You can start with a clean build by removing
> the ${TMPDIR} directory or using a new Build Directory."

You are right, thanks for pointing that out. I had somehow assumed that I am already in the "Clean Build" state as the packages were only fetched and no actual build of the packages has been performed.

When I remove the ${TMPDIR} directory before invoking "bitbake core-image-minimal" the build works as expected without any network access. I did not see the problem with do_package_write_rpm for openssl. I'm building the Mickledore release.

I'm sorry for the confusion, this bug report is invalid.