Bug 14822

Created attachment 4871 [details]
Contains log.do_fetch.0 and log.do_fetch.1

---------------Summary---------------
If some packages on local machine for offline build is damaged, the log.do_fetch will change when running the same command.

---------------Details step to reproduce---------------
1. According to the instructions on https://docs.yoctoproject.org/dev-manual/common-tasks.html#replicating-a-build-offline, prepare an offline premirrors on the local machine, the path is <offline_downloads>.
2. Create a new build directory by `source oe-init-build-env <my_build_path>`. Make sure <my_build_path> is not under git repository, i.e. its ancestor diretories and itself does not contain .git directory.
3. Edit conf/local.conf in <my_build_path>. Add the following lines to the default one.
###############
BB_GENERATE_MIRROR_TARBALLS = "1"
INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "file://<offline_downloads>"
#The location where nativesdk comes from
UNINATIVE_URL = "file://<offline_downloads>/uninative/9bfc4c970495b3716b2f9e52c4df9f968c02463a9a95000f6657fbc3fde1f098/"
UNINATIVE_TARBALL = "x86_64-nativesdk-libc-3.6.tar.xz"
UNINATIVE_CHECKSUM[x86_64] = "9bfc4c970495b3716b2f9e52c4df9f968c02463a9a95000f6657fbc3fde1f098"
PREMIRRORS:prepend = "\
 git://.*/.* http://example.com/ \n \
 ftp://.*/.* http://example.com/ \n \
 http://.*/.* http://example.com/ \n \
 https://.*/.* http://example.com/ \n"
BB_NO_NETWORK = "1"
###############
4. Change the content of git2_github.com.besser82.libxcrypt.git.tar.gz in <offline_downloads> so that it is not a valid tar.gz format.
5. Run the command `bitbake core-image-minimal --runall=unpack`. The content of log.do_fetch is like log.do_fetch.0. This is some analysis for the log.
The download(self, urls=None) method of Fetch class calls m.try_premirror(ud, self.d) to check whether to try premirrors. Here, m is Git class type.  Because BB_FETCH_PREMIRRORONLY is false and downloads/git2/github.com.besser82.libxcrypt.git/ does not exist, try_premirror(self, ud, d) of Git class return true.
The download(self, urls=None) method of Fetch class calls the method try_mirrors(self, fetch, urldata, d, mirrors, check=False) of FetchMethod class because Git class dose not have try_mirrors().
The static method try_mirrors(fetch, d, origud, mirrors, check = False) is called.
The static method try_mirror_url(fetch, origud, ud, ld, check = False) is called.
In line 64, try_mirror_url() calls origud.method.download(origud, ld). Here, origud is Git class type. I add some log messages to debug this issue.
In line 65, the download(self, ud, d) method of Git class calls runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir) to decompress the offline package.
In line 69~71, runfetchcmd() fails because the file is an invalid tar.gz format.
In line 73,  bb.fetch2.BBFetchException is thrown by origud.method.download().
The method try_mirror_url() in __init__.py calls ud.method.clean(ud, ld). Note that ud is Local class here, and origud is Git class.
The empty directory downloads/git2/github.com.besser82.libxcrypt.git/ will remain in <my_build_path>.

6. Run the command `bitbake core-image-minimal --runall=unpack` again. The content of log.do_fetch is like log.do_fetch.1. This is some analysis for the log.
The download(self, urls=None) method of Fetch class calls m.try_premirror(ud, self.d) to check whether to try premirrors. Here, m is Git class type.  Because BB_FETCH_PREMIRRORONLY is false and downloads/git2/github.com.besser82.libxcrypt.git/ already exist, try_premirror(self, ud, d) of Git class return false.
In line 8, the download(self, urls=None) method of Fetch class calls m.download(ud, self.d) of Git class to download from Git server.
In line 16~17, the download(self, urls=None) method of Fetch class calls m.clean(ud, self.d) to remove downloads/git2/github.com.besser82.libxcrypt.git/.

7. Repeat the command `bitbake core-image-minimal --runall=unpack`. The content of log.do_fetch for git2_github.com.besser82.libxcrypt.git.tar.gz will switch between log.do_fetch.0 and log.do_fetch.1.

---------------Simple conclusion---------------
Currently, the static method try_mirror_url(fetch, origud, ud, ld, check = False) just calls ud.method.clean() for exceptions. In my case, origud.method.download() is called and leaves an empty directory. And it affects the second command.
It may be necessary for try_mirror_url() to call origud.method.clean() if some exceptions is thrown by origud.method.download().
If I make such changes to try_mirror_url(), the log.do_fetch will be always like log.do_fetch.0.

Note:
- In step 2, it is important that the build directory <my_build_path> is not a descendant of a git repository. Otherwise, the behaviour will be different.
- In step 4, I just choose the package for my example. The behaviour is same if another package using Git protocol is chosen.
- In step 4, alternatively I could change the permission of the file git2_github.com.besser82.libxcrypt.git.tar.gz to un-readable.
- The use case is unusual. Although we should take care of the offline packages and keep them consistent, it is still possible to corrupt them by accident. The log file should help find the root cause.