OS: Ubuntu 18.04.04 Poky version: 3.2+snapshot (last sync to master branch at commit-sha: ee867453ff2116620c07f0bb3bea725d7aa60731) https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ee867453ff2116620c07f0bb3bea725d7aa60731 (committed on 2020-12-31) Bitbake output: Sun Jan 3 07:08:37 2021: NOTE: recipe brcm-nic-mgr-gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0: task do_ar_patched: Started Sun Jan 3 07:08:37 2021: NOTE: recipe brcm-nic-mgr-gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0: task do_populate_lic: Started Sun Jan 3 07:08:37 2021: NOTE: recipe brcm-nic-mgr-gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0: task do_deploy_source_date_epoch: Succeeded Sun Jan 3 07:08:37 2021: NOTE: Running task 6033 of 6089 (mc:ns3:/tmp/yocto_builds/20473263_nxs_ns3-overlake/poky/build/../meta-brcm/meta-datacenter/recipes-core/nic-mgr/brcm-nic-mgr.bb:do_configure) Sun Jan 3 07:08:37 2021: NOTE: recipe brcm-nic-mgr-gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0: task do_populate_lic: Succeeded Sun Jan 3 07:08:37 2021: NOTE: recipe brcm-nic-mgr-gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0: task do_configure: Started Sun Jan 3 07:08:39 2021: ERROR: mc:ns3:brcm-nic-mgr-gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0 do_ar_patched: Error executing a python function in exec_python_func() autogenerated: Sun Jan 3 07:08:39 2021: Sun Jan 3 07:08:39 2021: The stack trace of python calls that resulted in this exception/failure was: Sun Jan 3 07:08:39 2021: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> Sun Jan 3 07:08:39 2021: 0001: Sun Jan 3 07:08:39 2021: *** 0002:do_ar_patched(d) Sun Jan 3 07:08:39 2021: 0003: Sun Jan 3 07:08:39 2021: File: '/tmp/yocto_builds/20473263_nxs_ns3-overlake/poky/build/../meta-brcm/classes/sparse-checkout.bbclass', lineno: 30, function: do_ar_patched Sun Jan 3 07:08:39 2021: 0026: src_dir = d.getVar('ARCHIVER_SRCDIR', True) Sun Jan 3 07:08:39 2021: 0027: if not src_dir: Sun Jan 3 07:08:39 2021: 0028: src_dir = os.path.join(d.getVar('WORKDIR', True),'git') Sun Jan 3 07:08:39 2021: 0029: ar_outdir = d.getVar('ARCHIVER_OUTDIR') Sun Jan 3 07:08:39 2021: *** 0030: create_tarball(d, src_dir , '', ar_outdir) Sun Jan 3 07:08:39 2021: 0031:} Sun Jan 3 07:08:39 2021: 0032: Sun Jan 3 07:08:39 2021: 0033:def get_allowed_list(d): Sun Jan 3 07:08:39 2021: 0034: import json Sun Jan 3 07:08:39 2021: File: '/tmp/yocto_builds/20473263_nxs_ns3-overlake/poky/build/../meta/classes/archiver.bbclass', lineno: 465, function: create_tarball Sun Jan 3 07:08:39 2021: 0461: bb.note('archiver_exclude_regex=%s' % archiver_exclude_regex) Sun Jan 3 07:08:39 2021: 0462: Sun Jan 3 07:08:39 2021: 0463: bb.note('Creating %s' % tarname) Sun Jan 3 07:08:39 2021: 0464: tar = tarfile.open(tarname, 'w:gz') Sun Jan 3 07:08:39 2021: *** 0465: tar.add(srcdir, arcname=os.path.basename(srcdir), filter=lambda tarinfo: None if re.match(archiver_exclude_regex, tarinfo.name) or not exclude_useless_paths(tarinfo) else tarinfo) Sun Jan 3 07:08:39 2021: 0466: tar.close() Sun Jan 3 07:08:39 2021: 0467: Sun Jan 3 07:08:39 2021: 0468:# creating .diff.gz between source.orig and source Sun Jan 3 07:08:39 2021: 0469:def create_diff_gz(d, src_orig, src, ar_outdir): Sun Jan 3 07:08:39 2021: File: '/usr/lib/python3.6/tarfile.py', lineno: 1961, function: add Sun Jan 3 07:08:39 2021: 1957: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1958: if recursive: Sun Jan 3 07:08:39 2021: 1959: for f in os.listdir(name): Sun Jan 3 07:08:39 2021: 1960: self.add(os.path.join(name, f), os.path.join(arcname, f), Sun Jan 3 07:08:39 2021: *** 1961: recursive, exclude, filter=filter) Sun Jan 3 07:08:39 2021: 1962: Sun Jan 3 07:08:39 2021: 1963: else: Sun Jan 3 07:08:39 2021: 1964: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1965: Sun Jan 3 07:08:39 2021: File: '/usr/lib/python3.6/tarfile.py', lineno: 1961, function: add Sun Jan 3 07:08:39 2021: 1957: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1958: if recursive: Sun Jan 3 07:08:39 2021: 1959: for f in os.listdir(name): Sun Jan 3 07:08:39 2021: 1960: self.add(os.path.join(name, f), os.path.join(arcname, f), Sun Jan 3 07:08:39 2021: *** 1961: recursive, exclude, filter=filter) Sun Jan 3 07:08:39 2021: 1962: Sun Jan 3 07:08:39 2021: 1963: else: Sun Jan 3 07:08:39 2021: 1964: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1965: Sun Jan 3 07:08:39 2021: File: '/usr/lib/python3.6/tarfile.py', lineno: 1961, function: add Sun Jan 3 07:08:39 2021: 1957: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1958: if recursive: Sun Jan 3 07:08:39 2021: 1959: for f in os.listdir(name): Sun Jan 3 07:08:39 2021: 1960: self.add(os.path.join(name, f), os.path.join(arcname, f), Sun Jan 3 07:08:39 2021: *** 1961: recursive, exclude, filter=filter) Sun Jan 3 07:08:39 2021: 1962: Sun Jan 3 07:08:39 2021: 1963: else: Sun Jan 3 07:08:39 2021: 1964: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1965: Sun Jan 3 07:08:39 2021: File: '/usr/lib/python3.6/tarfile.py', lineno: 1961, function: add Sun Jan 3 07:08:39 2021: 1957: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1958: if recursive: Sun Jan 3 07:08:39 2021: 1959: for f in os.listdir(name): Sun Jan 3 07:08:39 2021: 1960: self.add(os.path.join(name, f), os.path.join(arcname, f), Sun Jan 3 07:08:39 2021: *** 1961: recursive, exclude, filter=filter) Sun Jan 3 07:08:39 2021: 1962: Sun Jan 3 07:08:39 2021: 1963: else: Sun Jan 3 07:08:39 2021: 1964: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1965: Sun Jan 3 07:08:39 2021: File: '/usr/lib/python3.6/tarfile.py', lineno: 1961, function: add Sun Jan 3 07:08:39 2021: 1957: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1958: if recursive: Sun Jan 3 07:08:39 2021: 1959: for f in os.listdir(name): Sun Jan 3 07:08:39 2021: 1960: self.add(os.path.join(name, f), os.path.join(arcname, f), Sun Jan 3 07:08:39 2021: *** 1961: recursive, exclude, filter=filter) Sun Jan 3 07:08:39 2021: 1962: Sun Jan 3 07:08:39 2021: 1963: else: Sun Jan 3 07:08:39 2021: 1964: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1965: Sun Jan 3 07:08:39 2021: File: '/usr/lib/python3.6/tarfile.py', lineno: 1961, function: add Sun Jan 3 07:08:39 2021: 1957: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1958: if recursive: Sun Jan 3 07:08:39 2021: 1959: for f in os.listdir(name): Sun Jan 3 07:08:39 2021: 1960: self.add(os.path.join(name, f), os.path.join(arcname, f), Sun Jan 3 07:08:39 2021: *** 1961: recursive, exclude, filter=filter) Sun Jan 3 07:08:39 2021: 1962: Sun Jan 3 07:08:39 2021: 1963: else: Sun Jan 3 07:08:39 2021: 1964: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1965: Sun Jan 3 07:08:39 2021: File: '/usr/lib/python3.6/tarfile.py', lineno: 1961, function: add Sun Jan 3 07:08:39 2021: 1957: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1958: if recursive: Sun Jan 3 07:08:39 2021: 1959: for f in os.listdir(name): Sun Jan 3 07:08:39 2021: 1960: self.add(os.path.join(name, f), os.path.join(arcname, f), Sun Jan 3 07:08:39 2021: *** 1961: recursive, exclude, filter=filter) Sun Jan 3 07:08:39 2021: 1962: Sun Jan 3 07:08:39 2021: 1963: else: Sun Jan 3 07:08:39 2021: 1964: self.addfile(tarinfo) Sun Jan 3 07:08:39 2021: 1965: Sun Jan 3 07:08:39 2021: File: '/usr/lib/python3.6/tarfile.py', lineno: 1938, function: add Sun Jan 3 07:08:39 2021: 1934: Sun Jan 3 07:08:39 2021: 1935: self._dbg(1, name) Sun Jan 3 07:08:39 2021: 1936: Sun Jan 3 07:08:39 2021: 1937: # Create a TarInfo object from the file. Sun Jan 3 07:08:39 2021: *** 1938: tarinfo = self.gettarinfo(name, arcname) Sun Jan 3 07:08:39 2021: 1939: Sun Jan 3 07:08:39 2021: 1940: if tarinfo is None: Sun Jan 3 07:08:39 2021: 1941: self._dbg(1, "tarfile: Unsupported type %r" % name) Sun Jan 3 07:08:39 2021: 1942: return Sun Jan 3 07:08:39 2021: File: '/usr/lib/python3.6/tarfile.py', lineno: 1807, function: gettarinfo Sun Jan 3 07:08:39 2021: 1803: # Use os.stat or os.lstat, depending on platform Sun Jan 3 07:08:39 2021: 1804: # and if symlinks shall be resolved. Sun Jan 3 07:08:39 2021: 1805: if fileobj is None: Sun Jan 3 07:08:39 2021: 1806: if hasattr(os, "lstat") and not self.dereference: Sun Jan 3 07:08:39 2021: *** 1807: statres = os.lstat(name) Sun Jan 3 07:08:39 2021: 1808: else: Sun Jan 3 07:08:39 2021: 1809: statres = os.stat(name) Sun Jan 3 07:08:39 2021: 1810: else: Sun Jan 3 07:08:39 2021: 1811: statres = os.fstat(fileobj.fileno()) Sun Jan 3 07:08:39 2021: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/yocto_builds/20473263_nxs_ns3-overlake/poky/build/tmp/work/aarch64-poky-linux/brcm-nic-mgr/gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0/git/bnxt-ctl/main/Cumulus/util/bnxt-ctl/autom4te.cache/output.1t' Sun Jan 3 07:08:39 2021: Sun Jan 3 07:08:39 2021: ERROR: Logfile of failure stored in: /tmp/yocto_builds/20473263_nxs_ns3-overlake/poky/build/tmp/work/aarch64-poky-linux/brcm-nic-mgr/gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0/temp/log.do_ar_patched.35820 Sun Jan 3 07:08:39 2021: NOTE: recipe brcm-nic-mgr-gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0: task do_ar_patched: Failed Sun Jan 3 07:08:39 2021: ERROR: Task (mc:ns3:/tmp/yocto_builds/20473263_nxs_ns3-overlake/poky/build/../meta-brcm/meta-datacenter/recipes-core/nic-mgr/brcm-nic-mgr.bb:do_ar_patched) failed with exit code '1' Sun Jan 3 07:08:40 2021: NOTE: recipe brcm-nic-mgr-gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0: task do_configure: Succeeded do_ar_patched() is failed for missing a file from "git/bnxt-ctl/main/Cumulus/util/bnxt-ctl/autom4te.cache/output.1t". From the bitbake output, do_configure() started before do_ar_patched() completed. The missing file is from autoconf cache, which is created/deleted in do_configre(). do_configure() should start only after do_ar_patched() is completed.
Created attachment 4756 [details] bitbake-console-output-219.0.0.32.log
Created attachment 4757 [details] 219.0.0.32-brcm-nic-mgr.log.tgz brcm-nic-mgr recipe log (build/tmp/work/aarch64-poky-linux/brcm-nic-mgr/gitAUTOINC+9026833037_9651aeb672_a120b9b8dd-r0/temp/*)
Created attachment 4758 [details] brcm-nic-mgr.bb brcm-nic-mgr.bb
Your intuition seems right in that it's trying to archive the file while it's trying to run configure. Do we need to a dependency or is the a problem with dependency code? Are you able to submit a patch? How easy is it to reproduce?
I'm a little confused why this is happening, could you share the way the archiver is configured please? Looking at the code, do_ar_patched depends upon do_unpack_and_patch which extracts the source into a separate directory. do_ar_patched() then runs on that separate directory. So the question is why is the code looking in ${S} at all?
It's not very easy. At least I haven't been able to produce manually, and it's occasionally running into this issue during automation CI/CD builds.
Created attachment 4759 [details] archiver.bbclass This our archiver.bbclass. We have some local modification in this class to take out GIT branch/files from the archive tarball. I don't think these modification would cause this race issue.
I think most of time when I see this issue on CI/CD servers, the servers were busy. So I'd assume this race condition happens when CPU or I/O is fully loaded. The tasks might take longer than usual to complete etc.
(In reply to comment #5) > I'm a little confused why this is happening, could you share the way the > archiver is configured please? > > Looking at the code, do_ar_patched depends upon do_unpack_and_patch which > extracts the source into a separate directory. do_ar_patched() then runs on > that separate directory. > > So the question is why is the code looking in ${S} at all? I am seeing a similar issue. In our case this is happening for a Linux kernel recipe. Our error looks exactly like the one described in the following patch from 2019: https://patchwork.openembedded.org/patch/163387/ But the issue looks the same: * configure (or kernel_config_me) changes ${S} (temporary config files) * the archiver seems to run at the same time As far as I understand the archiver, the workdir is changed and the archiver should not interfere with the original ${S}. But for recipes inheriting the kernel.bbclass (which uses the shared workdir), this step is skipped.
Build move to 5.0 -- ../Randy
Bulk move of 5.0 medium importance issues to 5.99. Move to 5.1 if you want to actively work on an issue.
Move to 5.99 since no one is actively working on this issue. Move to 5.x if you start on a task, please.
There have been changes to archiver.bbclass for kernels and shared workdir, if there are still issues, a new bug report may be easiest at this point.