Bug 15948 - PREMIRROR does not work with git lfs
Summary: PREMIRROR does not work with git lfs
Status: IN PROGRESS REVIEW
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 5.0.15
Hardware: x86 x86_64
: Medium+ normal
Target Milestone: 6.1 M1
Assignee: Unassigned
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2025-08-06 07:23 UTC by Oliver Feilner
Modified: 2026-08-20 12:53 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Oliver Feilner 2025-08-06 07:23:24 UTC
When fetching a repo using git lfs from a MIRROR / PREMIRROR configured in the local.conf the fetch failed.

PREMIRRORS = " git://.*\.xxx\.com.*/(yyyy|zzzz)/([^/]+)\.git                         git://dev.azure.com/project/Synchronization/_git/yyyy.\2;protocol=https;user=${ACCESSTOKEN} \n"

PREMIRRORS =+ "${MY_MIRRORS}"
MIRRORS    =+ "${MY_MIRRORS}"

```
ERROR: prebuild-libs-8.30.6885.501-r0 do_fetch: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:base_do_fetch(d)
     0003:
File: '/home/myBuild/autobuild/../../poky/meta/classes-global/base.bbclass', lineno: 150, function: base_do_fetch
     0146:        return
     0147:
     0148:    try:
     0149:        fetcher = bb.fetch2.Fetch(src_uri, d)
 *** 0150:        fetcher.download()
     0151:    except bb.fetch2.BBFetchException as e:
     0152:        bb.fatal("Bitbake Fetcher Error: " + repr(e))
     0153:}
     0154:
File: '/home/myBuild/poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 1851, function: download
     1847:                    done = True
     1848:                elif m.try_premirror(ud, self.d):
     1849:                    logger.debug("Trying PREMIRRORS")
     1850:                    mirrors = mirror_from_string(self.d.getVar('PREMIRRORS'))
 *** 1851:                    done = m.try_mirrors(self, ud, self.d, mirrors)
     1852:                    if done:
     1853:                        try:
     1854:                            # early checksum verification so that if the checksum of the premirror
     1855:                            # contents mismatch the fetcher can still try upstream and mirrors
File: '/home/myBuild/poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 1648, function: try_mirrors
     1644:    def try_mirrors(self, fetch, urldata, d, mirrors, check=False):
     1645:        """
     1646:        Try to use a mirror
     1647:        """
 *** 1648:        return bool(try_mirrors(fetch, d, urldata, mirrors, check))
     1649:
     1650:    def checkstatus(self, fetch, urldata, d):
     1651:        """
     1652:        Check the status of a URL
File: '/home/myBuild/poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 1163, function: try_mirrors
     1159:
     1160:    uris, uds = build_mirroruris(origud, mirrors, ld)
     1161:
     1162:    for index, uri in enumerate(uris):
 *** 1163:        ret = try_mirror_url(fetch, origud, uds[index], ld, check)
     1164:        if ret:
     1165:            return ret
     1166:    return None
     1167:
File: '/home/myBuild/poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 1061, function: try_mirror_url
     1057:                return found
     1058:            return False
     1059:
     1060:        if not verify_donestamp(ud, ld, origud) or ud.method.need_update(ud, ld):
 *** 1061:            ud.method.download(ud, ld)
     1062:            if hasattr(ud.method,"build_mirror_data"):
     1063:                ud.method.build_mirror_data(ud, ld)
     1064:
     1065:        if not ud.localpath or not os.path.exists(ud.localpath):
File: '/home/myBuild/poky/bitbake/lib/bb/fetch2/git.py', lineno: 494, function: download
     0490:            # on the bare clonedir, but that operation requires a working copy on some
     0491:            # releases of Git LFS.
     0492:            with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
     0493:                # Do the checkout. This implicitly involves a Git LFS fetch.
 *** 0494:                Git.unpack(self, ud, tmpdir, d)
     0495:
     0496:                # Scoop up a copy of any stuff that Git LFS downloaded. Merge them into
     0497:                # the bare clonedir.
     0498:                #
File: '/home/myBuild/poky/bitbake/lib/bb/fetch2/git.py', lineno: 636, function: unpack
     0632:        destdir = ud.destdir = os.path.join(destdir, destsuffix)
     0633:        if os.path.exists(destdir):
     0634:            bb.utils.prunedir(destdir)
     0635:        if not ud.bareclone:
 *** 0636:            ud.unpack_tracer.unpack("git", destdir)
     0637:
     0638:        need_lfs = self._need_lfs(ud)
     0639:
     0640:        if not need_lfs:
Exception: AttributeError: 'FetchData' object has no attribute 'unpack_tracer'

ERROR: Logfile of failure stored in: /home/myBuild/autobuild/build/tmp/work/armv7vet2hf-neon-vfpv4-yaskawa-linux-gnueabi/prebuild-libs/8.30.6885.501/temp/log.do_fetch.3715563
ERROR: Task (/home/myBuild/autobuild/../../meta-arp/recipes-arp/prebuild-libs/prebuild-libs_8.30.6885.501.bb:do_fetch) failed with exit code '1'
```

Without using a mirror download is called with an object from type Fetch. Which has the attribute unpack_tracer. 
In case of mirrors  download is called with an object from the type FetchData. Which has no attribute unpack_tracer.


The behavior could be fixed by adding the unpack_tracer also to the mirror ud

From f8e3d48ea9cc609c0c9dd96be90a34cd52822e86 Mon Sep 17 00:00:00 2001
From: Oliver Feilner <oliver.feilner@yaskawa.eu>
Date: Wed, 6 Aug 2025 09:16:13 +0200
Subject: [PATCH] Copy unpack_tracker also mirror uds

---
 bitbake/lib/bb/fetch2/__init__.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index e698fde19c..aed5a28c69 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -1020,6 +1020,7 @@ def build_mirroruris(origud, mirrors, ld):
                     except UnboundLocalError:
                         pass
                     continue
+                newud.unpack_tracer = ud.unpack_tracer
                 uris.append(newuri)
                 uds.append(newud)

--
2.34.1
Comment 1 Randy MacLeod 2025-08-07 14:43:49 UTC
Oliver,
Can you send the patch to the oe-core email list please?

https://docs.yoctoproject.org/contributor-guide/index.html

-- YP bug review / Randy
Comment 2 Randy MacLeod 2025-09-25 15:05:12 UTC
Oliver, 

We don't have many git lfs users so unless you can send a patch or convince someone else to do that, this issue won't be fixed.

If you need help getting started, see:
https://docs.yoctoproject.org/contributor-guide/index.html
and/or ask me.
../Randy
Comment 3 Siva Balasubramanian 2026-07-03 11:20:17 UTC
Patch submitted to the bitbake-devel mailing list:

[bitbake-devel][PATCH] fetch2: give every FetchData a default unpack_tracer
https://lists.openembedded.org/g/bitbake-devel/topic/120097900
[add Patchwork link after sending]

Thanks Oliver for the report, the root-cause analysis, and the
proposed patch.

I took a slightly different approach than the patch attached here.
Copying the tracer onto the mirror ud in build_mirroruris()
(newud.unpack_tracer = ud.unpack_tracer) assumes the origin ud always
has an unpack_tracer, but that isn't guaranteed: Fetch() only sets it
on the URLs it manages, so bare FetchData objects don't have it.
Applying that change makes build_mirroruris() raise the same
AttributeError and breaks all six existing
bb.tests.fetch.MirrorUriTest tests.

Instead, the fix initialises unpack_tracer to a DummyUnpackTracer in
FetchData.init(), so the attribute always exists. Fetch() still
overrides it with the real (or BB_UNPACK_TRACER_CLASS-configured)
tracer for managed URLs; the mirror FetchData objects only do an
internal, throwaway git-lfs checkout that shouldn't be traced anyway.

Verified:


bb.tests.fetch.MirrorUriTest: 6 pass on master, all 6 fail with the
originally-proposed patch, 7 pass with this fix (added a regression
test asserting mirror uds carry an unpack_tracer)
Broader offline fetcher suite: 45 tests pass
Comment 4 Mathieu Dubois-Briand 2026-08-20 12:53:06 UTC
Patch got merged: https://git.openembedded.org/bitbake/commit/?id=f0c9cf8d3885c5b1c2ba448f064421dae476fcd0 .

Oliver, can you confirm we can close the bug?