When I try to use (PRE)MIRROR for svn SCR_URI it not change only server name, but add tar ball name to mirror URI too. I find out that there is no test for svn mirrors so I created one. I think it is easiest way to explain what is wrong. I tied to use authentic svn repo path. Here what I add to tests git diff diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 85c1f79f..66c93282 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -511,7 +511,8 @@ class MirrorUriTest(FetcherTest): mirrorvar = "http://.*/.* file:///somepath/downloads/ " \ "git://someserver.org/bitbake git://git.openembedded.org/bitbake " \ "https://.*/.* file:///someotherpath/downloads/ " \ - "http://.*/.* file:///someotherpath/downloads/" + "http://.*/.* file:///someotherpath/downloads/ " \ + "svn://svn.server1.com/ svn://svn.server2.com/" def test_urireplace(self): self.d.setVar("FILESPATH", ".") @@ -535,6 +536,13 @@ class MirrorUriTest(FetcherTest): uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d) self.assertEqual(uris, ['file:///someotherpath/downloads/bitbake-1.0.tar.gz']) + def test_urilistsvn(self): + # Catch svn:// -> svn:// bug + fetcher = bb.fetch.FetchData("svn://svn.server1.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2", self.d) + mirrors = bb.fetch2.mirror_from_string(self.mirrorvar) + uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d) + self.assertEqual(uris, ['svn://svn.server2.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2']) + def test_mirror_of_mirror(self): # Test if mirror of a mirror works mirrorvar = self.mirrorvar + " http://.*/.* http://otherdownloads.yoctoproject.org/downloads/" in short this should change from svn url svn.server1.com to svn-server2.com, but that is not what happens. Here what happens: bin/bitbake-selftest -c bb.tests.fetch.MirrorUriTest ....F. ====================================================================== FAIL: test_urilistsvn (bb.tests.fetch.MirrorUriTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/ephemeral/karisivo/meta-ps-rel/bitbake/lib/bb/tests/fetch.py", line 544, in test_urilistsvn self.assertEqual(uris, ['svn://svn.server2.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2']) AssertionError: Lists differ: ['svn[44 chars]tags/path_in_tagnamefolder_svn.server1.com_.is[93 chars]v=2'] != ['svn[44 chars]tags/tagname;module=path_in_tagnamefolder;prot[13 chars]v=2'] First differing element 0: 'svn:[43 chars]tags/path_in_tagnamefolder_svn.server1.com_.is[92 chars]ev=2' 'svn:[43 chars]tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2' - ['svn://svn.server2.com/isource/svnroot/reponame/tags/path_in_tagnamefolder_svn.server1.com_.isource.svnroot.reponame.tags.tagname_2_1.tar.gz;module=path_in_tagnamefolder;protocol=https;rev=2'] + ['svn://svn.server2.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2'] ---------------------------------------------------------------------- Ran 6 tests in 0.078s FAILED (failures=1) I have already a fix for this. I debug this to lib/bb/fetch2/__init__.py line 473. elif ud.localpath and ud.method.supports_checksum(ud): basename = os.path.basename(ud.localpath) This check is true and cause this wrong name in path. This can be fixed by this change: git diff diff --git a/lib/bb/fetch2/svn.py b/lib/bb/fetch2/svn.py index d40e4d29..0852108e 100644 --- a/lib/bb/fetch2/svn.py +++ b/lib/bb/fetch2/svn.py @@ -210,3 +210,6 @@ class Svn(FetchMethod): def _build_revision(self, ud, d): return ud.revision + + def supports_checksum(self, urldata): + return False After this tests arepassed. bin/bitbake-selftest -c bb.tests.fetch.MirrorUriTest ...... ---------------------------------------------------------------------- Ran 6 tests in 0.074s OK
I use bitbake 2.0.0 but I run this test and done modifications to latest bitbake master version bitbake --version BitBake Build Tool Core version 2.9.0 git log -3 commit 97ffe14311407f6e705ec24b70870ab32f0637b9 (HEAD -> master, origin/master, origin/HEAD) Author: Sven Schwermer <sven.schwermer@disruptive-technologies.com> Date: Thu Apr 18 15:41:47 2024 +0200 fetch2/gcp: Add missing runfetchcmd import This adds the missing import. This bug was introduced with 1ab1d36c. Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Created attachment 5042 [details] SVN mirror fix and test for it
Could you send the patch to the bitbake-devel mailing list please?
(In reply to Richard Purdie from comment #3) > Could you send the patch to the bitbake-devel mailing list please? Now send it again. I find out that I have to be registered to openembedded too to send that patch. Should be now send.
Patch on list, likely merged soon.
Hi, I find bug from this bug fix. Now it works for svn to svn mirrors but other mirrors now work. E.g svn to https not works anymore. This might be used when tarballs used for svn repos. I am working with fix
Created attachment 5047 [details] Patch of new fix for this There was bug in first fix of svn mirror fix for [YOCTO #15473] svn to svn mirror fixed in first commit but it broke svn to other mirrors This commit fix those other mirrors back to original way of working --- lib/bb/fetch2/__init__.py | 10 ++++++++++ lib/bb/tests/fetch.py | 5 +++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 5bf2c4b8c..cfe6d5025 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -472,6 +472,16 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None): uri_find_decoded[5] = {} elif ud.localpath and ud.method.supports_checksum(ud): basename = os.path.basename(ud.localpath) + elif ud.localpath and uri_decoded[0] == "svn" and uri_replace_decoded[0] != "svn": + #When added svn fetcher that supports_checksum return false + #all mirrors for svn skip all basename sets. That is ok for svn -> svn mirrors + #but other mirrors not work anymore. Example you want use tar balls for svn from https server + #This elif will catch cases where svn mirros gone before svn mirror fix + #This is basically same than first if branch except no mirrortarball set. Can it some how set for svn uris? + basename = os.path.basename(ud.localpath) + # Kill parameters, they make no sense for mirror tarballs + uri_decoded[5] = {} + uri_find_decoded[5] = {} if basename: uri_basename = os.path.basename(uri_decoded[loc]) # Prefix with a slash as a sentinel in case diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index ed7a39a72..934c19e0e 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -512,7 +512,8 @@ class MirrorUriTest(FetcherTest): "git://someserver.org/bitbake git://git.openembedded.org/bitbake " \ "https://.*/.* file:///someotherpath/downloads/ " \ "http://.*/.* file:///someotherpath/downloads/ " \ - "svn://svn.server1.com/ svn://svn.server2.com/" + "svn://svn.server1.com/ svn://svn.server2.com/ " \ + "svn://svn.server1.com/ file:///someotherpath/downloads/ " def test_urireplace(self): self.d.setVar("FILESPATH", ".") @@ -541,7 +542,7 @@ class MirrorUriTest(FetcherTest): fetcher = bb.fetch.FetchData("svn://svn.server1.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2", self.d) mirrors = bb.fetch2.mirror_from_string(self.mirrorvar) uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d) - self.assertEqual(uris, ['svn://svn.server2.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2']) + self.assertEqual(uris, ['svn://svn.server2.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2', 'file:///someotherpath/downloads/isource/svnroot/reponame/tags/path_in_tagnamefolder_svn.server1.com_.isource.svnroot.reponame.tags.tagname_2_1.tar.gz']) def test_mirror_of_mirror(self): # Test if mirror of a mirror works -- 2.40.1
Hi Kari, Few people are using svn but if you send the patch to the bitbake list, it may get reviewed and hopefully merged. Can you do that? For tips, see: https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html or ask here. Randy
(In reply to Randy MacLeod from comment #8) > Hi Kari, > > Few people are using svn but if you send the patch to the bitbake list, it > may get reviewed and hopefully merged. Can you do that? > > For tips, see: > https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html > or ask here. > Randy I send patch to bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org>. Because of some settings on my linux machine it not send it with git command so I copied patch to email and send it from outlook. I tried to add subject and content like it was in email git tried to send but not sure if it is exactly same. Subjetc is "[bitbake-devel] [PATCH] fetch2/svn: Fix mirroring issue with svn" Richard Purdie already reply it. I like to find a way to do get this change in.
https://git.openembedded.org/bitbake/commit/?id=21cfc7ae9a19f39ac8904e1c3466e7e499ac523f