Bug 14706 - AB-INT-NET: network failure when fetching repositories
Summary: AB-INT-NET: network failure when fetching repositories
Status: RESOLVED WORKSFORME
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 3.4
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 4.0 M4
Assignee: Michael Halstead
QA Contact:
URL:
Whiteboard: AB-INT AB-INT-NET
Depends on:
Blocks:
 
Reported: 2022-01-27 14:52 UTC by Alexandre Belloni
Modified: 2025-03-20 14:11 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexandre Belloni 2022-01-27 14:52:24 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3157/steps/4/logs/stdio

Checking for stash at: /home/pokybuild/git/mirror/meta-arm
Cloning from stash to /home/pokybuild/tmp/shared-repo-temp-6votyn33/meta-arm...
Cloning into '/home/pokybuild/tmp/shared-repo-temp-6votyn33/meta-arm'...
Updating from origin...
ssh_exchange_identification: Connection closed by remote host
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/prepare-shared-repos", line 40, in <module>
    utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir)
  File "/home/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/utils.py", line 252, in fetchgitrepo
    subprocess.check_call(["git", "fetch", "origin"] + fetchopt, cwd=sharedrepo)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'fetch', 'origin']' returned non-zero exit status 128.
Comment 1 Alexandre Belloni 2022-01-27 14:53:10 UTC
This was ubuntu1804-ty-3 non-SSD worker
Comment 2 Randy MacLeod 2022-01-27 15:48:54 UTC
Seems to be a problem with the AB host. Do we have a way to log (just the right!) data at endpoints that we control?
Comment 3 Saul Wold 2022-02-10 15:17:09 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/91/builds/5511/steps/12/logs/stdio

perf-centos7 non-SSD

INFO: Committing data into to branch perf-centos7.yoctoproject.org/master/qemux86
INFO: Creating tag perf-centos7.yoctoproject.org/master/qemux86/64996-g472a99447fe0f89bbdb959c1d9b7f67f8a122935/6
INFO: Pushing data to remote
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/buildperf-centos7/build/scripts/oe-git-archive", line 118, in <module>
    sys.exit(main())
  File "/home/pokybuild/yocto-worker/buildperf-centos7/build/scripts/oe-git-archive", line 106, in main
    gitarchive.gitarchive(args.data_dir, args.git_dir, args.no_create, args.bare,
  File "/home/pokybuild/yocto-worker/buildperf-centos7/build/meta/lib/oeqa/utils/gitarchive.py", line 167, in gitarchive
    data_repo.run_cmd(cmd)
  File "/home/pokybuild/yocto-worker/buildperf-centos7/build/meta/lib/oeqa/utils/git.py", line 62, in run_cmd
    return self._run_git_cmd_at(git_args, self.top_dir, env=env)
  File "/home/pokybuild/yocto-worker/buildperf-centos7/build/meta/lib/oeqa/utils/git.py", line 43, in _run_git_cmd_at
    raise GitError("'{}' failed with exit code {}: {}".format(
oeqa.utils.git.GitError: 'git push --tags git@push.yoctoproject.org:yocto-buildstats perf-centos7.yoctoproject.org/master/qemux86 refs/notes/buildstats/perf-centos7.yoctoproject.org/master/qemux86' failed with exit code 128: ssh: Could not resolve hostname push.yoctoproject.org: Name or service not known
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Running on perf-centos7.yoctoproject.org
Using working dir /home/pokybuild/build-perf-test
Archiving results in /home/pokybuild/buildperf-results/archive-repo
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/buildperf-centos7/yocto-autobuilder-helper/scripts/build-perf-test-wrapper", line 169, in <module>
    subprocess.check_call("BDIR=%s . ./oe-init-build-env >/dev/null; oe-git-archive " % build_dir + \
  File "/home/pokybuild/yocto-worker/buildperf-centos7/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'BDIR=/home/pokybuild/build-perf-test/build-472a99447f-20220205031512 . ./oe-init-build-env >/dev/null; oe-git-archive --git-dir /home/pokybuild/buildperf-results/archive-repo --branch-name '{hostname}/{branch}/{machine}' --tag-name '{hostname}/{branch}/{machine}/{commit_count}-g{commit}/{tag_number}' --exclude 'buildstats.json' --notes 'buildstats/{branch_name}' /home/pokybuild/build-perf-test/results-472a99447f-20220205031512/buildstats.json --push git@push.yoctoproject.org:yocto-buildstats /home/pokybuild/build-perf-test/results-472a99447f-20220205031512' returned non-zero exit status 1.
Comment 4 Saul Wold 2022-02-10 15:18:25 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/4675/steps/10/logs/stdio
alma-ty-2 SSD

Initialized empty Git repository in /home/pokybuild/yocto-worker/qemumips/build/build/buildhistory/.git/
Switched to a new branch 'poky/master-next/qemumips'
remote: fatal: unable to read a60e35e6b42f64b993a678daf232ad620eb178bb        
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed
fatal: ambiguous argument 'refs/remotes/poky/master/qemumips': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: Failed to resolve 'HEAD' as a valid ref.
Comment 5 Saul Wold 2022-02-10 15:28:32 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/4677/steps/10/logs/stdio
fedora35-ty-1 SSD

hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
Switched to a new branch 'poky/master/qemumips'
Initialized empty Git repository in /home/pokybuild/yocto-worker/qemumips/build/build/buildhistory/.git/
remote: fatal: unable to read 2e8b17a312413206c990c9d8359084ec04a2b291        
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
fatal: ambiguous argument 'refs/remotes/poky/master/qemumips': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: Failed to resolve 'HEAD' as a valid ref.
Comment 6 Saul Wold 2022-02-10 15:30:11 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3200
stream8-ty-1 SSD

fatal: Needed a single revision
Already on 'master'
Your branch is up to date with 'origin/master'.
Switched to branch 'master-next'
INFO: Reading files from /srv/autobuilder/autobuilder.yoctoproject.org/pub/non-release/20220208-14/testresults
INFO: Found 1 revisions to store
INFO: Storing test result into git repository /tmp/sendqaemail.evy0ie4d
INFO: Committing data into to branch master-next
INFO: Updating /tmp/sendqaemail.evy0ie4d HEAD to latest commit
INFO: Creating tag master-next/65112-g45046711589d0434be1033b1323973e815ac79ca/0
remote: error: cannot lock ref 'refs/heads/master-next': is at 6317d44fdad43e842429019ca0ee6fccd8b15221 but expected bdf31f23aa8fbba52517159b5779579bc51a2a74        
To push.yoctoproject.org:yocto-testresults
 ! [remote rejected] master-next -> master-next (failed to update ref)
error: failed to push some refs to 'push.yoctoproject.org:yocto-testresults'
Working off ssh://git@push.yoctoproject.org/poky:master-next
Comment 7 Alexandre Belloni 2022-03-03 10:28:17 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3303/steps/4/logs/stdio

Checking for stash at: /home/pokybuild/git/mirror/poky
Cloning from stash to /home/pokybuild/tmp/shared-repo-temp-3_rsdtuf/poky...
Cloning into '/home/pokybuild/tmp/shared-repo-temp-3_rsdtuf/poky'...
Updating from origin...
kex_exchange_identification: Connection closed by remote host
Connection closed by 44.237.223.161 port 22
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/prepare-shared-repos", line 40, in <module>
    utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir)
  File "/home/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/utils.py", line 252, in fetchgitrepo
    subprocess.check_call(["git", "fetch", "origin"] + fetchopt, cwd=sharedrepo)
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'fetch', 'origin']' returned non-zero exit status 128.
Comment 11 Alexandre Belloni 2022-04-14 13:09:21 UTC
Cloning into '.'...
command timed out: 1200 seconds without output running [b'git', b'clone', b'--branch', b'master-next', b'ssh://git@push.yoctoproject.org/yocto-autobuilder-helper', b'.', b'--progress'], attempting to kill
process killed by signal 15
program finished with exit code -1
elapsedTime=1200.022065

https://autobuilder.yoctoproject.org/typhoon/#builders/73/builds/5011/steps/2/logs/stdio

qemux86-64 alma8-ty-2 had an SSD
Comment 12 Alexandre Belloni 2022-04-14 16:30:54 UTC
Closing this one out to track new pattern following the datacenter move
Comment 13 Randy MacLeod 2025-02-14 16:46:22 UTC
bulk change to add Saul's non-WR address.
Comment 14 Randy MacLeod 2025-02-14 16:48:20 UTC
Bulk change: Remove Saul's old WR address.
Comment 15 Mathieu Dubois-Briand 2025-03-06 13:54:17 UTC
build-appliance ubuntu2204-vk-3 master-next completed at 2025-03-05T19:35:16Z
https://autobuilder.yoctoproject.org/valkyrie/#/builders/78/builds/1157/steps/11/logs/stdio
Comment 16 Mathieu Dubois-Briand 2025-03-20 14:11:33 UTC
qemuppc alma8-vk-2 master-next completed at 2025-03-19T15:44:22Z
https://autobuilder.yoctoproject.org/valkyrie/#/builders/31/builds/1206/steps/15/logs/stdio