我在Google Colab上运行“dvc pull”时遇到了问题。我有两个存储库(我们称之为A和B),其中存储库A用于我的机器学习代码,存储库B用于我的数据集。
我已经成功地使用DVC(使用gdrive作为我的远程存储)将我的数据集推送到了存储库B,并且我还成功地在存储库A的本地项目上运行了“dvc import”(以及“dvc pull/update”)。
当我使用colab来运行我的项目时,问题就来了。所以我做了以下事情:
1.在colab上创建新笔记本
1.成功git克隆了我的机器学习项目(存储库A)
1.已运行“!pip安装dvc”
1.运行“!dvc pull -v”(这是导致错误的原因)
在第4步,我得到了错误(这是完整的堆栈跟踪。注意,出于保密原因,我更改了堆栈跟踪中的repo URL)
2022-03-08 08:53:31,863 DEBUG: Adding '/content/<my_project_A>/.dvc/config.local' to gitignore file.
2022-03-08 08:53:31,866 DEBUG: Adding '/content/<my_project_A>/.dvc/tmp' to gitignore file.
2022-03-08 08:53:31,866 DEBUG: Adding '/content/<my_project_A>/.dvc/cache' to gitignore file.
2022-03-08 08:53:31,916 DEBUG: Creating external repo https://gitlab.com/<my-dataset-repo-B>.git@3a3f2019efabff8ec71429da39b86688d1c98e75
2022-03-08 08:53:31,916 DEBUG: erepo: git clone 'https://gitlab.com/<my-dataset-repo-B>.git' to a temporary dir
Everything is up to date.
2022-03-08 08:53:32,154 ERROR: failed to pull data from the cloud - Failed to clone repo 'https://gitlab.com/<my-dataset-repo-B>.git' to '/tmp/tmp2x6y9z0edvc-clone'
------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/scmrepo/git/backend/gitpython.py", line 185, in clone
tmp_repo = clone_from()
File "/usr/local/lib/python3.7/dist-packages/git/repo/base.py", line 1148, in clone_from
return cls._clone(git, url, to_path, GitCmdObjectDB, progress, multi_options, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/git/repo/base.py", line 1079, in _clone
finalize_process, decode_streams=False)
File "/usr/local/lib/python3.7/dist-packages/git/cmd.py", line 176, in handle_process_output
return finalizer(process)
File "/usr/local/lib/python3.7/dist-packages/git/util.py", line 386, in finalize_process
proc.wait(**kwargs)
File "/usr/local/lib/python3.7/dist-packages/git/cmd.py", line 502, in wait
raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git clone -v --no-single-branch --progress https://gitlab.com/<my-dataset-repo-B>.git /tmp/tmp2x6y9z0edvc-clone
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/dvc/scm.py", line 104, in clone
return Git.clone(url, to_path, progress=pbar.update_git, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/scmrepo/git/__init__.py", line 121, in clone
backend.clone(url, to_path, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/scmrepo/git/backend/gitpython.py", line 190, in clone
raise CloneError(url, to_path) from exc
scmrepo.exceptions.CloneError: Failed to clone repo 'https://gitlab.com/<my-dataset-repo-B>.git' to '/tmp/tmp2x6y9z0edvc-clone'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/dvc/command/data_sync.py", line 41, in run
glob=self.args.glob,
File "/usr/local/lib/python3.7/dist-packages/dvc/repo/__init__.py", line 49, in wrapper
return f(repo, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/dvc/repo/pull.py", line 38, in pull
run_cache=run_cache,
File "/usr/local/lib/python3.7/dist-packages/dvc/repo/__init__.py", line 49, in wrapper
return f(repo, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/dvc/repo/fetch.py", line 50, in fetch
revs=revs,
File "/usr/local/lib/python3.7/dist-packages/dvc/repo/__init__.py", line 437, in used_objs
with_deps=with_deps,
File "/usr/local/lib/python3.7/dist-packages/dvc/repo/index.py", line 190, in used_objs
filter_info=filter_info,
File "/usr/local/lib/python3.7/dist-packages/dvc/stage/__init__.py", line 660, in get_used_objs
for odb, objs in out.get_used_objs(*args, **kwargs).items():
File "/usr/local/lib/python3.7/dist-packages/dvc/output.py", line 918, in get_used_objs
return self.get_used_external(**kwargs)
File "/usr/local/lib/python3.7/dist-packages/dvc/output.py", line 973, in get_used_external
return dep.get_used_objs(**kwargs)
File "/usr/local/lib/python3.7/dist-packages/dvc/dependency/repo.py", line 94, in get_used_objs
used, _ = self._get_used_and_obj(**kwargs)
File "/usr/local/lib/python3.7/dist-packages/dvc/dependency/repo.py", line 108, in _get_used_and_obj
locked=locked, cache_dir=local_odb.cache_dir
File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.7/dist-packages/dvc/external_repo.py", line 35, in external_repo
path = _cached_clone(url, rev, for_write=for_write)
File "/usr/local/lib/python3.7/dist-packages/dvc/external_repo.py", line 155, in _cached_clone
clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write)
File "/usr/local/lib/python3.7/dist-packages/funcy/decorators.py", line 45, in wrapper
return deco(call, *dargs, **dkwargs)
File "/usr/local/lib/python3.7/dist-packages/funcy/flow.py", line 274, in wrap_with
return call()
File "/usr/local/lib/python3.7/dist-packages/funcy/decorators.py", line 66, in __call__
return self._func(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/dist-packages/dvc/external_repo.py", line 220, in _clone_default_branch
git = clone(url, clone_path)
File "/usr/local/lib/python3.7/dist-packages/dvc/scm.py", line 106, in clone
raise CloneError(str(exc))
dvc.scm.CloneError: Failed to clone repo 'https://gitlab.com/<my-dataset-repo-B>.git' to '/tmp/tmp2x6y9z0edvc-clone'
------------------------------------------------------------
2022-03-08 08:53:32,161 DEBUG: Analytics is enabled.
2022-03-08 08:53:32,192 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp4x5js0dk']'
2022-03-08 08:53:32,193 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp6x11s0dk']'
顺便说一句,这是我克隆git仓库(repo A)的方式
!git config - global user.name "Zharfan"
!git config - global user.email "zharfan@myemail.com"
!git clone https://<MyTokenName>:<MyToken>@link-to-my-repo-A.git
有人知道为什么吗?任何帮助都将不胜感激。提前感谢你!
1条答案
按热度按时间ssgvzors1#
在评论线程中总结讨论。
最有可能的原因是DVC无法访问GitLab上的私有repo(错误信息很模糊,应该修复)。
就像你不能跑一样:
它还返回一个相当模糊的错误:
(我认为这与Colab中如何设置tty有关?)
解决此问题的最佳方法是使用SSH,例如所描述的here。