LibGit2 can not clone or fetch on Julia 1.3/master #33111

fredrikekre · 2019-08-29T11:35:04Z

Opening this issue here since I am pretty sure Pkg has not changed anything, maybe this is related to the libgit2 upgrade (#32806)?

$ julia13 -e 'using Pkg; Pkg.Registry.update()'
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
┌ Warning: Some registries failed to update:
│     — /home/fredrik/.julia/registries/General — failed to fetch from repo
└ @ Pkg.Types ~/julia13/usr/share/julia/stdlib/v1.3/Pkg/src/Types.jl:1189

$ julia12 -e 'using Pkg; Pkg.Registry.update()'
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`

The text was updated successfully, but these errors were encountered:

KristofferC · 2019-08-29T12:06:09Z

Works for me on a mac.

fredrikekre · 2019-08-29T12:12:47Z

Ok, I'm on Linux. Reverting #32806 fixes this issue for me.

fredrikekre · 2019-08-29T12:46:12Z

Reproducer without any Pkg code:

julia> import LibGit2

julia> repo = LibGit2.GitRepo("/home/fredrik/.julia/registries/General");

julia> remoteurl = "https://github.com/JuliaRegistries/General.git";

julia> LibGit2.fetch(repo; remoteurl=remoteurl)
ERROR: GitError(Code:ERROR, Class:Net, unrecognized URL prefix)
Stacktrace:
 [1] #fetch#62(::LibGit2.FetchOptions, ::String, ::typeof(LibGit2.fetch), ::LibGit2.GitRemote, ::Array{AbstractString,1}) at /home/fredrik/julia-master/usr/share/julia/stdlib/v1.4/LibGit2/src/error.jl:101
 [2] #fetch at /home/fredrik/julia-master/usr/share/julia/stdlib/v1.4/LibGit2/src/remote.jl:297 [inlined]
 [3] #fetch#115(::String, ::String, ::Array{AbstractString,1}, ::Nothing, ::Dict{Symbol,Tuple{Ptr{Nothing},Any}}, ::typeof(LibGit2.fetch), ::LibGit2.GitRepo) at /home/fredrik/julia-master/usr/share/julia/stdlib/v1.4/LibGit2/src/LibGit2.jl:290
 [4] (::LibGit2.var"#kw##fetch")(::NamedTuple{(:remoteurl,),Tuple{String}}, ::typeof(LibGit2.fetch), ::LibGit2.GitRepo) at /home/fredrik/julia-master/usr/share/julia/stdlib/v1.4/LibGit2/src/LibGit2.jl:272
 [5] top-level scope at REPL[4]:1

omus · 2019-08-29T16:48:46Z

Possibly unrelated: I saw the same error on Julia 1.0.4 today on macOS. I fixed it by going to the registry in my terminal and running git fetch. After that Pkg updates stopped producing the warning.

fredrikekre · 2019-08-30T07:57:28Z

Possibly unrelated: I saw the same error on Julia 1.0.4 today on macOS. I fixed it by going to the registry in my terminal and running git fetch. After that Pkg updates stopped producing the warning.

That did not fix it for me. Also, I tried from a different machine but could not reproduce. Any ideas on how to debug?

fredrikekre · 2019-08-30T08:01:35Z

Cloning from scratch also fails, with the same error message about unrecognized URL prefix.

fredrikekre · 2019-09-03T08:15:12Z

This also happens with the generic binaries btw.

KristofferC · 2019-09-03T08:27:25Z

Possible future way to debug this is to reduce it to just ccalls and then look into libgit2 where the error is generated.

fredrikekre · 2019-09-03T10:01:00Z

import LibGit2

const repo_url = "https://github.com/JuliaRegistries/General.git"
const repo_path = "/tmp/General"
const credentials = nothing
const callbacks = LibGit2.Callbacks()
const cred_payload = LibGit2.reset!(LibGit2.CredentialPayload(credentials))
callbacks[:credentials] = (LibGit2.credentials_cb(), cred_payload)
const lbranch = Base.cconvert(Cstring, "")
const remote_callbacks = LibGit2.RemoteCallbacks(callbacks)
const fetch_opts = LibGit2.FetchOptions(callbacks=remote_callbacks)
const clone_opts = LibGit2.CloneOptions(
            bare = Cint(false),
            checkout_branch = Cstring(C_NULL),
            fetch_opts = fetch_opts,
            remote_cb = C_NULL
        )
LibGit2.ensure_initialized()
const clone_opts_ref = Ref(clone_opts)
const repo_ptr_ptr = Ref{Ptr{Cvoid}}(C_NULL)

ccall((:git_clone, :libgit2), Cint,
            (Ptr{Ptr{Cvoid}}, Cstring, Cstring, Ref{LibGit2.CloneOptions}),
            repo_ptr_ptr, repo_url, repo_path, clone_opts_ref)

returns -1.

nalimilan · 2019-09-03T20:19:11Z

FWIW both examples work here on latest master. Can anybody else reproduce?

omus · 2019-09-03T20:58:52Z

@fredrikekre can you post your git config?

fredrikekre · 2019-09-05T08:11:16Z

Interesting, I reduced the .gitconfig file to

[http]
	proxy =

I am not sure why it is there, I don't think I have added it myself. Seems like libgit2@0.28 fails for this, while libgit2@0.27 handles it (as does regular git)

fredrikekre · 2019-09-05T08:25:51Z

Maybe related to this point from libgit2 0.28 release notes:

HTTP proxy support is now builtin; libcurl is no longer used to support
proxies and is removed as a dependency.

nalimilan · 2019-09-05T08:37:03Z

Maybe file an issue against libgit2? You can also tried reverting the changes I made to gitcredential.jl in #32806, just in case (they are required for some specific tests, but not for most operations).

fredrikekre · 2019-09-05T12:57:43Z

See libgit2/libgit2#5220.

fredrikekre · 2019-09-06T09:52:43Z

Removing from milestone since it is kinda weird to have an empty http.proxy, and we can upgrade libgit2 in 1.3.1 or whatever.

kcajf · 2019-09-12T20:47:50Z

I can reproduce something similar, but the snippet you sent above returns -12. It does appear to have been introduced in a recent commit - julia 1.2.0 runs fine but ~~1.3.0-rc1 and~~ master / latest nightlies (47f2800) fail.

tkluck · 2019-10-10T12:04:22Z

This issue also appeared for me with http(s)_proxy set as an environment variable. It needs to be unset (not set to empty string) to avoid it.

kcajf · 2019-10-10T17:56:30Z

Thanks for the suggestion @tkluck - unfortunately unsetting http_proxy and http_proxy doesn't make a difference more me - still getting -12 return code :(

wsshin · 2019-10-11T19:43:11Z

I had this issue just now and would like to report the test results with three fresh-built Julia binaries in release-1.0, release-1.3, and master branches. (I was not able to test release-1.2 due to build errors.) All three tests were conducted with empty ~/.julia. The environment variables http_proxy and https_proxy are defined in ~/.bashrc.

release-1.0 (no error):

julia> VERSION
v"1.0.6-pre.0"

(v1.0) pkg> update
 Resolving package versions...
   Cloning default registries into /Users/MY_USER_ID/.julia/registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git"
    Fetching: [========================================>]  99.9 %

release-1.3 (error):

julia> VERSION
v"1.3.0-rc3.0"

(v1.3) pkg> update
   Cloning default registries into `~/.julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
ERROR: Git repository not found at 'https://github.com/JuliaRegistries/General.git'

master (error):

julia> VERSION
v"1.4.0-DEV.298"

(v1.4) pkg> update
   Cloning default registries into `~/.julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
ERROR: Git repository not found at 'https://github.com/JuliaRegistries/General.git'

Hope this issue to be resolved soon!

kcajf · 2019-10-11T20:01:21Z

Is your http_proxymissing a trailing slash? See libgit2/libgit2#5255

fredrikekre · 2020-03-20T21:25:19Z

Please report to the libgit2 repo instead (unless it is a duplicate of libgit2/libgit2#5220, in which you can maybe bump that issue.)

diabonas · 2020-03-23T11:12:13Z

The issues with LibGit2 0.99.0 reported in the comments are unrelated to the original bug report, see issue #35043 instead and PR #35232 for a suggested fix.

getzze · 2020-03-23T11:33:04Z

You can use the soon-to-be-released Julia 1.4 with JULIA_PKG_SERVER=pkg.julialang.org and it won't use libgit2 at all for installing registered packages.

This workaround doesn't work unfortunately, I am using julia 1.4.

tuhtah · 2020-03-23T15:06:08Z

@getzze How are you invoking julia? Maybe you forgot to export the JULIA_PKG_SERVER environment variable in your shell before invoking Julia in a separate shell prompt?

The workaround works on Arch Linux using Julia 1.4.0:

[tamer@zinc ~]$ pacman -Q libgit2 julia
libgit2 1:0.99.0-2
julia 2:1.4.0-1

[tamer@zinc ~]$ rm -r ~/.julia/registries/General/

[tamer@zinc ~]$ julia -e 'using Pkg; Pkg.update()'
    Cloning default registries into `~/.julia`
    Cloning registry from "https://github.com/JuliaRegistries/General.git"
ERROR: failed to clone from https://github.com/JuliaRegistries/General.git, error: GitError(Code:ERROR, Class:Invalid, invalid version 0 on git_proxy_options)
Stacktrace:
 [1] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:206
 [2] up at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:201 [inlined]
 [3] #up#47 at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:196 [inlined]
 [4] up at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:196 [inlined]
 [5] #up#44 at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:193 [inlined]
 [6] up() at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:193
 [7] top-level scope at none:1
caused by [exception 1]
GitError(Code:ERROR, Class:Invalid, invalid version 0 on git_proxy_options)
Stacktrace:
 [1] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:206
 [2] up at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:201 [inlined]
 [3] #up#47 at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:196 [inlined]
 [4] up at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:196 [inlined]
 [5] #up#44 at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:193 [inlined]
 [6] up() at /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:193
 [7] top-level scope at none:1

[tamer@zinc ~]$ JULIA_PKG_SERVER=pkg.julialang.org julia -e 'using Pkg; Pkg.update()'
    Cloning default registries into `~/.julia`
######################################################################## 100.0%
      Added registry `General` to `~/.julia/registries/General`
   Updating registry at `~/.julia/registries/General`
   Updating `~/.julia/environments/v1.4/Project.toml`
 [no changes]
   Updating `~/.julia/environments/v1.4/Manifest.toml`
 [no changes]

fredrikekre · 2020-03-23T15:12:50Z

If you already have the registry as a git clone it will keep using it as a git clone instead of using the Pkg server, thats why you have to delete it.

getzze · 2020-03-23T23:01:09Z

Thanks, I didn't get that you have to remove the registries before.
However, I am using the development version of some packages so I need to fetch from git and one package failing to be fetched makes all the update fail.

tastyminerals · 2020-03-29T19:17:17Z

Running Julia 1.4.0. Not using any proxy. Windows 10 Pkg.add works, on Linux Manjaro for the same package it throws the error above :(

@tuhtah solution works though

tbhaxor · 2020-04-04T11:33:25Z

Hi, I am on manjaro Linux. In my case, this helped

mkdir -p /home/terabyte/.julia/registries/General && git clone https://github.com/JuliaRegistries/General.git ~/.julia/registries/General

Then in julia

julia> using Pkg

julia> Pkg.add("IJulia")

ghost · 2020-04-11T11:05:34Z

There is no need to delete any folder. With...

X@alienware: ~/ pacman -Q libgit2 julia
libgit2 1:1.0.0-1
julia 2:1.4.0-1

I get

X@alienware: ~/ julia -e 'using Pkg; Pkg.update()'
   Updating registry at '/usr/applications/academia/julia/registries/General'
   Updating git-repo 'https://github.com/JuliaRegistries/General.git'
┌ Warning: Some registries failed to update:
│     — /usr/applications/academia/julia/registries/General — failed to fetch from repo
└ @ Pkg.Types /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:1122
   Updating '/usr/applications/academia/julia/environments/v1.4/Project.toml'
 [no changes]
   Updating '/usr/applications/academia/julia/environments/v1.4/Manifest.toml'
 [no changes]

I originally applied @gwatcha workaround, however using

X@alienware: General/ pwd
/usr/applications/academia/julia/registries/General
                              --   --   (master  origin)     
X@alienware: General/ git pull

Does the same job. When I went back to julia and tried to upgrade the packages... they did upgrade:

(@v1.4) pkg> update
   Updating registry at `/usr/applications/academia/julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
┌ Warning: Some registries failed to update:
│     — /usr/applications/academia/julia/registries/General — failed to fetch from repo
└ @ Pkg.Types /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:1122
  Installed RecipesPipeline ─ v0.1.2
  Installed Plots ─────────── v1.0.8
   Updating `/usr/applications/academia/julia/environments/v1.4/Project.toml`
  [91a5bcdd] ↑ Plots v1.0.4 ⇒ v1.0.8
   Updating `/usr/applications/academia/julia/environments/v1.4/Manifest.toml`
  [91a5bcdd] ↑ Plots v1.0.4 ⇒ v1.0.8
  [01d81517] + RecipesPipeline v0.1.2
   Building Plots → `/usr/applications/academia/julia/packages/Plots/7U0ob/deps/build.log`

All this because I couldn't get Plots 1.0 from JULIA_PKG_SERVER=pkg.julialang.org. So, if you fancy/need the latest version of the packages, updating the registries by hand can do the trick until julia and libgit2 talk to each other again.

Edit: apparently it is now in pkg.julialang.org (only up to v0.28 when I tried). It's worth a shot if you don't want to be messing with folders and downgrading packages.

hzhangxyz · 2020-04-11T14:56:41Z

git issue same here

ERROR: failed to clone from https://github.com/JuliaRegistries/General.git, error: GitError(Code:ERROR, Class:Invalid, invalid version 0 on git_proxy_options)

env: Arch, julia1.4.0, libgit2 1.0.0

StefanKarpinski · 2020-04-12T16:05:02Z

If you're on Julia 1.4, you can do export JULIA_PKG_SERVER=pkg.julialang.org and you won't need to use git for Pkg operations. You'll need to do rm -rf ~/.julia/registries/General first to upgrade the General registry to using the Pkg protocol instead of git though.

nalimilan · 2020-04-12T18:06:42Z

That issue should be reported to Arch, they need a patch like #35233 if they use libgit2 1.0.

barche · 2020-08-21T10:43:08Z

For the record, I got hit by this again trying Julia 1.5 for the first time at work again, trailing slash needed on the proxy in ~/.gitconfig.

StefanKarpinski · 2020-08-21T13:31:10Z

Someone who cares about this is going to have to harass the libgit2 project and make sure they actually make a release that includes a bug fix for this. Clearly waiting for them to fix it on their own isn't working out.

TLDay · 2020-08-22T17:53:42Z

For me, this error happend because I'm using a socks5 proxy(.gitconfig, and git clone mannully works with socks5 proxy). Change to http proxy fixed it.

yalwan-iqvia · 2020-08-26T07:29:53Z

Jan 21

So I guess the question is whether we want to patch libgit2 or just wait until they make a release that includes this fix.

Aug 21

Clearly waiting for them to fix it on their own isn't working out.

So go with the former? (patch libgit2) You might even be able to contribute the patch to them.

Asking users to file issues/deal with upstream libraries is a little unfair, IMO (its the users who will care in the end, after all)

StefanKarpinski · 2020-08-27T15:55:28Z

Fair or not, I don't encounter this issue and I don't feel like mucking about in libgit2 to fix this. If you want to pay for someone to work on this, contact info@juliacomputing.com and we can figure out a consulting contract. You can also purchase JuliaTeam and then you won't have this issue because you can connect to a package server that's inside of your firewall, so you don't need to go through a proxy at all (there are lots of other nice benefits too). At the very least, you could make some noise on libgit2/libgit2#5220 instead of here. @fredrikekre is literally the only person who has posted there. Instead everyone bugs us and I guess expects us to harass the the libgit2 developers for them? If everyone who has posted here also posted a complaint on the libgit2 repo they might have fixed it by now.

StefanKarpinski · 2020-08-27T16:06:56Z

I have now pinged everyone who has posted here and in related issues where this libgit2 bug was the underlying cause of their problem on the relevant libgit2 issue. Please chime in there with your bug reports regarding this.

KristofferC · 2020-08-27T16:09:27Z

Maybe #35233 would help? I haven't looked into the patch notes.

caleb-allen · 2021-11-05T01:58:50Z

Can this change in Julia 1.7 be used as a workaround?

It is now possible to use an external git executable instead of the default libgit2 library for the downloads that happen via the Git protocol by setting the environment variable JULIA_PKG_USE_CLI_GIT=true.

https://github.com/JuliaLang/julia/blob/v1.7.0-rc2/NEWS.md#package-manager

bhawkins · 2023-02-16T19:46:13Z

I discovered that the JULIA_PKG_USE_CLI_GIT=true trick works in Julia 1.6.7 as well, which was a relief since I was a reluctant to migrate from the 1.6 LTS series. This was the only way I could find to get a SOCKS proxy working.

fredrikekre added the bug Indicates an unexpected problem or unintended behavior label Aug 29, 2019

fredrikekre added this to the 1.3 milestone Aug 29, 2019

fredrikekre added the external dependencies Involves LLVM, OpenBLAS, or other linked libraries label Aug 29, 2019

fredrikekre changed the title ~~Julia 1.3/master can not upgrade registries~~ LibGit2 can not fetch on Julia 1.3/master Aug 29, 2019

fredrikekre changed the title ~~LibGit2 can not fetch on Julia 1.3/master~~ LibGit2 can not clone or fetch on Julia 1.3/master Sep 2, 2019

fredrikekre mentioned this issue Sep 5, 2019

Bad interaction with empty http.proxy in .gitconfig and GIT_PROXY_AUTO libgit2/libgit2#5220

Closed

fredrikekre added the upstream The issue is with an upstream dependency, e.g. LLVM label Sep 5, 2019

fredrikekre removed this from the 1.3 milestone Sep 6, 2019

fredrikekre mentioned this issue Sep 27, 2019

Git issues on 1.3.0-rc3 #33405

Closed

fredrikekre mentioned this issue Mar 23, 2020

Bump LibGit2 to 1.0.0 JuliaPackaging/Yggdrasil#638

Merged

fredrikekre mentioned this issue Apr 6, 2020

Pkg behind Corporate Proxy #35373

Closed

CSchoel mentioned this issue Apr 21, 2020

Release of next version of OMJulia? OpenModelica/OMJulia.jl#37

Closed

cycomanic mentioned this issue Jun 19, 2020

Can't Run pyjulia in Ubuntu 20.04 JuliaPy/pyjulia#384

Open

yalwan-iqvia mentioned this issue Aug 26, 2020

Certificate bundle being ignored? #37190

Open

fredrikekre mentioned this issue Apr 21, 2021

Can't install packages on corporate pc JuliaLang/Pkg.jl#2516

Closed

felixcremer mentioned this issue Jun 4, 2021

Pkg instantiate hangs forever behind proxy even if gitconfig is set up #41088

Open

demurgos mentioned this issue Sep 10, 2021

cargo audit always fails with "invalid version 0 on git_proxy_options" rustsec/rustsec#431

Closed

ViralBShah closed this as completed Mar 14, 2022

LibGit2 can not clone or fetch on Julia 1.3/master #33111

LibGit2 can not clone or fetch on Julia 1.3/master #33111

Comments

fredrikekre commented Aug 29, 2019

KristofferC commented Aug 29, 2019

fredrikekre commented Aug 29, 2019

fredrikekre commented Aug 29, 2019 • edited

omus commented Aug 29, 2019 • edited

fredrikekre commented Aug 30, 2019

fredrikekre commented Aug 30, 2019

fredrikekre commented Sep 3, 2019

KristofferC commented Sep 3, 2019

fredrikekre commented Sep 3, 2019

nalimilan commented Sep 3, 2019

omus commented Sep 3, 2019

fredrikekre commented Sep 5, 2019 • edited

fredrikekre commented Sep 5, 2019

nalimilan commented Sep 5, 2019

fredrikekre commented Sep 5, 2019

fredrikekre commented Sep 6, 2019

kcajf commented Sep 12, 2019 • edited

tkluck commented Oct 10, 2019

kcajf commented Oct 10, 2019

wsshin commented Oct 11, 2019 • edited

kcajf commented Oct 11, 2019

fredrikekre commented Mar 20, 2020

diabonas commented Mar 23, 2020

getzze commented Mar 23, 2020

tuhtah commented Mar 23, 2020

fredrikekre commented Mar 23, 2020

getzze commented Mar 23, 2020

tastyminerals commented Mar 29, 2020 • edited

tbhaxor commented Apr 4, 2020 • edited

ghost commented Apr 11, 2020 • edited by ghost

hzhangxyz commented Apr 11, 2020 • edited

StefanKarpinski commented Apr 12, 2020

nalimilan commented Apr 12, 2020

barche commented Aug 21, 2020

StefanKarpinski commented Aug 21, 2020

TLDay commented Aug 22, 2020

yalwan-iqvia commented Aug 26, 2020 • edited

StefanKarpinski commented Aug 27, 2020

StefanKarpinski commented Aug 27, 2020 • edited

KristofferC commented Aug 27, 2020

caleb-allen commented Nov 5, 2021

bhawkins commented Feb 16, 2023

fredrikekre commented Aug 29, 2019 •

edited

omus commented Aug 29, 2019 •

edited

fredrikekre commented Sep 5, 2019 •

edited

kcajf commented Sep 12, 2019 •

edited

wsshin commented Oct 11, 2019 •

edited

tastyminerals commented Mar 29, 2020 •

edited

tbhaxor commented Apr 4, 2020 •

edited

ghost commented Apr 11, 2020 •

edited by ghost

hzhangxyz commented Apr 11, 2020 •

edited

yalwan-iqvia commented Aug 26, 2020 •

edited

StefanKarpinski commented Aug 27, 2020 •

edited