[Bug][gitextractor] Azure devops repos fail to clone with 400 errors

[wouldd]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Soemwhere around the v1.0 release the underlying implementation for cloning git repos was changed, and since that point cloning azure deveops git repositories has not been working (on prem, though I would assume cloud is the same)
The error reported is a 400 response.

What do you expect to happen

I expected git repos to continue to clone successfully, they were working fine before I updated the code base (to pick up an unrelated fix)

How to reproduce

Configure the azure devops connector, provide a valid PAT and setup one or more repositories. run collect data.
I believe this would happen to azure cloud, if not then this could be specific to azure devops server which I know is technically unsupported. however this was working prior to picking up v1 changes and I can make it work with code changes now

Anything else

So from what I can tell the azure devops git server requires an AUTHORIZATION Basic header to be set with the base64 encoded version of an empty user and the pat token, eg- : encoded as base64. It seems to need this in addition to the values being passed on the commandline in the url where we are setting https://git:@url
In testing a fix locally I found that invoking the git cli directly as happens from this line: https://github.com/apache/incubator-devlake/blob/8a6bbe2b84d709e30d329cc7317ac85aa2398f10/backend/plugins/gitextractor/parser/clone_gitcli.go#L170C32-L171C2
did not work, something about hte way go was passing the parameters to the command were corrupting the header. However if I isntead call bash and pass the git command as an argument then repository cloning works no problem

Obviously my local fix is very specific to me, in taht I know I'm talking to azure devops so I can extract the PAT from the passed in url and encode it into a header etc. I'm not sure what hte general purpose fix for something like this would be.

Version

master

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[klesh]

Hi, can you share the error message of the 400 response? Thanks.

[wouldd]

@klesh sorry I didn't capture the exact wording, and my machine rebooted since. obviously I'm now running a version with my own local fix. Ostensibly it was just a 400 bad request response from AZD reported by the git cli. I guess I'm assuming that other people using AZD would be seeing the same issue with the latest code?

I can confirm that for me switching the base command to being bash and the args to be:
args := []string{"-c", "git -c http.extraHeader="Authorization: Basic "+sEnc+"" clone "+ taskData.Options.Url+" "+localDir+" --bare --progress"}

fixed my issue. where I obviously pulled the PAT out of the Options.Url anad b64 encoded it into the sEnc value.

[klesh]

@wouldd It is working fine on my local machine, I wonder if it is caused by the configuration on the Azure Portal. Is it possible for you to provide a Restricted token and Repo for me to investigate? You may find me on the Slack

[wouldd]

So we are not using azure devops services, we're currently still using on-prem. so I guess if this is working for you then the issue relates to something odd about the on-prem setup. I'm happy to consider this a quirk of me being unsupported pointing to our server rather than services.
It's weird that it used to work prior to the code changes in the gitextractor logic. but I suppose for now I can continue to solve for our case with my local fix.