Failed to install a Github package due to metadata database update failure

[renozao]

@gaborcsardi Following my issue posted on remotes: r-lib/remotes#793

Recap:
I tried it now on my private package and got an error like this (installed pak from the binary built from Github):

✔ Updated metadata database: 3.05 MB in 8 files.                           
✖ Updating metadata database ... failed                                    
Error:                                                                      
! error in pak subprocess
Caused by error: 
! Could not solve package dependencies:
* <org>/<repo>@<commit>:
  * Can't install dependency memoise (>= 0.31.2) (>= 2.0.1) (>= 1.0.7.1)
  * Can't install dependency R6 (>= 0.31.2) (>= 2.0.1) (>= 1.0.7.1)
  * Can't install dependency Biobase (>= 0.31.2) (>= 2.0.1) (>= 1.0.7.1)
  * Can't install dependency stringr (>= 0.31.2) (>= 2.0.1) (>= 1.0.7.1)
  * Can't install dependency checkmate (>= 0.31.2) (>= 2.0.1) (>= 1.0.7.1)
  * Can't install dependency pkgmaker (>= 0.31.2) (>= 2.0.1) (>= 1.0.7.1)
  * Can't install dependency devtools (>= 2.0.1) (>= 1.0.7.1)
  * Can't install dependency github (>= 1.0.7.1)
  * Can't install dependency plyr (>= 1.0.7.1)
  * Can't install dependency jsonlite (>= 1.0.7.1)
  * Can't install dependency BiocManager (>= 1.0.7.1)
  * Can't install dependency knitr (>= 1.0.7.1)
  * Can't install dependency digest (>= 1.0.7.1)
  * Can't install dependency uuid (>= 1.0.7.1)
  * Can't install dependency reshape2 (>= 1.0.7.1)
  * Can't install dependency qs (>= 1.0.7.1)
  * Can't install dependency withr
* memoise: ! pkgdepends resolution error for .
Caused by error: 
! Invalid line 3 in PACKAGES file: must contain `:` @lib.c:392 (pkgcache_parse_packages_raw)
* R6: ! pkgdepends resolution error for .
Caused by error: 
! Invalid line 3 in PACKAGES file: must contain `:` @lib.c:392 (pkgcache_parse_packages_raw)
* Biobase: ! pkgdepends resolution error for .
Caused by error: 
! Invalid line 3 in PACKAGES file: must contain `:` @lib.c:392 (pkgcache_parse_packages_raw)
...
<list goes on>

There is no such issue when using remotes (except for the problem of the dependency upgrade of course).

• I tried with pak from both the CRAN version and the binary version installed with the command from https://github.com/r-lib/pak?tab=readme-ov-file#pre-built-binaries
• I don't know which PACKAGES file this is referring to. I assume it refers to CRAN-like repo index files. I am suspecting that somehow those files are not downloaded properly, which may be linked to particularities of my setup.
• Tried to look at how pkg_install() is fetching the index files, but following up the call chain is a bit hard: looks like pak is using packages installed in its own installation directory. 😅
  Isn't it using plain old download.file()?

[gaborcsardi]

What is getOption("repos")?

[renozao]

I am checking this indeed.
The repos points a private URL behind authentication. We have a special site setup that configures download.file() to use curl and with the necessary authorization bearer token in the header, only for urls within our domain.
In brief this calls download.file() after doing this:

withr::local_options(list(
      download.file.extra='-H "Authorization: Bearer <TOKEN>"',
      download.file.method='curl')
    )

Looking at how pkgcache downloads file to see if we can configure this to also work with pak.

But maybe there is a built-in way to configure private CRAN-like repositories and behind authentication?

[gaborcsardi]

But maybe there is a built-in way to configure private CRAN-like repositories and behind authentication?

No, there isn't currently. But we can add that, no problem. You essentially need to send an extra header?

[renozao]

Technically I need to send the authentication headers yes, but this would need to be a repo-specific setting.
The metadata database was correctly updated once I passed the Authentication header within pkgcache:::http_get().

What might be tricky is that I believe the token expires and then needs to be update by querying the authentication service. For this, one would probably need to enable specifying a repo-specific setup function that prepares the handle/headers.

[gaborcsardi]

Is this a protocol with a public spec, e.g. OAuth 2.0, or something in house?

[renozao]

I believe it is standard OAuth 2.0 (provided by google).
Note that passed the authentication, there still seems to be an issue with the fact that the dependency version requirement does not get resolved: when calling

pak::pkg_install('org/packageA@abcd123')

I get an error about not being able to find the right version for dependency packageB, despite the fact that commit org/repo@abcd123 contains a reference to org/packageB@<sha_with_required_version> in Remotes.

[gaborcsardi]

I get an error about not being able to find the right version for dependency packageB, despite the fact that commit org/repo@abcd123 contains a reference to org/packageB@<sha_with_required_version> in Remotes.

Can you try to create a reproducible example for this?