Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for private packages #521

Closed
jamiebuilds opened this issue Oct 6, 2016 · 70 comments
Closed

Support for private packages #521

jamiebuilds opened this issue Oct 6, 2016 · 70 comments

Comments

@jamiebuilds
Copy link
Contributor

In order to allow installing private packages Yarn will need to send a token to the headers of the request.

Private packages are @scoped/packages that were published with npm publish --access=restricted. The permissions of packages are managed through npm access and npm team which are not yet added

In the npm client, this token comes from the .npmrc and looks like this:

@nameofscope:registry=https://registry.npmjs.com/
//registry.npmjs.com/:_authToken=abc123

And it gets sent as this header:

Authorization: Bearer abc123
# alternatively:
Authorization: Basic username:password # <= base64

There's a package for retrieving the token. Although we may not want to store the token the same way npm does.

This token gets added to .npmrc on npm login. But yarn login doesn't even authenticate (it only stores username and email), so we may want to force the user to authenticate on install (in which case we need to solve scripting these installs for CI servers through some kind of environment variable).

We also need to make sure that Yarn users don't accidentally publish something publicly.

@jamiebuilds jamiebuilds changed the title Support for private packages on public registry Support for private packages Oct 6, 2016
@sebmck
Copy link
Contributor

sebmck commented Oct 6, 2016

We already have npm login and auth logic here. Just need to sort out the workflow.

@cpojer cpojer added this to the 1.0.0 - Open source milestone Oct 6, 2016
@cpojer cpojer removed this from the Open Source milestone Oct 11, 2016
@chicoxyzzy
Copy link
Contributor

chicoxyzzy commented Oct 11, 2016

Private registry doesn't always need auth token. For example we access our private registry through corporate VPN.

@knksmith57
Copy link

knksmith57 commented Oct 11, 2016

^^ Agreed. Allowing for the association of a separate registry per scope is sufficient for us (and I suspect many others).

@jmonster
Copy link

in which case we need to solve scripting these installs for CI servers through some kind of environment variable

@djMax
Copy link

djMax commented Oct 13, 2016

When we say "we already have this logic" - I don't see any path where an Authorization header would be sent to a registry. If there was one, perhaps there'd be a temporary workaround to make this all work while something more final is sorted out. Am I missing something?

@djforth
Copy link

djforth commented Oct 13, 2016

+1 looks like scoped packages even if they are public seem to fail.

@frederickfogerty
Copy link

To further @djforth's comment, I just installed from master, and I'm getting the same error - scoped packages are failing. It converts the / in the package name into %2f, which means the request to npm to find the package fails.

e.g. Error: https://registry.yarnpkg.com/@company%2fdata: Not found

@djMax
Copy link

djMax commented Oct 16, 2016

That's the way it fails if auth is required. I got it to work for public scoped packages

devongovett added a commit to devongovett/yarn that referenced this issue Oct 17, 2016
Sends the auth token for scoped packages, which may be private. Fixes yarnpkg#1134 and yarnpkg#521.
@devongovett
Copy link
Contributor

Should be fixed by #839 and #1146.

devongovett added a commit to devongovett/yarn that referenced this issue Oct 17, 2016
Sends the auth token for scoped packages, which may be private. Fixes yarnpkg#1134 and yarnpkg#521.
cpojer pushed a commit that referenced this issue Oct 17, 2016
Sends the auth token for scoped packages, which may be private. Fixes #1134 and #521.
@vjpr
Copy link

vjpr commented Oct 18, 2016

@devongovett I ran into a few issues:


This is the check for whether auth should be used:

    if (this.token || (alwaysAuth && requestUrl.startsWith(registry))) {
      headers.authorization = this.getAuth(pathname);
    }

If registry is http://registry.npmjs.org/ then an https request will fail to have auth attached because of requestUrl.startsWith(registry).


I had yarn config get registry set to registry.yarnpkg.org and that was being used when trying to get my private module, instead of using @my-org:registry': 'https://registry.npmjs.org/',.


So the fix for me was:

//if (this.token || (alwaysAuth && requestUrl.startsWith(registry))) {
if (this.token || (alwaysAuth)) {

I was also getting an initial call to the NpmRegistry#request to @my-org%2fmodule.

@devongovett
Copy link
Contributor

Yes, currently it replaces https://registry.npmjs.com/ with https://registry.yarnpkg.com/ here, which confuses the check here.

@vjpr
Copy link

vjpr commented Oct 20, 2016

EDIT: Ignore this post - it just started working for some reason.

I had to make sure to login to the scope on npm, using npm adduser --registry=http://registry.npmjs.org --scope=@foo --always-auth.


When I run:

npm3 adduser --registry=http://registry.npmjs.org --scope=@foo --always-auth

My npm looks like this:

_auth="xxx"
email=foo@gmail.com
strict-ssl=false
//registry.npmjs.org/:_authToken=xxx
registry=http://registry.npmjs.org/
@foo:registry=http://registry.npmjs.org/
save=false
save-exact=false
save-prefix=^
always-auth=true

NpmRegistry#getAuth looks like this:

  getAuth(packageName: string): string {

    if (this.token) {
      return this.token;
    }

    for (let registry of [this.getRegistry(packageName), '', DEFAULT_REGISTRY]) {
      registry = registry.replace(/^https?:/, '');

      // Check for bearer token.
      console.log({registry})
      let auth = this.getScopedOption(registry, '_authToken');
      if (auth) {
        return `Bearer ${String(auth)}`;
      }

      // Check for basic auth token.
      auth = this.getScopedOption(registry, '_auth');
      if (auth) {
        return `Basic ${String(auth)}`;
      }

      // Check for basic username/password auth.
      const username = this.getScopedOption(registry, 'username');
      const password = this.getScopedOption(registry, '_password');
      if (username && password) {
        const pw = new Buffer(String(password), 'base64').toString();
        return 'Basic ' + new Buffer(String(username) + ':' + pw).toString('base64');
      }
    }

    return '';
  }

It ends up using the authorization header Basic xxx. It is using the _auth key.

@ikosenn
Copy link

ikosenn commented Oct 23, 2016

Hey,
Has anyone managed to publish to a private npm registry created with Sinopia. I am able to do so with npm publish but yarn publish takes forever on the publishing step. I have changed the registry with yarn config set registry. Something else I noted I am not prompted for my password in the login step

@Tapppi
Copy link

Tapppi commented Oct 24, 2016

Is a fix on the way for private packages? The problem @devongovett described above just bit me in CI. My current workaround is to yarn config set registry https://registry.npmjs.org/ so that yarn sets the auth token on requests for private packages.

@rovansteen
Copy link

I'm also running in the issue that yarn login doesn't ask for a password, therefore I am not able to use Gemfury (https://gemfury.com). I am not sure if it's related to this issue though. Should I create a separate issue for this?

@jmonster
Copy link

Another use case I haven't seen mentioned:

git repositories can be fetched via https or ssh. If the repo is private, you need credentials (duh). When deploying to Heroku, .netrc is the optimal way to authenticate using the .netrc buildpack

@jamiebuilds
Copy link
Contributor Author

@rovansteen yarn login intentionally does not ask for a password. We do not want to store credentials or api tokens because that's a bad security practice

@sylvesteraswin
Copy link

I am also having the same issue with Sinopia. Did anyone find a solution for this?

@BohdanTkachenko
Copy link

It does not work properly in all environments with .npmrc located at ~/.npmrc. On my local machine it works fine, but when I'm running this in Docker, it does not see ~/.npmrc when cwd is not ~. You can check it with yarn config list command.

On my local machine it outputs:

yarn config v0.23.2
info yarn config
{ 'version-tag-prefix': 'v',
  'version-git-tag': true,
  'version-git-sign': false,
  'version-git-message': 'v%s',
  'init-version': '1.0.0',
  'init-license': 'MIT',
  'save-prefix': '^',
  'ignore-scripts': false,
  'ignore-optional': false,
  registry: 'https://registry.yarnpkg.com',
  'strict-ssl': true,
  'user-agent': 'yarn/0.23.2 npm/? node/v7.9.0 darwin x64',
  lastUpdateCheck: 1492804696073 }
info npm config
{ '//npm.example.com/:_authToken': 'XXXXX-YYYYYY-ZZZZZ',
  '@example:registry': 'https://npm.example.com/' }
✨  Done in 0.05s.

While inside of Docker it outputs:

root@a1c3c4fb1fb8:/app# yarn config list
yarn config v0.23.2
info yarn config
{ 'version-tag-prefix': 'v',
  'version-git-tag': true,
  'version-git-sign': false,
  'version-git-message': 'v%s',
  'init-version': '1.0.0',
  'init-license': 'MIT',
  'save-prefix': '^',
  'ignore-scripts': false,
  'ignore-optional': false,
  registry: 'https://registry.yarnpkg.com',
  'strict-ssl': true,
  'user-agent': 'yarn/0.23.2 npm/? node/v7.9.0 linux x64',
  lastUpdateCheck: 1492856034840,
  version: '0.23.2' }
info npm config
{ version: '0.23.2',
  loglevel: 'info' }
Done in 0.03s.

So it looks like it does not execute npm config correctly.

As temporary workaround for this, in Docker I just copy ~/.npmrc to /app/.npmrc.

@stereobooster
Copy link

found here https://github.com/uber/react-map-gl

yarn start v0.23.2
$ (cd examples/custom-interactions && (path-exists node_modules || yarn) && yarn run start-local)
sh: path-exists: command not found
yarn install v0.23.2
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error An unexpected error occurred: "https://unpm.uberinternal.com/flow-remove-types/-/flow-remove-types-1.1.2.tgz: Request failed \"401 Unauthorized\"".
info If you think this is a bug, please open a bug report with the information provided in "/react-map-gl/examples/custom-interactions/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
error Command failed with exit code 1.

@elibal
Copy link

elibal commented May 8, 2017

I am also having the same issue with kendo-angular components.

C:\WorkingFolder\Projects\NG4\wck-management>yarn
yarn install v0.23.4
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
error An unexpected error occurred: "http://registry.npm.telerik.com/@progress%2
fkendo-angular-buttons/-/kendo-angular-buttons-1.0.0.tgz: Request failed "503 S
ervice Unavailable"".
info If you think this is a bug, please open a bug report with the information p
rovided in "C:\WorkingFolder\Projects\NG4\wck-management\yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this
command.

@bestander
Copy link
Member

It should be fixed now.
If you have some custom example where it does not work please open a new issue.
It is important to provide steps to reproduce in this cases.

@kachkaev
Copy link

Awesome @bestander! What's the minimum yarn version where it’s expected to work?

@bestander
Copy link
Member

bestander commented May 23, 2017 via email

@klofi
Copy link

klofi commented May 25, 2017

I can confirm that private scoped packages with scope and registry defined in .npmrc started working in Yarn 0.24.6 (did not work in Yarn 0.24.5). Thank you!

@Mart112358
Copy link

yarn install
yarn install v0.24.6
info No lockfile found.
[1/4] Resolving packages...
warning cldr-data > cldr-data-downloader > npmconf@2.0.9: this package has been reintegrated into npm and is now out of date with respect to npm
warning cldr-data > cldr-data-downloader > request > node-uuid@1.4.8: Use uuid module instead
[2/4] Fetching packages...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
error An unexpected error occurred: "http://registry.npm.telerik.com/@progress%2fkendo-angular-inputs/-/kendo-angular-inputs-1.0.3.tgz: ESOCKETTIMEDOUT".
info If you think this is a bug, please open a bug report with the information provided in "[...]\yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

@BAMdz
Copy link

BAMdz commented May 30, 2017

yarn install v0.24.6
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "http://registry.npm.telerik.com/@progress%2fkendo-angular-l10n/-/kendo-angular-l10n-1.0.0.tgz: connect ETIMEDOUT 23.253.4.114:80".
info If you think this is a bug, please open a bug report with the information p
rovided in "....\yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this
command.

@bestander
Copy link
Member

bestander commented May 30, 2017

Looks like you can't connect to telerik.com, is http proxy configured?

@BAMdz
Copy link

BAMdz commented May 30, 2017

The problem is yarn are searching by http://registry.npm.telerik.com, when I have configured yarn with strict-ssl to true:

info yarn config
{ 'version-tag-prefix': 'v',
'version-git-tag': true,
'version-git-sign': false,
'version-git-message': 'v%s',
'init-version': '1.0.0',
'init-license': 'MIT',
'save-prefix': '^',
'ignore-scripts': false,
'ignore-optional': false,
registry: 'https://registry.yarnpkg.com',
'strict-ssl': true,
'user-agent': 'yarn/0.24.6 npm/? node/v6.9.5 win32 x64',
lastUpdateCheck: 1496137030541 }
info npm config
{ 'strict-ssl': true,
'@progress:registry': 'https://registry.npm.telerik.com/',
'//registry.npm.telerik.com/:_authToken': '......' }
Done in 0.04s.

Previously I've configured the login with npm: "npm login --registry=https://registry.npm.telerik.com/ --scope=@progress"

It is neccesary that yarn searching for by "https" (https://registry.npm.telerik.com) :-)

Any idea what is the problem?

@bestander
Copy link
Member

bestander commented May 30, 2017 via email

@BAMdz
Copy link

BAMdz commented May 30, 2017

OK, I will create a project, with a telerik trial account, to test the problem and I will send you the project link in github.

@bestander
Copy link
Member

That would be great, @beatrizaldaz.
Can you open a new issue just for that case then?
It would be easier to track it isolated.

@dmiorandi
Copy link

About @beatrizaldaz post / Telerik. I've got same issue. In detail I've made some attemps
using following config (.npmrc). It seems almost to work but connections is made in http instead https
so is refused. Are there any temporary workaround about this (strict mode does not work)?

@progress:registry=https://registry.npm.telerik.com/
//registry.npm.telerik.com/:_authToken="YOUR_SECRET_HERE"
always-auth=true
registry="https://registry.npmjs.com/"

@balanceiskey
Copy link

balanceiskey commented Jun 14, 2017

So I just ran into this yesterday (yarn was at 0.24.6). I'm not sure what exactly caused it as it's been working fine for awhile. My solution was to remove both the .npm folder and .npmrc file altogether, run yarn cache clean, login again with npm login and things appeared to work fine after that. I've been jumping between versions of node and npm via nvm more lately, possible culprit? Worth noting, I also uninstalled and reinstalled yarn with brew using the --ignore-dependencies flag at some point during troubleshooting, but that by itself did not resolve the issue.

@balanceiskey
Copy link

One more note, if it's even relevant, when attempting yarn login during the course of these steps it would hang on the password prompt for some reason, which is why I did npm login.

@olalonde
Copy link

Related: #2738

@demurgos
Copy link

Hi,
It's been a few months: are there any news?

@datashaman
Copy link

For future travellers, .yarnrc.yml:

npmScopes:
    myScope:
        npmRegistryServer: 'https://npm.myregistry.com'
        npmAuthToken: 'myauthtoken'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests