Authenticate is called for every package install about verdaccio-gitlab HOT 29 CLOSED

We set up verdaccio for private modules. We want to use the features of npm and not install our (proprietary) modules from GitLab or publish them on npm (registry.npmjs.org)...

Would be great to use GitLab as authentication provider, because we already use it. Caching the groups would be it. Group owners are not changing very frequently.... Do you have any idea how that could be achieved?

from verdaccio-gitlab.

PR’s are welcome!

from verdaccio-gitlab.

Just FYI, I'm working on an implementation using node-cache as suggested by @bufferoverflow. It's not a perfect solution, but a small, manageable implementation. By comments of the maintainer in verdaccio/verdaccio/issues/732, I don't think support for it will be added in the parent project.

from verdaccio-gitlab.

I think it makes sense to use https://www.npmjs.com/package/node-cache

from verdaccio-gitlab.

We tried to implement a cache for the requests to GitLab and failed, because it seems that the file is reloaded every time and we need a persistent cache in a file for example. Caching in a file could be a bad idea, because I don't know how this file is loaded and if it is threadsafe.

from verdaccio-gitlab.

Why do you have to limit access? I suggest to only use authentication for publishing. As you figured out there is no caching of credentials in place.

from verdaccio-gitlab.

Any chance this is going to be fixed?

from verdaccio-gitlab.

I got here because of similar issue. The ldap plugin also get spammed and it is not possible to use user account.

It look not possible for that plugin,

We wanted to try that one too, GitLab should handle less the spam than OpenLdap but the ldap plugin does not work well at all for the same reason: Alexandre-io/verdaccio-ldap#33

From what I understand, the fix should be done in verdaccio, but right now there is no persistence layer in verdaccio.

from verdaccio-gitlab.

https://github.com/idangozlan/verdaccio-bitbucket/blob/master/src/cache.js could be an option to implement this, see also its usage https://github.com/idangozlan/verdaccio-bitbucket/blob/master/src/index.js#L92

from verdaccio-gitlab.

Interesting, the thing is that it use the application memory to store the object.
This will still result in multiple login if you scale verdaccio up. This could definitively work for the ldap plugin. Why don't you use this in the gitlab plugin? Do you prefere a db service?

from verdaccio-gitlab.

It would make sens for:

• htaccess auth
• ldap auth
• gitlab auth

Because it will solve the authentication spam.

Note that the difference will be minimal for htaccess, as it usually exist on the same host memory.

If you want to scale up verdaccio, we would definitly need an external database that can be reach by any instance.

I suggest the following to be done for solving the issue:

1. Implement node-cache or the bitbucket fix to solve auth spam issue on local instance.
2. Implement a database for scaling verdaccio to multiple instance

I would personally go directly for two, but I don't have enough time right now, I hope I will within this year

from verdaccio-gitlab.

@dlouzan very good news, let us know when you need people for testing it. I may be able to find time.

from verdaccio-gitlab.

PR in progress; it would be very good if at least two people could review / test this locally. This PR builds on top of another one I made yesterday, and it changes quite a lot of things in the auth handling (I hope for the good).

Definitely some automated testing is needed, but we will see :-)

from verdaccio-gitlab.

@kopax I would appreciate if you could test #23 and comment over there.

from verdaccio-gitlab.

Oh too bad =/, I thought you were doing it on the https://github.com/Alexandre-io/verdaccio-ldap which is the one I wanted to use. Both plugins share the same problem.

I will test it for you now, and will tell you. Maybe this can be implemented in verdacio-ldap too if it work?

from verdaccio-gitlab.

Ok, I'm sorry for you but I'm not sure how you though this was being done in the ldap plugin, we are on the gitlab plugin repo xD

The cache is completely generic, so adaptations to other plugins are quite easy.

from verdaccio-gitlab.

@dlouzan I think I was watching other repo to see how other dealted with the issue.

I got delayed today but I will definitively test it within next week. Maybe we will replace our LDAP auth with this one.

@juanpicado from verdaccio repo recommended here verdaccio/verdaccio#732 (comment) to implement this in a separate module. This way it could be used on top of any other plugin.

I am not sure how it is supposed to be done as I have not found one yet working like that and haven't worked on plugins yet.

from verdaccio-gitlab.

I am building a docker image in order to perform testing right now.

• Dockerfile:
  • is having old version 2.7.3 instead of 3.2.0 (I personally use 3.0.2)
  • is having error with the current image choosed during build, I instead do FROM node:10.1.0-alpine
• package.json
  • is having error with "files": ["lib/verdaccio.js"],
    It can be corrected by lib/*.js but it is not the recommended. One of the reasons we have forbidden the use of files and instead use .npmignore is because, with it, any file added by a developer would be included instead of missing which can break your user's application. You can take this example as a proof.

from verdaccio-gitlab.

@kopax you need to define gitlab: true within packages section, see https://github.com/bufferoverflow/verdaccio-gitlab#use-it then owner of groups are allowed to publish.

from verdaccio-gitlab.

That was the part I understand @bufferoverflow .

The part I am not understanding is how it works.

owner of which group is allowed to publish in where?

Again, a clear example would help.

from verdaccio-gitlab.

If you are owner of a group within gitlab, then you are allowed to publish a package or a package scope with the same name. So if you are owner of mycom and myorg on GitLab you can publish scoped packages @mycom/* and @myorg/* or mycom and myorg packages.

from verdaccio-gitlab.

Just a side note: you are also owner of packages under your user name as group.

from verdaccio-gitlab.

@kopax I can make another PR, but I don't understand everything (the formatting of your comment is a bit chaotic):

• I agree the verdaccio dockerfile version should be updated, I think it should be the same version as in package.json (as of today 3.1.1). I didn't really work with the docker build, it needs some work.
• I don't understand the issue with files in the build, could you please elaborate?
• I'm not sure I get the sentence We at first could login due to missing personal access token, but then in the log below a 401 is shown, which is expected, :-?

from verdaccio-gitlab.

I agree for the formatting.

About Dockerfile, you can update it directly. I am not sure if the image is released to the Docker hub but I believe it should be. (Travis-CI should be used for that but the owner must configure the publishing).

So far, I think the Dockerfile must be built by the client and deployed somewhere, @bufferoverflow needs to confirm.

Because of that, I had to manually add the verdaccio-gitlab package in the container in order to perform the installation (it was not released on npm at that time).

I have used npm pack which prepare the .tar.gz as it will be published on npm registry and the archive was missing all your new sources.

If you look at files, you will see that you will only include one source, so the script will be broken for every users that are updating.

This was my Dockerfile after fixing files and doing npm pack (I have added to the image my configuration but that is not required), I was able to start with your plugin.

FROM node:10.1.0-alpine
LABEL maintainer="https://github.com/bufferoverflow/verdaccio-gitlab"

RUN apk --no-cache add openssl && \
    wget -O /usr/local/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 && \
    chmod +x /usr/local/bin/dumb-init && \
    apk del openssl

ENV APPDIR /usr/local/app

WORKDIR $APPDIR

ADD . $APPDIR
ADD verdaccio-gitlab-1.0.0.tgz /usr/local/app
ENV NODE_ENV=production

RUN npm config set registry http://registry.npmjs.org/ && \
	npm install -g [email protected] && \
	npm install -g /usr/local/app/verdaccio-gitlab-1.0.0.tgz
#    npm install -g [email protected] && \
    #npm install -g verdaccio-gitlab@latest

RUN mkdir -p /verdaccio/storage /verdaccio/conf
ADD conf/config.yaml /verdaccio/conf/config.yaml
ADD conf/htpasswd /verdaccio/htpasswd

#ADD docker-verdaccio-gitlab.config.yaml /verdaccio/conf/config.yaml

RUN addgroup -S verdaccio && adduser -S -G verdaccio verdaccio && \
    chown -R verdaccio:verdaccio "$APPDIR" && \
    chown -R verdaccio:verdaccio /verdaccio

USER verdaccio

ENV PORT 4873
ENV PROTOCOL http

EXPOSE $PORT

VOLUME ["/verdaccio"]

ENTRYPOINT ["/usr/local/bin/dumb-init", "--"]

CMD verdaccio --config /verdaccio/conf/config.yaml --listen $PROTOCOL://0.0.0.0:${PORT}

Also, I was suggesting you both to never used files in package.json and use .npmignore instead, because people break thing as you did and you cannot prevent that.

You can ignore the last dot

from verdaccio-gitlab.

@kopax Ok, so your suggestion regarding the files is to remove it completely from package.json, and then setup what should be included in the release in the .npmignore file, right?

from verdaccio-gitlab.

Yes, files should be banned from package.json.

You can't set up what should be included using .npmignore, you can set up what should not be included which is much better for your package consumers.

It prevents publishing things with the missing source that will break other's apps. It is safer to later add in .npmignore those files and republish.

from verdaccio-gitlab.

Ok, thanks for the insights. Later today I'll be publishing some corrections in the currently active PR fixing this. I'll make another one for fixing docker.

from verdaccio-gitlab.

.@bufferoverflow regarding configuration :

  '@*/*':
    # scoped packages
    access: $authenticated
    publish: $authenticated
    proxy: npmjs
    gitlab: true

I am trying to undersdtand how does that work ? Does all the scoped modules get published to the private and public registry ?

What if a scoped module is private and we do not want to use npmjs?

Is there any chances to get a release soon?

from verdaccio-gitlab.

That configuration means that in order to access or publish any scoped package, you need to be authenticated in verdaccio. On top, the gitlab plugin will only allow you to publish the package if either:

• Your user name is exactly the same as the scope of the package, or
• You are owner in gitlab of a group that matches the name of the scope (equivalently, as long as you are owner of a group, you can publish any package below it)

It might be worth to implement further access control at the read (allow_access method) level in the future, but right now in order to read the package you just need to be authenticated.

Packages are only published in the private verdaccio registry, no upstream publish is done.

from verdaccio-gitlab.