Comments (27)
I am working on support for private packages in hex.pm. You can use private repositories right now, but you also have to mirror hex.pm also if you want to use packages from hex.pm as well as internal packages.
I am interested in supporting multiple repositories but it's not currently on the roadmap.
from hex.
We already support this through the environment variables HEX_URL
and HEX_CDN
.
In the future I want more sophisticated support for multiple registries so that you can use both the main hex.pm and your internal company registry at the same time. But that requires some major changes internally and I haven't decided yet how it should work. It is still quite low on the priority list because no one has said yet that they will use this feature.
I'm sure that people will ask for it in the future so I definitely want to support it and I'm thinking about the best solution.
from hex.
I think adding multiple registries so that you can use both the main hex.pm and your internal company registry at the same time would be fairly simple.
One way it could be tackled is by setting a master and sub repositories. This would work in a way that we have a tree where each repository is at a different layer in the tree. This would mean that if you had work
higher in the tree than public
, the client would check with the work
repository for a package before public
. So if your work
repo had a private fork of, say, cowboy
, it would pull your fork before the public
version due to the layout of the tree. This would allow for virtually infinite tree levels.
It would also be wise to add an option to explicitly set the repo
to pull from using an atom in the Mix deps
, possibly like so:
{:conform, "~> 0.10.0", repo: :work}
Any comments or thoughts?
from hex.
Yep, that's a possible solution.
We would also need to write the registry url to the mix.lock
file to ensure repeatable builds. But what if the registry is private, is there a chance that we can leak private information if someone accidentally shares their mix.lock file? Maybe it is fine if we just add support for authentication to retrieve the private registry?
I think that using a package outside of the main hex.pm registry should always require repo: :whatever
instead of having the tree of registries.
from hex.
I think we should highly recommend locking servers run privately, but ultimately it shouldn't force locking, in the case of mirrors, etc.
Adding to the mix.lock
is a great idea, I'm surprised I overlooked it.
Would we add the option to define repository {cdn, api}
's in the mix.exs
?
This could be a security issue if the Mix file is leaked in a private environment, but in the future, if someone is just using a mirror, this would be fine.
from hex.
Also, I think it would be an idea to define the default repository in the Mix file. In this case,
{:conform, "~> 0.10.0", repo: :hex}
Although I can't personally see a use for it, being able to disable the use of the public :hex
repository would be useful in organizations where only internal packages can be used.
If a repo is not defined on the system, we simply throw an error and prematurely exit Mix.
And now that I think about it, if we eventually add the ability to lock the repo, we could use several methods to do this authentication. Be it stored in the Mix file, the ~/.hex/endpoints.config
file, or simply prompted for on installation.
from hex.
There might be an issue when you have defined a dependency as {:ecto, "~> 0.1.0", repo: :other}
and someone else is using your project and has another repo named :other
in their hex config. Maybe we should just go with a straight URL in mix.exs
.
We need to look at how other package managers, such as npm and rubygems, support multiple repositories.
from hex.
NPM
The only support NPM has for multiple repositories is to run npm --registry http://registry.npmjs.eu/ install express
or by running npm set registry http://abc.xyz
where http://abc.xyz
is a couchdb replicated from NPM.
Maven
Maven uses the following format in it's pom.xml
<project>
...
<repositories>
<repository>
<id>my-repo1</id>
<name>your custom repo</name>
<url>http://jarsm2.dyndns.dk</url>
</repository>
<repository>
<id>my-repo2</id>
<name>your custom repo</name>
<url>http://jarsm2.dyndns.dk</url>
</repository>
</repositories>
...
</project>
PyPI
PyPI uses this format:
Single Usage: pip install -i http://<mirror>/simple <package>
Global settings: Add ~/.pip/pip.conf
that includes:
[global]
index-url = http://<mirror>/simple
RubyGems
RubyGems allows users to set the following in the Gemfile:
source "http://your.servers.ip:9292
Nuget
Nuget has the best support I've seen so far:
To add a repo, we run nuget sources Add -Name Artifactory -Source http://localhost:8081/artifactory/api/nuget/<repository key>
. After this is run, the source is added to a list, and packages are pulled from these sources. i believe it is expected that packages have unique identifiers, but I'm not 100% sure.
Conclusion
That's all the examples I could find, and other than NuGet and Maven, most of them don't actually do the same as this proposal, as they only allow one mirror/repo at a time!
from hex.
Possibly we could have global repos that are overridden in the Mixfile? Or if we're on a system where we're distributing to brand new clients that dont have the alias, we could substitute a named atom alias for a tuple with the api url and the cdn url.
from hex.
Thanks for the great compilation of the package managers ❤️.
So it seems like our proposal isn't too crazy. I would go with just a literal repo in the mixfile. Example: {:ecto, "0.1.2", repo: "http://path/to/registry.ets.gz"}
.
To publish and create accounts and so on you can do HEX_URL=... mix hex.publish
. We can tag the accounts by url in hex.config
so you can have multiple accounts for different servers.
from hex.
I also think the ability to globally assign registries to atoms would be nice, such as :es
or :eu
from hex.
Very nice! If we are allowing atoms, I believe they should be a key in the project config instead of something global. But we probably don't need atoms from the start. I agree we likely need to store them in the lock file too.
from hex.
Btw, this is relevant to this discussion: https://groups.google.com/forum/#!topic/ruby-security-ann/Rms5sZhLxdo (i.e. how to handle ambiguity if we go with something like Maven or Bundler).
from hex.
Interesting, thanks for the link @josevalim.
Atoms are unnecessary if we are going to store it in the mixfile anyway:
@company_repo "http://company.com/registry.ets.gz"
defp deps, do: {:ecto, "0.1.2", repo: @company_repo}
from hex.
Will this registry.ets.gz
be pulled down on each build, or cached and only updated when changed?
from hex.
Like today, it will be cached and only updated when changed.
from hex.
I have a couple queries about this:
- Just for clarification, will this be a global cache, or a project independent cache?
- If one or more packages sets
repo:
, should we force users to set it on each package to specifically request where packages are pulled from? - Looking at the link @josevalim sent, could we implement a DSL to automatically set the
repo:
field of packages in a block, such as
defp deps do
repo @company_repo do
[{:ecto, "~> 0.3.0"},
{:plug, github: "elixir-lang/plug"}]
end
end
which would set the repo:
field for each package in the set to @company_repo
from hex.
- The registry are globals.
- I think it is fine to assume the default package is Hex default (so the answer is no)
- No DSL for now (we can discuss it if it ever becomes an issue)
from hex.
After some further thought and consideration, I've come up with this proposal. Feedback would be appreciated. //cc @ericmj @josevalim
To pull a specific package, we can set the repo:
field with a tuple that contains two fields:
@store "https://cdn.hex.company.com/"
@api "https://hex.company.com/api/"
@company_repo {@store, @api}
# note: we need the base url of the store (not the registry.ets.gz) due to the fact that
# the registry doesn't contain the package url
# This can be used like so:
defp deps, do: {:ecto, "0.1.2", repo: @company_repo}
Once one or more packages have their repo:
field set, all packages must explicitly define their repo:
. (Or else, we just warn the user not all packages have their repo:
field set, as it can get confusing having dependencies that are marked with a repo:
field and others without).
Finally, it would be a good idea to check that the supplied CDN and API URLs are accessible prior to staring dependency resolution, as this may cause issues when installing dependencies.
The only thing I'm not sure about is if we should allow packages to be pushed to the public repo that include packages from private repositories?
from hex.
It is fine to always fallback to hex.pm when :repo
is not set. I don't there is any need for the API URL there. You are not allowed to have packages with dependencies from another repository.
from hex.
Come to think of it, we won't need to API URL to be included. Should we just throw an error and kill the upload of users attempt to upload a package that has dependencies from another repository?
from hex.
Just to add what I have discussed with @ericmj. Registries must be self-contained: i.e. a registry can only depend on packages it contains. This may be too restrictive at first but it is always a good way to start with the most restrictive design and then expand on it.
That said, once we specify the :repo option for a dependency, all dependencies of that dependency must come from the same :repo. We should also store the :repo in the lock file and use it when checking if two hex packages are equal. I.e. if two hex packages come from different repos, they should diverge and the user then needs to resolve the divergence in their mix.exs file.
from hex.
Is this supported now, or being considered for the future? No private repository support is an issue in corporate environments where you have lots of internal software releases.
from hex.
That said, once we specify the :repo option for a dependency, all dependencies of that dependency must come from the same :repo.
Why must this be the case? If I'm building a private package that depends on Ecto, for example, this would mean I have to duplicate Ecto into my registry and ensure I update it as Ecto updates.
from hex.
Why must this be the case? If I'm building a private package that depends on Ecto, for example, this would mean I have to duplicate Ecto into my registry and ensure I update it as Ecto updates.
That comment was from over two years ago :). The :repo
option is not live yet but when it is you can use it to set where the dependency should be sourced from. The only restriction is for the hex.pm repository, for it you can only depend on other packages from hex.pm.
from hex.
@ericmj Where can I track the status of the :repo
option? Definitely would like to know when that goes live. Thanks!
from hex.
Nvm...found it!
b05096e
https://github.com/hexpm/hex/releases/tag/v0.17.1
from hex.
Related Issues (20)
- Release canditate version resolution incorrect after two digits HOT 3
- Retire a package completely? HOT 3
- Full remove a package from hex.pm HOT 2
- [feature request] Sort `mix hex.outdated` output by status in default output HOT 4
- rebar3_auto - the package in HEX contains module which does not exist in the rebar3_auto repository HOT 1
- Error: Lock is missing HOT 2
- Bundled CA certs are not working now
- Fail to load function 'Elixir.Hex.Netrc.Cache':fetch/1: op make_fun2 u: on Erlang/OTP 27 rc.1
- Publish New Release for Updated CA-Bundle HOT 4
- Bug on dependency resolution for package published on 2/29 HOT 1
- Internal server error when attempting to publish package HOT 1
- mix hex.outdated --ignore <lib> HOT 2
- How to generate a fingerprint for public_key? HOT 5
- Suggesstion: Show readme file content on package page HOT 1
- Build .hex/cache.ets file offline for secure build applications HOT 6
- hex dependency resolution shows order dependent behaviour? HOT 1
- Crash during Mix.install([:jason]) HOT 4
- Installing github dependencies broken on main HOT 5
- Could not install Rebar because Mix could not download metadata HOT 1
- Dependency Resolution behaves differently under OTP-26 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hex.