Global store about pnpm HOT 20 CLOSED

I think this issue can be reopened now that #244 has been merged and it works on new NodeJS

The one thing that has to be implemented for a global store is the deletion of packages that are not used anymore.

from pnpm.

damn... not that easy. just realized there's some painful edge cases here...

from pnpm.

Okay, let me try to explain. imagine pg-then requires pg, and they're not bundled together. They'll be set up like this:

• /project/node_modules/.store/[email protected]
• /project/node_modules/.store/[email protected]
• /project/node_modules/pg-then (symlink)
• /project/node_modules/pg (symlink)

If, within /project, you do require('pg-then'), Node.js will read ./node_modules/pg-then, which resolves to the one in the store.

Keep in mind require.resolve('pg-then') actually gives you the "real" path of /project/node_modules/.store/[email protected]/index.js.

pg-then will now use require('pg') within itself, and this will work. The reason why this will work is because Node.js will search these paths:

• /project/node_modules/.store/[email protected]/node_modules/pg
• /project/node_modules/.store/node_modules/pg
• /project/node_modules/node_modules/pg
• /project/node_modules/pg (found in here)
• /node_modules/pg

If we move .store into, say, home/me/.pnpm/store, this lookup path now becomes:

• /home/me/.pnpm/store/[email protected]/node_modules/pg
• /home/me/.pnpm/store/node_modules/pg
• /home/me/.pnpm/node_modules/pg
• /home/me/node_modules/pg
• /home/node_modules/pg
• /node_modules/pg

As you can see, the project (/project) is no longer in the search path. Bummer.

This only affects packages that are meant to be used together, but are not included linked as dependencies. This is a very common pattern:

• babel-core and babel-preset-es2015
• eslint and eslint-plugin-react
• browserify and babelify
• webpack and style-loader
• and so on...

So in short, a global store turns out not to be feasible. Any thoughts and discussion on possible workarounds would be appreciated.

from pnpm.

what about the path to the store being ~/.pnpm/store/node_modules?

from pnpm.

In that case, require('lodash') won't work then--it needs to be
require('[email protected]'). It'd also be weird to be able to require packages
not in your project.

On Saturday, January 30, 2016, Chris McVittie [email protected]
wrote:

what about the path to the store being ~/.pnpm/store/node_modules?

—
Reply to this email directly or view it on GitHub
#19 (comment).

from pnpm.

Considering it seems impossible, I'm closing this for now.

from pnpm.

What about using npm's native linking?

If pg and pg-then are both installed into home/.store and
npm link ../pg is run inside the pg-then-directory, then pg becomes accessible.

The only annoying downside is the pollution of the global installation space but maybe there is a way to customise away that global directory with some prefix magic?

from pnpm.

Good try, but that's not entirely workable. For instance, if you want to store babel globally and have it used on your projects, it won't be able to see your projects's babel-preset-es2015 package because that resides in a different node_modules path.

from pnpm.

I don't quite understand. If I have a directory babel-core with exact copy of the original package and another one for babel-preset-es2015, wouldn't that npm link ... command make one module accessible from the other? It works for my local modules. Is babel-core different?

from pnpm.

Assuming your global store is in ~/.store (for simplicity's sake), you'll have these directories:

• ~/.store/babel-core
• /projects/xyz/node_modules/babel-preset-es2015
• /projects/xyz/node_modules/babel-core (symlink to ~/.store/babel-core)

Now imagine babel-core's index.js performs this call:

/* from babel-core/index.js */
require('babel-preset-es2015')

The way this resolves for modules is it will start from the canonical path of babel-core, that is, ~/.store/babel-core, and find the node_modules paths that are valid for that path. That is:

• ~/.store/babel-core/node_modules/babel-preset-es2015
• ~/.store/node_modules/babel-preset-es2015
• ~/node_modules/babel-preset-es2015
• (and so on)

Since it resolves from the canonical path of babel-core, symlinking (or npm linking, which is merely symlinking) your babel-core around your system won't let it see the node_modules in those symlinked paths.

from pnpm.

Related: npm/npm#5875

from pnpm.

I see. Thanks for the explanation!

Another idea:

Instead of direct symlinks, create "shell wrappers" that would change the context for the resolver. So in the local user directory I have the shell wrappers:

/projects/xyz/node_modules/babel-preset-es2015
/projects/xyz/node_modules/babel-core

So now I run locally babel-core, which is just the shell that in turn runs the actual symlink to the global babel-core but with additional context to tell the Node resolver to look under the local /node_modules. This way it would go again through the local shell-symlink-chain and would do the trick.

I understand that changing resolver context like that is possible in Webpack, so perhaps also in Node?

from pnpm.

I'd be very interested to see if it's possible in Node.js :)

On Monday, April 25, 2016, Dmitri Zaitsev [email protected] wrote:

I see. Thanks for the explanation!

Another idea:

Instead of direct symlinks, create "shell wrappers" that would change the
context for the resolver. So in the local user directory I have the shell
wrappers:

/projects/xyz/node_modules/babel-preset-es2015
/projects/xyz/node_modules/babel-core

So now I run locally babel-core, which is just the shell that in turn
runs the actual symlink to the global babel-core but with additional
context to tell the Node resolver to look under the local /node_modules.
This way it would go again through the local shell-symlink-chain and would
do the trick.

I understand that changing resolver context like that is possible in
Webpack, so perhaps also in Node?

—
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub
#19 (comment)

from pnpm.

@dmitriz there will be problems with including inner files. Probably most common example would be require('react-dom/server') that will not be resolved properly.

from pnpm.

The issue you mentioned earlier is getting a fix on node 6 nodejs/node#5950

from pnpm.

@Meroje I tried that new --preserve-symlinks option in node v6.3.0. I've changed the shebang of eslint in a shared pnpm store to #!/usr/bin/node --preserve-symlinks. However, the deps were being resolved to the original location as before. Maybe it works only when executed directly via the node command?

from pnpm.

I've managed to make it work on node v6.3.0!!! 😎

Steps (on Linux):

1. create a monorepo with subpackages

2. create a shared store for the subpackages

3. install eslint and some plugins to one of the packages

4. replace the eslint symlink in /node_modules/.bin with a real file that has this content:

   #!/usr/bin/node --preserve-symlinks
   
   require('../eslint/bin/eslint')

5. make the file an executable with this command chmod 755 node_modules/.bin/eslint

After these steps eslint finds its plugins without any problems even though the storage is in a different place!

from pnpm.

@zkochan I am having an issue the preserve-symlink proxy file.

When using create-react-app:

Error: Cannot find module '../react-scripts/bin/react-scripts.js'

node_modules/react-scripts/bin/react-scripts.js

#!/bin/sh
":" //# comment; exec /usr/bin/env node --preserve-symlinks "$0" "$@"
require('../react-scripts/bin/react-scripts.js')

It should be: https://github.com/facebookincubator/create-react-app/blob/master/packages/react-scripts/bin/react-scripts.js

This looks like a bug. The bin dir shouldn't be touched, only the top-level .bin.

from pnpm.

@vjpr please create an issue for this because now I can't look into it and I will most probably forget about this message 😄

from pnpm.

@rstacruz @zkochan
Hi guys, I'm using Node v10.9.0 and pnpm 2.13.5 on mac OS X 10.11, and I try to install create-react-app by pnpm install -g create-react-app, then when I run which create-react-app I still get nothing.
Any suggestions for me?

from pnpm.