Awesome work @juliangruber! I have been looking at this space recently, so thought I would brain dump a bit and see if you liked any of these thoughts. Sorry for the unstructured issue, but I didn't want to create too much noise if some of this would not move forward.

1. Sandboxing

So I see that you are using this Op platform for some form of sandboxing, but for folks looking to do this in other environments, it would be nice to abstract this away so that folks could provide their own sandbox. An example of this, a docker container would be a great simple sandbox (and allow for simple testing of my next point). If there was an abstraction layer about what an execution looks like, we could use the orchestration layer in this module but run the actual un-trusted code remotely.

2. Node versions

For node, there is CITGIM, which tests node versions against many popular packages. We talked a bit about the matrix testing we really want, which is a matrix of this module version X dependant module version X this module dependency version X node version. Obviously this is huge matrix, so I will talk about a point on this next, but having a few more knobs would be awesome to test a more detailed matrix.

3. The test matrix

As I mentioned above, the matrix can get huge. One of the things we encountered at Netflix was that the larger number of dependents and matrix variants, the larger the load and longer the tests take. To improve this we can do a few things:

1. Have a list of modules to test
2. Only pick a random selection
3. Add a learning algo to test the most likely to fail
4. Have a staged approach (depending on the change and the SDLC stage, choose a different approach)

4. CI environments

I see you have #2, so this is probably on your mind, but I think this is a very important consideration. If you take a simple OSS setup on GitHub Actions, you probably want to leverage their matrix strategy. So having some story for this type of testing in an environment like that would be awesome. Again, I think this needs to be a bit abstracted so that folks with different CI systems can also use it.

5. Multi-Registry Support/Fetching Source

Looks like the package fetch-package-source does not really have hooks for alternate source fetching. It seems like just making this an option you can inject would solve for folks needing to fetch from different sources.

I wanted to get the conversation started, and I think I missed some other points we brought up in the Package Maintenance Working Group, but hopefully this helps. Also, If you are open to it I an on board with working on some of these ideas here in this repo, so let me know if you are open to PRs.

To support use cases like unifiedjs/ideas#7

Where a project may want to canary test before releasing changes, it could be useful to be able to point to either a local folder or a git branch as a release source.

Not sure if you are familiar with npx, which is bundled with npm, but it's really good for running CLIs and similar "one-off" / one-time tools as an alternative to installing them globally. It execute just as if you installed it globally, except that it cleans up all its dependencies after it has completed.

An example using the current steps in the README would look like this:

$ npx test-npm-dependants -v

Another advantage (depending on how you look at it is) is that with a global install, users will be responsible for upgrading their global install to use newer versions. npx will always use the latest available pulled from the registry.

If a package includes test-npm-dependant as a test, there is the possibility that test-npm-dependent could call a project that would call test-npm-dependent.
This could cause exponential test growth (possibly infinite if there are circular dependencies).
It would be nice to have an option to disable test-npm-dependant in projects being tested.