Giter Club home page Giter Club logo

r2g's Introduction

Gitter Version


@oresoftware/r2g

Properly test your NPM packages before publishing.
This CLI tool allows you to easily test your package in the published format, without having to publish to an NPM registry.


Caveats + Disclaimer

This will not work with MS Windows. Only MacOS and *nix. If you are interested in getting to work on Windows, pls file a ticket.


Video demo

Watch this video to learn how to use r2g: TBD (video demo coming in the future)

The video will reference this example repo:
https://github.com/ORESoftware/r2g.example


Installation

$ npm i -g r2g

You can add the following to your ~/.bashrc and/or ~/.bash_profile files:

. "$HOME/.oresoftware/shell.sh"

=> Note you will also get bash completion for r2g, if you source the above shell script.


_____________________________________________________________________________________________

FAQ

see docs/faq.md


About The Tool

r2g tests your package after using npm pack and npm install --production. You can use your current test suite for testing, and also write some new smoke tests that are specific to r2g. r2g current has 3 phases, each phase is optional:


  • phase-Z: packs your project and installs the packed project as a dependency of itself then runs npm test on your project. You can override npm test with r2g.test in package.json.
  • phase-S: installs your project as a dependency of a dummy package in $HOME/.r2g/temp/project, then it executes the r2gSmokeTest function exported from your main.
  • phase-T: Copies the test scripts from .r2g/tests in your project, to $HOME/.r2g/temp/project/tests, and runs them.

By default all phases are run, but you can skip phases with the --skip=z,s,t option.


r2g is one of several tools that makes managing multiple locally developed NPM packages easier.

The current pieces are:

  • npm-link-up (NLU) => links multiple NPM packages together for local development
  • r2g => tests local packages properly before publishing to NPM
  • npp => publish multiple packages and sync their semver versions

Quick reference


$ r2g run  ### note: `r2g test` is an alias of `r2g run`
  • Runs the tool, and runs all phases.

$ r2g run --skip=z,s
  • This will skip phases Z and S

$ r2g run -z -s
  • This will also skip phases Z and S

$ r2g run --full
  • Installs other locally developed dependencies to your main package, defined in .r2g/config.js, and tests everything together

$ r2g run --full --pack
  • Installs other locally developed dependencies to your main package, npm packs them too, and tests everything together

$ r2g inspect
  • Copies your project to a temp folder and logs info about a temp tarball that gets created
  • AKA, you can see which contents/folders/files will get included in the tarball
  • Also warns you about any especially large files that you may have accidentally included.

$ r2g publish
  • Publish your package and ignore the .r2g folder for an even leaner tarball
  • Copies your project to a temp folder and the .r2g folder is excluded/ignored
  • Also copies symlinks so you can include symlinked files/folders easily when publishing

Important Info

  • This tool is only proven on MacOS/*nix, not tested on Windows. If you do Windows and want something to do - fork this and make it work for Windows - it won't be hard.
  • You can use r2g with zero-config, depending on what you want to do.
  • Testing does not happen in your local codebase - before anything, your codebase is copied to "$HOME/.r2g/temp/copy", and all writes happen within "$HOME/.r2g/temp".
  • If you use the --full option, the local deps of your package will copied to: "$HOME/.r2g/temp/deps"
  • You can and should put your regular tests in .npmignore, but your .r2g folder should not be in .npmignore

To make this README as clear and concise as possible:

  • Your NPM package is referred to as X. X is the name of the package you publish to NPM, which is the "name" field of your package.json.
  • The package.json file for X is simply referred to as X-package.json.
  • Your index.js file (or whatever the "main" property points to in X-package.json), is referred to as X-main


Purpose

This tool complements your standard CI/CD testing for NPM libraries. You might already be using Travis, CircleCI, etc, to test your library when you do a git push. Keep doing that. However, what you are already doing is likely to be insufficient because:

  1. You install using npm install instead of npm install --production, because you need your devDependencies for your tests. (whoops!).
  2. You are testing your package directly, instead of testing it as a dependency of another project. In reality, someone will be using your package via node_modules/X, and for example, your postinstall routine may behave differently here.
  3. You are not using npm pack to package your project before testing it. Your .npmignore file could mean you will be missing files, when someone goes to use your package in the wild. Likewise, if the "files" property in X-package.json is too passive, you might be missing files as well. Using npm pack before testing solves that.
  4. It is possible to check out a branch that has passed on a remote CI/CD platform but locally does not have the built/transpiled target files. This means the files will not make it into the tarball that gets published to NPM. Testing with r2g locally before publishing a local branch means you avoid this problem, because if the target files are not built, r2g tests will fail.

The above things are why you need to take some extra pre-cautions before publishing NPM packages. I think everyone has had an .npmignore file that accidentally ignored files we need in production. And we have all had dependencies listed in devDependencies instead of dependencies, which caused problems when people try to use the library. Those are the motivations for using this tool, to prove that X works in its final format.

  • There is a secret feature which is extremely badass - install other locally developed packages which are dependencies of X, as part of r2g testing. See "Linking with existing local dependencies" below.

How it works in detail

To learn more about how r2g works in detail, see: docs/r2g-runtime-steps.md


Basic usage / Getting started

You can use r2g with zero-config - you just need to implement a single function.


To start, execute this in a shell at the root of your project:

$ r2g run

This command will then fail. That's expected.


To get your test to pass, add this to X-main (your package's index file, whatever "main" in package.json points to):

exports.r2gSmokeTest = async () => { 
  return Promise.resolve(true);
};

the above function is called with Promise.resolve(X.r2gSmokeTest()), and in order to pass it must resolve to true (not just truthy).


To read more about the exported r2gSmokeTest function, see: docs/r2g-smoke-test-exported-main-function.md


Note: the exported function r2gSmokeTest allows you to smoke test your package. When this function is run you may use the production dependencies declared in your project. However, for other r2g tests, you may not directly use the dependencies declared in X-package.json, you may only require X itself.


Adding more tests beyond the r2gSmokeTest function

To do more sophisticated tests, we add some configuration in a folder called .r2g in the root of your project.


To do this, run:

r2g init

this will add a folder to your project called .r2g. Your .r2g folder should never be in .npmignore. (Running r2g init is safe, it will not overwrite any existing files). Your new .r2g folder contains a file called: .r2g/smoke-test.js.


Now when r2g run executes, it will run .r2g/smoke-test.js, but it will run this test in the context of the main project, meaning it will copy:


$HOME/.r2g/temp/project/node_modules/X/.r2g/smoke-test.js --> $HOME/.r2g/temp/project/smoke-test.js


The above is very important to understand, because it means that this smoke test should not include any dependencies from X-package.json. In fact, the only dependency .r2g/smoke-test.js should require, besides core modules, is X itself.


Linking with existing local dependencies:

Run r2g init and in your .r2g/config.js file, add local package names to the packages property:

exports.default = {

  // ...

  packages: {
     // local packages will be installed to your project --> test your local dependencies instead of installing from NPM
     'your-local-package-b': true,
     'your-local-package-c': true
   }
}

If you execute r2g run --full these packages (b,c) will be discovered on your local fs, and copied to "$HOME/.r2g/temp/deps/*", and then X's package.json will look like this:

{
  "dependencies": {
   "your-local-package-b": "file:///home/user/.r2g/temp/deps/your-local-package-b",
   "your-local-package-c": "file:///home/user/.r2g/temp/deps/your-local-package-c",
  }
}

If you execute r2g run --full --pack, then this awesomeness happens:

{
  "dependencies": {
   "your-local-package-b": "file:///home/user/.r2g/temp/deps/your-local-package-b.tgz",
   "your-local-package-c": "file:///home/user/.r2g/temp/deps/your-local-package-c.tgz",
  }
}

So using r2g run --full => we install local deps instead of the deps from NPM.


And using r2g run --full --pack => we pack the local deps before installing them.


Awesome.

To read more about using local deps for testing instead of installing your local deps from NPM, see:
=> docs/r2g-using-local-deps.md


Usage in a Docker image/container

Use a Docker container for a fresh/independent/isolated testing env. For packages that do more complex/system things, it will be useful to use a locally running Docker container. To use r2g in a Docker container, see: https://github.com/ORESoftware/r2g.docker


Alternatively, you can just run r2g as part of your normal CI/CD library testing on remote servers. First, make sure you have Docker installed on your local machine. See standard installation instructions for MacOS/*nix.


Run this in the root of your project:

$ r2g init --docker  # you only need to run this once per project, but won't hurt if you do it more than once

Then run this:

$ r2g docker

The above command actually uses this command line tool:
https://github.com/ORESoftware/r2g.docker


For the future:

  • Instead a dummy NPM project which will depend on X, we can allow users to use their own test projects, and pull those in with git clone or what not.

TBD

Just adding some spaces here

r2g's People

Contributors

oresoftware avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

r2g's Issues

Allow users to install X with -D flag

When users want to test their dependency X for another project Y, they may want to test X as a devDependency of Y. Right now r2g always installs it is a regular dep of Y.

use `r2g --use-latest`

new ideas:

r2g --use-latest # would install with "latest" deps being filled with the local deps

r2g --keep # already exists

add this answer to SO question

for the following SO question:

https://stackoverflow.com/questions/25124844/should-i-npmignore-my-tests

add this answer:

Don't include your tests. Oftentimes tests are like 5x the size of the actual codebase.

But what you absolutely should do is test your NPM package after removing the tests. Create some smoke tests that reside in the actual codebase, but are not part of the test suite.

You can read about testing your package after tarballing it, here:

https://github.com/ORESoftware/r2g

https://stackoverflow.com/questions/50206729/how-to-test-an-npm-publish-result-without-actually-publishing-to-npm

r2g output should be uniform

To make it pretty, go from this:

r2g: Re-creating folders "$HOME/.r2g/temp"...
r2g: Copying your user defined tests to: "/Users/alex/.r2g/temp/project" ...
r2g: Running the following command from your project copy root: npm pack --loglevel=warn;
r2g: Running the following command via this dir: "/Users/alex/.r2g/temp/project" ...
r2g: npm install --loglevel=warn --cache-min 9999999 --no-optional --production "/Users/alex/.r2g/temp/copy/linked-queue/oresoftware-linked-queue-0.1.104.tgz";
r2g:            npm i --loglevel=warn --cache-min 9999999 --no-optional --production;
r2g: Running "npm install --cache-min 9999999 --loglevel=warn" in project copy.
r2g: No custom actions registered. Use custom.actions.js to add custom actions.

to this:

r2g: Re-creating folders "$HOME/.r2g/temp"...
r2g: Copying your user defined tests to: "/Users/alex/.r2g/temp/project" ...
r2g: Running the following command from your project copy root: npm pack --loglevel=warn;
r2g: Running the following command via this dir: "/Users/alex/.r2g/temp/project" ...
r2g: npm install --loglevel=warn --cache-min 9999999 --no-optional --production "/Users/alex/.r2g/temp/copy/linked-queue/oresoftware-linked-queue-0.1.104.tgz";
r2g: npm i --loglevel=warn --cache-min 9999999 --no-optional --production;
r2g: Running "npm install --cache-min 9999999 --loglevel=warn" in project copy.
r2g: No custom actions registered. Use custom.actions.js to add custom actions.

feature: allow users to include a package.json instead of just using `npm init -f`

allow users to include a package.json instead of just using npm init -f

Users can add a package.json file:

.r2g/package.json

which will be the package.json file of Y, the testbed project.

In that package.json file, they can include test dependencies if they want.

This might not be a good idea, because it may conflate deps that won't actually exist IRL.

As requested, the "work on windows" ticket...

Feature, bug, or other?

Other

Please run the bash function and copy the output here:




#### What is the issue? How to replicate?

Following the gist of what's suggested in the Readme:  If you want r2g to work with windows, file a ticket.

Well, here's the ticket.

r2g init yields error

I am seeing this after running: r2g init in a project:

r2g: events.js:167
r2g: throw er; // Unhandled 'error' event
r2g: ^
r2g: 
r2g: Error: ENOENT: no such file or directory, open '/Users/Olegzandr/.nvm/versions/node/v10.6.0/lib/node_modules/@oresoftware/r2g/assets/contents/Dockerfile.r2g.original'
r2g: Emitted 'error' event at:
r2g: at fs.open (internal/fs/streams.js:100:12)
r2g: at FSReqWrap.oncomplete (fs.js:145:20)

"could not read your .r2g/config.js files"

Most likely because:

const path = require('path');

if (!path.isAbsolute(process.env.MY_DOCKER_R2G_SEARCH_ROOT || '')) {
  throw new Error('Please set the env var "MY_DOCKER_R2G_SEARCH_ROOT" to an absolute path.');
}

Actual error:

r2g: r2g warn: Could not read your .r2g/config.js file at path: /Users/alex/codes/ores/linked-queue/.r2g/config.js

we need to actually show the stack trace of the error.

and perhaps MY_DOCKER_R2G_SEARCH_ROOT should default to the pwd if it's above $HOME, o/w throw an error.

rm unnecessary feedback message

For users who won't need to link other local packages, this warning seems superfluous:

"You should supply some packages to link, otherwise this is somewhat pointless"

r2g --docker

r2g docker
r2g --docker

should run the damn thing in a docker container

test alpha/beta/rc packages

This is unlikely to be useful, but we could use a command line switch to use a url instead of the local project and test that with r2g. This means we are testing a tarball on NPM instead of a local folder. The thing is, now that we are no longer publishing the .r2g folder there probably just is no use for testing against something already published on NPM.

r2g init adds Docker stuff

but r2g init should not add Docker stuff

r2g docker --init should do that instead

r2g docker --init can run r2g init and then run it's thing.

r2g rename node_modules dirs

'use strict'

const fs = require('fs');
const path = require('path');
let current = path.dirname(process.cwd());
const home = process.env.HOME;

let renameThisDir = 'node_modules';
let renameTo = 'r2g_nodemodules';


if (process.argv[2] === 'undo') {
    // swap the variables
    [renameTo, renameThisDir] = [renameThisDir, renameTo];
}

while (current.length >= home.length) {
    
    let nm = path.resolve(current, renameThisDir);
    let renamed = path.resolve(current, renameTo);
    
    fs.stat(nm, (err, stats) => {
        
        if (!(stats && stats.isDirectory())) {
            return
        }
        
        fs.rename(nm, renamed, err => {
            err ? console.error(err.message) :
                console.log('renamed:', nm, 'to:', renamed)
        });
    });
    
    current = path.join(current, '..');
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.