wix-incubator / octopus Goto Github PK

View Code? Open in Web Editor NEW

13.0 232.0 4.0 250 KB

build tools for multimodule npm repos

Shell 0.17% JavaScript 99.83%

octopus's Introduction

octopus

Core modules
Presets
Example usage

Octopus is a set of modules for start but to manage a multi-module npm project with traits:

tasks to recursively discover/load npm modules;
have means for incremental task running - run tests for all, change 1 module, run tests for just one changed module and its dependees;
presets for managing versions across multiple modules, sync module versions;
generic tasks that you compose and achieve basically anything.

Core modules

octopus-modules - exports functions for finding modules, filtering, etc.
octopus-start-modules - npm module management: syncing versions, listing, showing where one of modules is used in repo;
octopus-start-modules-tasks - start tasks around modules and extras;
octopus-start-reporter - custom start reporter to be used within octopus ecosystem;
octopus-start-tasks - misc useful start tasks;
octopus-test-utils - helpers for testing multi-module npm project scenarios.
octopus-start-git - miscelneous tasks for git-based operations.

Presets

octopus-start-preset-prepush - start task that allows to add git prepush hook with custom logic composable from start tasks.
octopus-start-preset-dependencies - manage dependencies across multiple modules;
octopus-start-preset-modules - manage module versions across monorepo;
octopus-start-preset-idea - generate multi-module IntelliJ Idea project with run-tasks for mocha, etc.;
octopus-start-preset-depcheck - preset for running depcheck across multiple modules.
octopus-start-npm-tasks - tasks for all-npm things.

Example usage

Octopus does literally eats it's own dogfood and octopus repo is managed with start task-runner and octopus tasks including:

using git prepush hook to:
- unify versions of dependencies across all modules via sync present;
- unify versions of modules across all modules via sync present;
- check for extraneous managed dependencies via extraneous preset;
- check for unmanaged dependencies via unmanaged preset;
- enrich readme via markdown-magic;
incrementally npm install and link modules in repo (see module.exports.bootstrap in tasks.js below);
incrementally run tests for modules in repo (see module.exports.test in tasks.js below);
clean all modules in repo (see module.exports.clean in tasks.js below);
release/publish new versions of changed modules (see module.exports.release in tasks.js below);
add custom tasks for what you need (ee module.exports.docs in tasks below).

Sample tasks.js file as prescribed by start:

const Start = require('start').default,
  reporter = require('octopus-start-reporter'),
  modules = require('octopus-start-preset-modules'),
  dependencies = require('octopus-start-preset-dependencies'),
  startTasks = require('octopus-start-tasks'),
  startModulesTasks = require('octopus-start-modules-tasks'),
  prepush = require('octopus-start-preset-prepush'),
  idea = require('octopus-start-preset-idea'),
  depcheck = require('octopus-start-preset-depcheck'),
  markdownMagic = require('markdown-magic');

const start = Start(reporter());

module.exports['modules:list'] = () => start(modules.list());
module.exports['modules:where'] = moduleName => start(modules.where(moduleName));
module.exports['modules:sync'] = () => start(modules.sync());
module.exports['deps:sync'] = () => start(dependencies.sync());
module.exports['deps:unmanaged'] = () => start(dependencies.unmanaged());
module.exports['deps:extraneous'] = () => start(dependencies.extraneous());
module.exports['deps:latest'] = () => start(dependencies.latest());
module.exports['idea'] = () => start(idea());
module.exports['init'] = () => start(prepush());
module.exports['depcheck'] = () => start(depcheck({ignoreMatches: ['start-simple-cli']}));

module.exports.sync = () => start(
  modules.sync(),
  dependencies.sync(),
  dependencies.unmanaged(),
  dependencies.extraneous(),
  module.exports.docs
)

module.exports.bootstrap = () => start(
  startModulesTasks.modules.load(),
  startModulesTasks.modules.removeUnchanged('bootstrap'),
  startModulesTasks.iter.async()((module, input, asyncReporter) => Start(asyncReporter)(
    startTasks.ifTrue(module.dependencies.length > 0)(() =>
      Start(asyncReporter)(startModulesTasks.module.exec(module)(`npm link ${module.dependencies.map(item => item.name).join(' ')}`))
    ),
    startModulesTasks.module.exec(module)('npm install --cache-min 3600 && npm link'),
    startModulesTasks.module.markBuilt(module, 'bootstrap')
  ))
)

module.exports.test = () => start(
  startModulesTasks.modules.load(),
  startModulesTasks.modules.removeUnchanged('test'),
  startModulesTasks.iter.async()((module, input, asyncReporter) => Start(asyncReporter)(
    startModulesTasks.module.exec(module)('npm run test'),
    startModulesTasks.module.markBuilt(module, 'test')
  ))
)

module.exports.clean = () => start(
  startModulesTasks.modules.load(),
  startModulesTasks.iter.async()((module, input, asyncReporter) => Start(asyncReporter)(
    startModulesTasks.module.exec(module)('rm -rf node_modules && rm -rf target && rm -f npm-shrinkwarp.json && rm -f yarn.lock && rm -f npm-debug.log*')
    )
  )
)

module.exports.release = () => start(
  startModulesTasks.modules.load(),
  startModulesTasks.iter.async()((module, input, asyncReporter) => Start(asyncReporter)(
    startModulesTasks.module.exec(module)('npm run release'),
    startModulesTasks.module.exec(module)('npm publish || true')
  ))
)

module.exports.docs = () => start(() => {
  return function generateDocs(log /*, reporter*/) {
    return Promise.resolve().then(() => {
      markdownMagic('./*.md');
    })
  }
});

And package.json that contains needed dependencies and npm scripts, of which:

postinstall - installs git prepush hook to execute sync export before pushing to remote;
start - to be able to execute any tasks defined in tasks.js like:

npm start bootstrap

That's it - for available tasks explore repo, see examples. Or open issues/pull requests. Enjoy!

octopus's People

Contributors

Stargazers

Watchers

Forkers

laurynaslubys mnjstwins hugebdu darwantep

octopus's Issues

cli: fails strangely when running in non-octo repo

$ octo idea

Usage: octo idea

Options:
  -h, --help     Show help                                             [boolean]
  -V, --version  Show version number                                   [boolean]

Cannot read property 'error' of undefined

depcheck: use iterable reporter

start-preset-dependencies: add unmanaged task

That would display dependencies defined in modules but not in managedDependencies

preserve original shell colors

now shelljs does not do that - maybe fallback to native exec?

Ignore yarn.lock file during bootstrap

yarn install --no-lockfile

cli: add support for package.json in root of git repo

So that:

you could have package.json with octopus-cli dependency and define build scripts for common tasks (bootstrap, test, ...);

modules: do not traverse into target folder

as target is currently considered as tmp and there might be some leftover modules from tests

add run-script command?

So that you could define scripts in octopus.json and execute those. I would use for chores like checking if there are no unused deps, etc.

start-preset-dependencies: add extraneous task

That would display dependencies defined in managedDependencies but not in any of modules

cli run: make npm scripts optional

Now it mandates to have scripts or else fails. Maybe make it optional?

add support to build tree for only changed modules

Now given you do clean checkout of x branch and do octo bootstrap or octo run test it builds all tree all modules. Idea is to filter graph and instruct it to build only modules and dependees where there were (are) changes when compared to master. This would give quite a benefit for verifying pull requests as feedback time would decrease alot in many cases.

It could be controlled via flag -g or smth like that.

Allow to run the npm client with extra arguments

Specifically, we would like to run yarn install without using the lockfile.

fix error feedback

Now on failure there is no useful input (swallows output) whereas it should print stdout/stderr.

octopus-start-modules: async-tasks: wrong numbering of modules

if you have 20 modules, and run them in async, then task logs module-name (1/20). Now it does repeat some numbers, miss some others.

octo-release: add

Ok, so octopus-cli supports engines: yarn, npm. wnpm-release does not. Maybe wnpm-release should be enhanced to support engines and then octo-release would just use wnpm-release.

How it would look like:

octo-release --help

  - engine: yarn, npm, autodetect, where autodetect detects engine from octopus.json. autodetect is on by default.

test-utils: add a way to generate fs-less modules

For testing tasks that do not really require filesystem.

do not dump output if -v flag is provided for commands

As you get verbose + dump output again on error.

parallel bootstrap

This one is tricky, as you have to bootstrap in respect to dependency graph, but oh well.

Working code for inspiration is: https://github.com/laurynasl-wix/wnpm/commit/444cac4d7d8542fcae5a22c7bd94a940f1ffd845 + following commits for fixes.

Execution should just be properly tdd'ed of course:)

does not print help if command does not exist

add option to unbuild one module

octopus-start-modules-tasks: async-task: does not run at all

given we do a partial run:

  startModulesTasks.modules.load(),
  startModulesTasks.modules.removeUnchanged(),

it fails to run, as it does not find modules that can run (as there are unexecuted dependencies in current run, though those are actually built).

Need to filter-out dependencies that are already built

add -r flag for run

to retry. This would be useful for tests

add support for pre-* scripts in octopus.json

It's useful when you want to customize some actions.

octopus.json
js

{
  "scripts": {
    "pre-bootstrap": "rm -f yarn.lock"
  }
}

Add support for bootstrap, run, exec

add `all` as type of task

now we have forEach and async that respect dependency graph. In some cases we just need to do mass changes disrespecting dependency grap - 'rm -rf node_modules'. Add all task that resembles Promise.all.

add support for precache

now we have some support for -p (parallel), but it mostly fails for:

npm due to existing links;
yarn - as it cannot download/add to cache in parallel.

what worked for me well is when I:

add all dependencies to managed;
use same versions;
do install on those (yarn downloads and caches them);
then do bootstrap -cap 24.

It's still not bulletproof, but it worked nicely and I did cut bootstrap time to < 10 mins for 80 module repo.

depcheck: make it incremental

do not use target folder and instead store sentinels in .octopus ?

no need to add extra folder files to module dirs.

start-preset-dependencies: add latest task

That would display latest versions of dependencies for managedDeps

do not change package.json files on sync's

as this marks whole tree as unbuilt.

outputs help on failure

octo bootstrap - add -c flag

To clear before bootstrap. use scripts: {clean: ''} from octopus.json

add octopus-start-preset-mdmagic

To run markdownmagic processor. question is should it run for all module readmes, or accept globs, or both?

add depcheck module

so that I could do smth like this in my tasks.js

const startDepcheck = require('octopus-start-depcheck');

module.exports.depCheck = startpDepcheck({ignoreMatches: ['mocha*']});
// rest of tasks

async, forEach - should suppress error logging

as it's logged by root logger.

make prepush hooks scriptable

now prepush hooks are hardcoded (octo modules sync --save && octo deps sync --save).

make them customazable via:
octopus.json

{
  "scripts": {
    "prepush": "octo deps sync --save && octo modules sync --save && echo 1"
  }
}

modules: traverse into private packages

Traverse into private packages, and if they contain other modules, ignore them - otherwise add them.

Some packages might be private just because they are not intended to be published - ex. docker apps, where published thing is docker image and not npm package, or test-apps that should run in ci, but no need to publish