Giter Club home page Giter Club logo

mono_repo.dart's Introduction

Manage multiple Dart packages within a single repository.

pub package package publisher Dart CI Coverage Status

Installation

> dart pub global activate mono_repo

Running

> dart pub global run mono_repo

Or, once you've setup your PATH:

> mono_repo

Prints the following help message:

Manage multiple packages in one source repository.

Usage: mono_repo <command> [arguments]

Global options:
-h, --help              Print this usage information.
    --version           Prints the version of mono_repo.
    --[no-]recursive    Whether to recursively walk sub-directories looking for packages.
                        (defaults to on)
    --verbose           Show full stack trace on error. (Useful for debugging.)

Available commands:
  check       Check the state of the repository.
  dart        Runs the `dart` command with the provided arguments across all packages.
  generate    Generates the CI configuration for child packages.
  list        List all packages configured for mono_repo.
  presubmit   Run the CI presubmits locally.
  pub         Runs the `pub` command with the provided arguments across all packages.
  readme      Generate a markdown table of all packages configured for mono_repo.

Run "mono_repo help <command>" for more information about a command.

Configuration

Repo level configuration

To start, you should create a mono_repo.yaml file at the root of your repo.

This controls repo wide configuration.

One option you likely want to configure is which CI providers you want to generate config for. github can be configured by adding a corresponding entry.

You probably also want to enable the self_validate option, which will add a job to ensure that your configuration is up to date.

So, an example config might look like this:

# Enabled GitHub actions - https://docs.github.com/actions
# If you have no configuration, you can set the value to `true` or just leave it
# empty.
github:
  # Specify the `on` key to configure triggering events.
  # See https://docs.github.com/actions/reference/workflow-syntax-for-github-actions#on
  # The default values is
  # on:
  #   push:
  #     branches:
  #       - main
  #       - master
  #   pull_request:

  # Setting just `cron` is a shortcut to keep the defaults for `push` and
  # `pull_request` while adding a single `schedule` entry.
  # `on` and `cron` cannot both be set.
  cron: '0 0 * * 0' # “At 00:00 (UTC) on Sunday.”

  # Specify additional environment variables accessible to all jobs
  env:
    FOO: BAR

  # You can group stages into individual workflows
  #
  # Any stages that are omitted here are put in a default workflow
  # named `dart.yml`.
  workflows:
    # The key here is the name of the file - .github/workflows/lint.yml
    lint:
      # This populates `name` in the workflow
      name: Dart Lint CI
      # These are the stages that are populated in the workflow file
      stages:
      - analyze

  # You can add custom github actions configurations to run after completion
  # of all other jobs here. This accepts normal github job config except that
  # the `needs` config is filled in for you, and you aren't allowed to pass it.
  on_completion:
    # Example job that pings a web hook url stored in a github secret with a
    # json payload linking to the failed run.
    - name: "Notify failure"
      runs-on: ubuntu-latest
      # By default this job will only run if all dependent jobs are successful,
      # but we want to run in the failure case for this purpose.
      if: failure()
      steps:
        - run: >
            curl -H "Content-Type: application/json" -X POST -d \
              "{'text':'Build failed! ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}'}" \
              "${CHAT_WEBHOOK_URL}"
          env:
            CHAT_WEBHOOK_URL: ${{ secrets.CHAT_WEBHOOK_URL }}

  # You can customize stage ordering as well as make certain stages be
  # conditional here, this is supported for all CI providers. The `if`
  # condition should use the appropriate syntax for the provider it is being
  # configured for.
  stages:
    - name: cron
      # Only run this stage for scheduled cron jobs
      if: github.event_name == 'schedule'

# Adds a job that runs `mono_repo generate --validate` to check that everything
# is up to date. You can specify the value as just `true` or give a `stage`
# you'd like this job to run in.
self_validate: analyze

# Use this key to merge stages across packages to create fewer jobs
merge_stages:
- analyze

# When using `test_with_coverage`, this setting configures the service that
# results are uploaded to.
# Note: you can configure both options, but this would be unusual.
# Note: you can configure this key with no values, to just generate the files
#   locally. This may be to enable other, custom processing.
coverage_service:
# https://coveralls.io/ - the default
- coveralls
# https://codecov.io/ – the other option
- codecov

Adding a package config

To configure a package directory to be included it must contain a mono_pkg.yaml file (along with the normal pubspec.yaml file).

You can use an empty mono_pkg.yaml file to enable the check and pub commands.

To enable generate and presubmit, you must populate mono_pkg.yaml with details on how you'd like tests to be run.

Generating .github/dependabot.yml

By adding

github:
  dependabot: {}

To your mono_repo.yaml configuration, mono_repo will generate a .github/dependabot.yml file that updates all your packages with a default configuration.

Any further configuration can go into the map and will be merged with the updates of the packages. For example:

github:
  dependabot:
    updates:
      - package-ecosystem: "github-actions"
        directory: "/"
        schedule:
          interval: "monthly"

mono_pkg.yaml example

# Every entry must be associated with at least one SDK version – corresponding
# to the Dart SDK version or the Flutter framework version, depending on the
# type of package. It can be specified at the top-lever as a single value or
# an array. Alternatively, you can specify the SDK version(s) within each job.
sdk:
 - dev
 # Specific `pubspec` to test the lower-bound SDK defined in pubspec.yaml
 # This is only supported for Dart packages (not Flutter).
 - pubspec

stages:
  # Register two jobs to run under the `analyze` stage.
  - analyze:
    - analyze
    - format
  - unit_test:
    - test
  # Example cron stage which will only run for scheduled jobs (here we run
  # multiple OS configs for extra validation as an example).
  #
  # See the `mono_repo.yaml` example above for where this stage is specially
  # configured.
  - cron:
    - test:
      os:
        - linux
        - windows

Running mono_repo generate in the root directory generates two or more files: tool/ci.sh and a configuration file for each configured ci provider.

Look at these repositories for examples of mono_repo usage:

Mono_repo and Dependabot

Historically, package:mono_repo and Dependabot couldn't be used together - they both wanted to maintain your GitHub workflow file. We've adopted mono_repo so that you can now use both it and Dependabot in your repo.

When generating your workflow configuration (mono_repo generate) mono_repo will write out its current default action versions into the workflow file. If however it sees that the repo has a Dependabot configuration - has a file named .github/dependabot.yaml in the repo - mono_repo will instead parse the workflow file, read out the current action versions, and use those versions when re-generating the file. This lets mono_repo manage the overall structure of the file, while allowing Dependabot to independently move various action versions forward as necessary.

mono_repo.dart's People

Contributors

alorenzen avatar benvercammen avatar dependabot[bot] avatar devoncarew avatar drewroengoogle avatar himanshuranjan30 avatar jack-r-warren avatar jakemac53 avatar jonasfj avatar kevmoo avatar lukaszkolodziejczyk avatar matanlurey avatar natebosch avatar parlough avatar pq avatar sigurdm avatar srawlins avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mono_repo.dart's Issues

Support some way of grouping tasks across packages

Rationale

As you mono_repo grows in size, the number of travis jobs tends to increase dramatically. It would be nice if we had a way of grouping tasks across packages to run together in the same travis job.

For instance see a recent build on the build repo, https://travis-ci.org/dart-lang/build/builds/512125780?utm_source=github_status&utm_medium=notification. We now have 25 different analyze/format tasks, which spend the majority of their time in setup and not actually doing work. These should ideally be groupable into a single travis job (as an opt in).

Proposal

Add a group_stages (or maybe group_by?) key to the travis section of the mono_repo.yaml. This will accept a list of stage names.

We will create a single travis job (per sdk version) for each of those named stages.

This would reduce the number of travis jobs for the analyze_and_format stage in the build mono_repo to 4, from 25 (we test on 4 different sdks right now).

Add a way of installing chrome

I had to hack the produced travis.yml file to install chrome for the build package, it would be good if we could enable this generally somehow... ideally with static config so you don't have to remember to pass the command line option each time.

Allow stages to accept a list of strings instead of a maps

Right now the stages section of the top level mono_repo.yaml file is required to be a map where you have to provide the keys name and if.

I had to write:

travis:
  stages:
    - name: analyze_and_format
      if: 'true'
    - name: unit_test
      if: 'true'
    - name: build
      if: 'true'

When all I want is:

travis:
  stages:
    - analyze_and_format
    - unit_test
    - build

`mono_repo check` doesn't support git paths

We added a dependency_override that contained a reference to github:

build:
  git:
    url: https://github.com/dart-lang/build.git
    path: build

running mono_repo check had the following error:

Unhandled exception:
Invalid argument (data): No clue how to deal with `{git: {url: https://github.com/dart-lang/build.git, path: build_runner}}`.: Instance of 'YamlMap'
#0      new DependencyData (package:mono_repo/src/pubspec.dart:41)
#1      new Dependency.parse (package:mono_repo/src/pubspec.dart:17)
#2      Pubspec._getDependencies.<anonymous closure> (package:mono_repo/src/pubspec.dart:109)
#3      YamlNode&MapMixin.forEach (dart:collection/maps.dart:52)
#4      Pubspec._getDependencies (package:mono_repo/src/pubspec.dart:108)
#5      new Pubspec.fromJson (package:mono_repo/src/pubspec.dart:99)
#6      check.<anonymous closure> (package:mono_repo/src/check_command.dart:38)
#7      _HashVMBase&MapMixin&&_LinkedHashMapMixin.forEach (dart:collection-patch/dart:collection/compact_hash.dart:360)
#8      check (package:mono_repo/src/check_command.dart:33)
<asynchronous suspension>
#9      CheckCommand.run (package:mono_repo/src/check_command.dart:26)
#10     CommandRunner.runCommand (package:args/command_runner.dart:194)
<asynchronous suspension>
#11     CommandRunner.run.<anonymous closure> (package:args/command_runner.dart:109)
#12     new Future.sync (dart:async/future.dart:222)
#13     CommandRunner.run (package:args/command_runner.dart:109)
#14     main (file:///Users/alorenzen/.pub-cache/hosted/pub.dartlang.org/mono_repo-0.1.0/bin/mono_repo.dart:19)
<asynchronous suspension>
#15     _startIsolate.<anonymous closure> (dart:isolate-patch/dart:isolate/isolate_patch.dart:275)
#16     _RawReceivePortImpl._handleMessage (dart:isolate-patch/dart:isolate/isolate_patch.dart:163)

Add ability to publish all packages

We should have a command for publishing all of the packages, based on a specified order.

Ideally, it would also remove the dependency overrides while publishing.

Support version constraints

What I want is to set the version for a specific package in the base repo. If the version changes all submodules using the this package should update their version accordingly. Should work for "path" also.

A way to retain cache:

I need this today:

cache:
  directories:
    - $HOME/.pub-cache
    # MANUALLY ADDED. Do not remove.
    - _benchmarks/.dart_tool
    - _goldens/.dart_tool
    - angular_router/.dart_tool
    - angular_test/.dart_tool
    # END MANUAL EDITS.

Override commands from the root mono_repo.yaml

Add the ability to override a command on a specific subfolder in the mono_repo.yaml

Potentially we'd have

// mono_repo.yaml
pkg_a:
  published: true
pkg_b:
  published: true
  packages_get: flutter pakages get
  packages_update: flutter packages upgrade

Enable custom ordering of tasks on travis

Travis appears to schedule tasks in roughly the order they appear in the matrix - it would be nice if we could change the ordering of the things in the env section so that we can start the slower tasks first.

One way this could be done is by allowing users to specify an ordering by task name and/or package name?

Add a custom task for checking for changelog updates

I think mono_repo would need a custom solution for this because of the nested packages etc, for any PR build it should check what packages had changes, and fail if the corresponding CHANGELOG was not updated.

It should also be overridable - probably with a custom label on the PR or specific message in the description.

Add a `presubmit` command

We already have a tool/travis.sh script, it just needs to be invoked with all the configurations of the PKG and TASK environment variables.

adding test args to a sub-repo adds duplicate task names

For instance if you have:

dart_task:
  # Run the presubmit tests as well (usually skipped).
  - test: --run-skipped

then in the tool/travis.sh you will get:

  pub run test --run-skipped

but it will do that for all packages, not just the one package which had it added

Sync common files

When working with multiple packages its a pain to keep files in sync. For example you often want the same analysis_options.yaml across packages. It would be great to have a consistent way of doing this.

stages in mono_repo

dart:
  - dev
  - stable

stages:
  - sample1: "valid dart task string name"
  - sample2:
    - "valid dart task string name"
    - script: "one line script invocation"
    - name: foo
      script:
        - "script1"
        - "script2"
    - dartanalyzer: --hints-as-warnings .
    - name: analyze_fun
      dartanalyzer:
    - name: analyze_fun2
      dartanalyzer: --hint-as-warnings .
    - name: analyze_fun3
      dartanalyzer:
  - analyze # ERROR! must have at least one action!
  - test #ERROR! not allowed!
  - sample1 #ERROR! duplicate!
  - first_in_another_.monorepo.yml #ERROR! ordering cannot contradict across...things

exclude:
  - dart: stable
    jobs:
      - foo
      - bar # ERROR! doesn't exist as a name above!

Add ability to crosscheck "friend" packages

I realize this is not directly related to being a "mono repo", but I'm more and more seeing this package becoming something that is just useful for developing packages, not only mono repositories (we need a cooler name, too).

I'd love to spot-check packages that rely on mine before I publish. Maybe as easy as a mechanism for either locally (or on travis) running the tests/presubmit for another package with the versions overridden to the master of my package.

Add ability to customize test command for presubmit

For angular, we need to have custom commands for running tests (as opposed to just pub run test). Right now, there is no way to specify this in .travis.yml files.

One alternative is to do what we're doing now, which is to have custom tool/travis.sh for sub-packages.

Allows population of dependency_overrides

In many mono_repo setups we have a sibling packages that depends on each others.

It would be neat there was a command to generate a dependency_overrides for dependencies that are siblings packages.

I'm not sure how this would work, we'll probably need a command to add/remove the dependency overwrites, unless pub ignores them when publishing packages.

presubmit should check the version of the script or fail more gracefully

I just confused myself when I ran mono_repo presubmit and got the same error printed to the file for all actions "PKG environment variable must be set!".

I finally just ran mono_repo travis to regenerate the script and this fixed everything. Maybe the command should detect a version mismatch with the script and do something smarter.

Support packages using flutter SDK

we have a mono_repo with both flutter, web, and pure dart packages. when we run "mono_repo check" we get this error:

mono_repo check
Unhandled exception:
Invalid argument (data): No clue how to deal with `{sdk: flutter}`.: Instance of 'YamlMap'

A way to retain sudo:/addons:

I need this today:

# MANUALLY ADDED. Do not remove.
# Gives more resources on Travis (8GB Ram, 2 CPUs).
# Do not remove without verifying w/ Travis.
sudo: required
addons:
  chrome: stable
# END MANUAL EDITS.

Travis command should include cache config

In angular, we declare a few cache directories in our travis config. We should be able to define these in child configs and copy them to the main one as part of the mono_repo travis command.

Enable combining certain tasks on travis

It would be nice if we could specify certain tasks that should be combined into a single travis job. This would be useful for tasks that are dominated by startup time like dartfmt, and would help speed up presubmits.

Add proper documentation

Prerequisites:

  1. Install mono_repo, according to README.md

    > pub global activate mono_repo
    
  2. git repo with folder structure

    drwxr-xr-x  11 eugene.kot  admin   374 May 28 18:30 .
    drwxr-xr-x   9 eugene.kot  admin   306 May 25 01:11 ..
    drwxr-xr-x  15 eugene.kot  admin   510 May 28 18:54 .git
    -rw-r--r--   1 eugene.kot  admin    17 May 28 17:42 .gitignore
    drwxr-xr-x   9 eugene.kot  admin   306 May 28 18:49 .idea
    -rw-r--r--   1 eugene.kot  admin  1067 May 22 00:29 LICENSE
    -rw-r--r--   1 eugene.kot  admin  2134 May 28 17:42 README.md
    drwxr-xr-x  16 eugene.kot  admin   544 May 28 17:42 mobile
    drwxr-xr-x  11 eugene.kot  admin   374 May 28 17:42 models
    drwxr-xr-x  11 eugene.kot  admin   374 May 28 17:42 server
    drwxr-xr-x  12 eugene.kot  admin   408 May 28 18:50 web
    

Subfolders are dart packages. And it's really annoying to run pub get in all of them. I thought that this tool can help me with that, so i ran

$ pub run mono_repo pub get

> Could not find a file named "pubspec.yaml" in "/Users/eugene.kot/reps/test".

Okay, maybe i should init it first? 🤔

$ pub run mono_repo pub init

> Could not find a file named "pubspec.yaml" in "/Users/eugene.kot/reps/test".

Maybe try inside of any repo

$ cd web
$ pub run mono_repo init
> Found `pubspec.yaml` in the current directory. Not supported.

So how to use it? And for what?
Thank you!

Smartly run tasks for only updated packages (for pull requests only)

The logic would be essentially this:

  • at the start of each task, figure out which packages are different by comparing the the branch that the pull request wants to merge into
  • for the current package being ran, skip the task if both of these hold true:
    • no files changed in the package
    • it does not have a path dependency on any changed package

Undefined class Future in lib/src/runner.dart

The file is missing the import 'dart:async'; directive.

This means that pub global activate mono_repo currently exits with

Failed to precompile mono_repo:mono_repo:
file:///Users/philippschiffmann/.pub-cache/hosted/pub.dartlang.org/mono_repo-1.2.0/lib/src/runner.dart:36:3: Error: 'Future' expects 0 type arguments.
  Future<Null> runCommand(ArgResults topLevelResults) async {
  ^
file:///Users/philippschiffmann/.pub-cache/hosted/pub.dartlang.org/mono_repo-1.2.0/lib/src/runner.dart:36:16: Error: The return type of the method 'MonoRepoRunner::runCommand' is invalid-type, which does not match the return type of the overridden method (dart.async::Future<dart.core::Null>).
Change to a subtype of dart.async::Future<dart.core::Null>.
  Future<Null> runCommand(ArgResults topLevelResults) async {
               ^

on OS X with Dart Version 2.0.0.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.