Giter Club home page Giter Club logo

lazy-build's Introduction

lazy-build

Build Status Coverage Status PyPI version

Cache build artifacts based on files on disk.

Interface

Building

$ lazy-build build \
    context= requirements*.txt setup.py /etc/lsb_release \
    output= venv \
    after-download= venv/bin/python -m virtualenv_tools --update-path {pwd}/venv venv \
    command= bash -euxc 'virtualenv -ppython3 venv && venv/bin/pip install -r requirements.txt'

Invalidating artifacts

lazy-build invalidate takes all of the same arguments as lazy-build build. It builds up the context, then deletes the artifact matching it.

$ lazy-build invalidate \
    context= requirements*.txt setup.py /etc/lsb_release \
    output= venv \
    after-download= venv/bin/python -m virtualenv_tools --update-path {pwd}/venv venv \
    command= bash -euxc 'virtualenv -ppython3 venv && venv/bin/pip install -r requirements.txt'

Ideally it would have an option to mark that the artifact should not be used/re-uploaded again, in cases where something might be broken with the generated artifacts. This could be used as an off-switch until fixes can be made.

The config file

Located as .lazy-build.json at the root of your project.

{
    "cache": {
        "source": "s3",
        "bucket": "my-cool-bucket",
        "path": "cache/my-cool-service"
    },
    "ignore": [
        "*.py[co]",
        "*~",
        ".*.sw*",
        ".DS_Store",
        ".nfs*"
    ]
}

Contributing

To start, run make minimal to set up a development virtualenv, then activate it with . venv/bin/activate. This will put a lazy-build executable on your path using the local version of the code.

There's an example project inside testing/example-project which contains a simple build script and Makefile. It uses a filesystem cache under a cache directory which you can easily inspect and clear during your testing. To attempt a build, run make thing. You should observe that the first run isn't cached, but the second is.

lazy-build's People

Contributors

chriskuehl avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar

lazy-build's Issues

Have a story for corrupted multipart downloads from S3

Ideally we would never replace artifacts and always use new names. One option could be to include a unique identifier and then list keys in the bucket instead, but the S3 consistency model doesn't guarantee that keys can be listed immediately after upload. (Maybe that's okay?)

Support for invalidate command

I'd also like a "persistent invalidate" which prevents artifacts from being uploaded. This could be used as a temporary off switch in cases where the produced artifacts are broken for consumers (e.g. if the post-download script is broken, or the context is missing something important).

Upload some metadata along with the artifact

It'd be cool to see things like:

  • What user built this artifact
  • On what host (and details about the host)
  • At what time
  • How long it took to build

One concern: race during build could result in metadata not matching the artifact.

cli errors when .lazy-build.json doesn't exist

(venv) ckuehl@raziel:~/proj/lazy-build$ lazy-build --help
Traceback (most recent call last):
  File "/home/ckuehl/proj/lazy-build/venv/bin/lazy-build", line 11, in <module>
    load_entry_point('lazy-build', 'console_scripts', 'lazy-build')()
  File "/home/ckuehl/proj/lazy-build/lazy_build/main.py", line 83, in main
    conf = config.Config.from_args(argv or sys.argv[1:])
  File "/home/ckuehl/proj/lazy-build/lazy_build/config.py", line 132, in from_args
    conf = read_config()
  File "/home/ckuehl/proj/lazy-build/lazy_build/config.py", line 10, in read_config
    with open('.lazy-build.json') as f:
FileNotFoundError: [Errno 2] No such file or directory: '.lazy-build.json'

Be defensive to avoid breaking builds

It's important that lazy-build try hard never to make things worse. Some things I think are essential before I'd be comfortable using it in a large project:

  • It needs to handle errors from S3 gracefully by falling back to the regular command (authentication errors, network errors, etc.)
  • It needs to time out if downloading or uploading from S3 is taking too long and fall back to the regular command

Ideally it also needs a way to be temporarily disabled without a code push (sort of covered by #9). That could either be some metadata in S3, or maybe a file elsewhere on disk (the "srv-configs" model?).

Anything else?

cli needs tests

Then we can turn on --fail-under 100 for coverage.

I was holding off on this until I was happy with that module since the CLI tests are relatively low value and it was changing a lot. I think it's in a good shape now, though, and worth testing.

gzip before upload

In some common cases (e.g. node_modules), this reduces ~300MB artifacts to ~100MB artifacts. Probably worth the CPU time.

Support for external facts (e.g. architecture, os version)

Not quite sure how exactly to handle this. We could support a small number in lazy-build, but that's not super flexible. (Might be okay, though, since we downstream users could just create their own files containing their "facts" and include it in the context?)

How to handle existing outputs when a rebuild is needed?

Imagine my artifact is a virtualenv. I'm on a commit on master, but several versions behind. My venv is up-to-date.

I pull master, then run make venv to update it. It could either:

  1. Delete my entire venv and grab the new artifact from S3.
  2. Bypass lazy-build and do the normal action (e.g. using venv-update to uninstall/install packages as needed).

Which do we do? Both are significantly faster than the other in at least some cases.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.