Giter Club home page Giter Club logo

arsenylab_dj_cloud's Introduction

ArsenyFinkelsteinLab DataJoint Workflow

ArsenyFinkelsteinLab DataJoint workflow - cloud deployment

Workflow operation

This workflow is fully containerized using Docker and is currently setup for automatic deployment to AWS EC2 instance that is already provisioned. The credentials to access this EC2 instance is set as GitHub secrets (see GitHub Actions secrets for details)

Once the required credentials are added, initialization of this workflow is done by manually trigger a predefined GitHub Actions named start_process (see code here) Your GitHub username needs to be specified in the allow-list for you to be able to trigger the GitHub Actions. If you are not in the list yet, make changes to the list and issue a PR for Arseny's approval.

How to start a new round of workflow operation

  1. From any browser, navigate to the GitHub repository for this workflow - make sure to log in to your GitHub account
  2. Click on the Actions tab, in which you will see the list of available "GitHub workflows" on the left pannel. There is only one for this repo, named start_process
  3. Click on the GitHub workflow start_process, then on right side, from the dropdown menu "Run Workflow", click the green button "Run Workflow" to trigger one round of operation

What is the expected behavior?

Once a new round of operation is triggered with the GitHub Actions above, the pre-provisioned EC2 instance will start, and then the following steps are performed

  1. Clone this repository - this is recloned everytime the flow is triggered, ensuring the latest version of the code is used
  2. Rebuild the Docker image - again, docker image is rebuilt everytime, ensuring latest version of the workflow is deployed
  3. Lauch the Docker container with the newly built image - this will then run all of the DataJoint computation, essentially all .populate() calls defined here
  4. Once finished (successful or not), the Docker container will terminate itself, thus trigger the EC2 instance to shutdown

Installation and deployment instruction

For installation and deployment on any other resources (e.g. local or other cloud resources), see local deployment

arsenylab_dj_cloud's People

Contributors

lsusman avatar yambottle avatar arsenyf avatar ttngu207 avatar amukovozov avatar dimitri-yatsenko avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.