Giter Club home page Giter Club logo

ofelia's Introduction

Ofelia - a job scheduler GitHub version Test

Ofelia is a modern and low footprint job scheduler for docker environments, built on Go. Ofelia aims to be a replacement for the old fashioned cron.

Why?

It has been a long time since cron was released, actually more than 28 years. The world has changed a lot and especially since the Docker revolution. Vixie's cron works great but it's not extensible and it's hard to debug when something goes wrong.

Many solutions are available: ready to go containerized crons, wrappers for your commands, etc. but in the end simple tasks become complex.

How?

The main feature of Ofelia is the ability to execute commands directly on Docker containers. Using Docker's API Ofelia emulates the behavior of exec, being able to run a command inside of a running container. Also you can run the command in a new container destroying it at the end of the execution.

Configuration

Jobs

Scheduling format is the same as the Go implementation of cron. E.g. @every 10s or 0 0 1 * * * (every night at 1 AM).

Note: the format starts with seconds, instead of minutes.

you can configure four different kind of jobs:

  • job-exec: this job is executed inside of a running container.
  • job-run: runs a command inside of a new container, using a specific image.
  • job-local: runs the command inside of the host running ofelia.
  • job-service-run: runs the command inside a new "run-once" service, for running inside a swarm

See Jobs reference documentation for all available parameters.

INI-style config

Run with ofelia daemon --config=/path/to/config.ini

[job-exec "job-executed-on-running-container"]
schedule = @hourly
container = my-container
command = touch /tmp/example

[job-run "job-executed-on-new-container"]
schedule = @hourly
image = ubuntu:latest
command = touch /tmp/example

[job-local "job-executed-on-current-host"]
schedule = @hourly
command = touch /tmp/example


[job-service-run "service-executed-on-new-container"]
schedule = 0,20,40 * * * *
image = ubuntu
network = swarm_network
command =  touch /tmp/example

Docker labels configurations

In order to use this type of configurations, ofelia need access to docker socket.

docker run -it --rm \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    --label ofelia.job-local.my-test-job.schedule="@every 5s" \
    --label ofelia.job-local.my-test-job.command="date" \
        mcuadros/ofelia:latest daemon --docker

Labels format: ofelia.<JOB_TYPE>.<JOB_NAME>.<JOB_PARAMETER>=<PARAMETER_VALUE>. This type of configuration supports all the capabilities provided by INI files.

Also, it is possible to configure job-exec by setting labels configurations on the target container. To do that, additional label ofelia.enabled=true need to be present on the target container.

For example, we want ofelia to execute uname -a command in the existing container called my_nginx. To do that, we need to we need to start my_nginx container with next configurations:

docker run -it --rm \
    --label ofelia.enabled=true \
    --label ofelia.job-exec.test-exec-job.schedule="@every 5s" \
    --label ofelia.job-exec.test-exec-job.command="uname -a" \
        nginx

Now if we start ofelia container with the command provided above, it will pickup 2 jobs:

  • Local - date
  • Exec - uname -a

Or with docker-compose:

version: "3"
services:
  ofelia:
    image: mcuadros/ofelia:latest
    depends_on:
      - nginx
    command: daemon --docker
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      ofelia.job-local.my-test-job.schedule: "@every 5s"
      ofelia.job-local.my-test-job.command: "date"

  nginx:
    image: nginx
    labels:
      ofelia.enabled: "true"
      ofelia.job-exec.datecron.schedule: "@every 5s"
      ofelia.job-exec.datecron.command: "uname -a"

Ofelia reads labels of all Docker containers for configuration by default. To apply on a subset of containers only, use the flag --docker-filter (or -f) similar to the filtering for docker ps. E.g. to apply to current docker compose project only using label filter:

version: "3"
services:
  ofelia:
    image: mcuadros/ofelia:latest
    depends_on:
      - nginx
    command: daemon --docker -f label=com.docker.compose.project=${COMPOSE_PROJECT_NAME}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      ofelia.job-local.my-test-job.schedule: "@every 5s"
      ofelia.job-local.my-test-job.command: "date"

  nginx:
    image: nginx
    labels:
      ofelia.enabled: "true"
      ofelia.job-exec.datecron.schedule: "@every 5s"
      ofelia.job-exec.datecron.command: "uname -a"

Logging

Ofelia comes with three different logging drivers:

  • mail to send mails
  • save to save structured execution reports to a directory
  • slack to send messages via a slack webhook

These can be configured by setting the options listed below in the [global] section of your config.ini, or via docker labels on the ofelia container (regardless of where your job will actually be running).

Options

  • smtp-host - address of the SMTP server.

  • smtp-port - port number of the SMTP server.

  • smtp-user - user name used to connect to the SMTP server.

  • smtp-password - password used to connect to the SMTP server.

  • smtp-tls-skip-verify - when true ignores certificate signed by unknown authority error.

  • email-to - mail address of the receiver of the mail.

  • email-from - mail address of the sender of the mail.

  • mail-only-on-error - only send a mail if the execution was not successful.

  • save-folder - directory in which the reports shall be written (must already exist).

  • save-only-on-error - only save a report if the execution was not successful.

  • slack-webhook - URL of the slack webhook.

  • slack-only-on-error - only send a slack message if the execution was not successful.

Overlap

Ofelia can prevent that a job is run twice in parallel (e.g. if the first execution didn't complete before a second execution was scheduled. If a job has the option no-overlap set, it will not be run concurrently.

Installation

The easiest way to deploy ofelia is using Docker. See examples above.

If don't want to run ofelia using our Docker image you can download a binary from releases page.

Why the project is named Ofelia? Ofelia is the name of the office assistant from the Spanish comic Mortadelo y Filemón

ofelia's People

Contributors

0xerr0r avatar aaron97neu avatar ashleysommer avatar dependabot[bot] avatar fohlen avatar fresus avatar gedasfx avatar gerundt avatar github-actions[bot] avatar jeluf avatar jenswbe avatar jgough avatar jogwen avatar jsixface avatar kirtangajjar avatar ksurl avatar lacek avatar magiccc avatar mcuadros avatar scls19fr avatar tanoabeleyra avatar taraspos avatar testwill avatar thatside-zaraffa avatar thornycrackers avatar toqueteos avatar varlogerr avatar vitaliytv avatar vmarkovtsev avatar wolfulus avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ofelia's Issues

How to run a local image

Suppose I built a local image like this

docker build -t "myimage:local" .

It seems I can't run such an image via Ofelia, e.g

[job-run "fetch_sources"]
schedule = @hourly
image = myimage:local
command = mycommand

This will throw the issue:

2018/10/09 15:53:45 scheduler.go:120 ▶ ERROR mycommand - Job finished "fff139bda4de" in "1.816653626s", failed: true, skipped: false, error: error pulling image "myimage:local": API error (404): pull access denied for myimage, repository does not exist or may require 'docker login'

Run Job in new container

Hello,

I really like your job on ofelia and I wanted to know if I could run a job in a new container with environment variable? It will help me a lot for a new project.

Best, and please keep going on it :)

job-service-run: can't store data at section

When setting a job to run a service, I get the error in the title.
I'm running ofelia as a service in docker swarm.
Here is my config.

[job-service-run "servie-executed-on-new-container"]
schedule = "* */10 * * *"
image = svc-backup:latest
network = traefik-network

TLS/SSL error when trying to send mail to TLS enabled mail server

Hi,

Ofelia looks like a great job scheduler - thanks for your work :-D

When trying to use your mcuadros/ofelia image, sending mails to TLS enabled mail server returns an error. I'm not a Go expert, but it looks like the image is missing root certificates to validate the server cert:

2018/06/29 20:53:45 mail.go:57 ▶ ERROR Mail error: "x509: failed to load system roots and no roots provided"

Exec job in stopped container

I have a container with several mounted volumes, and I wish ofelia to run a command on it. The only problem is, that this container is not running by default.

Would it be complicated to add an option to start the container in case it isn't running?

can't store data at section "job-run"

What does this mean?

docker run -it -v /etc/ofelia:/etc/ofelia mcuadros/ofelia:latest
warning:
can't store data at section "job-run", subsection "job-executed-on-new-container", variable "container"


[job-run "job-executed-on-new-container"]
schedule = @every 30s
container = zopanix/casperjs
command = casper.js --web-security=no --ssl-protocol=any --ignore-ssl-errors=ye$

[job-run "job-executed-on-new-container"]
schedule = @every 60s
image = node:4
command = node xmltojson.js

Readme Installation

Hello,

I don't know if this is intentionally missing from the installation section, but I had to mount docker socket otherwise ofelia is complaning:
2017/01/30 13:39:25 scheduler.go:120 ▶ ERROR job-executed-on-running-container - Job finished "4b89cd11bf19" in "330.614µs", failed: true, skipped: false, error: error creating exec: Post http://unix.sock/containers/opendj/exec: dial unix /var/run/docker.sock: connect: no such file or directory

I fixed this with:

-v /var/run/docker.sock:/var/run/docker.sock

Any alternative? Thanks!

Give env variables to container to execute job

In configuration config.ini we can give only the image to execute a container but we need to give environment variables to start this container.

For example :
[job-run "job-executed-on-new-container"]
schedule = @hourly
image = ubuntu:latest
command = touch /tmp/example
environment = ENV1=example,ENV2=example2...

Display output of jobs

Is there any way of outputing the output of ran jobs (especially thinking about the local jobs here) to the console?

Thank you,

Syntax error: Unterminated quoted string

/usr/local/bin/ofelia: 1: /usr/local/bin/ofelia: ��������: not found
/usr/local/bin/ofelia: 1: /usr/local/bin/ofelia: Syntax error: Unterminated quoted string

Getting this error message with this config.ini

[job-exec "job-executed-on-running-container"]
schedule = @every 5m
container = my-container
command = /usr/local/bin/php -q /var/www/html/scripts/jobs.php

What am I doing wrong?

Use private images

Hi,

First i would like to thank you for your great work.

Since we are going to use Ofelia to setup some cron jobs, we want to know if there is a way to authenticate to docker hub and be able to retrieve private images.
The issue we had is when we use the job-run, we get an error about docker not being able to retrieve the private image.

If there is a way to do so, it would be great to add it to the documentation.

thanks in advance.

remove old log files

Hi there,

I really like the logging feature. But i want to be able to remove log files older then say 7 days.
I mounted the ofelia logs volume: logs/cron:/tmp/logs

Next i tried to just remove all files:

[job-exec "clean logs"]
schedule = @every 20s
container = ofelia
command = rm /tmp/logs/*

Unfortunately this fails with stdout unable to find user root: no matching entries in passwd file

I also tried to run it on the host:

[job-local "clean logs"]
schedule = @every 20s
command = rm  /home/florian/Projects/myproject/logs/cron/*

But it fails with : failed: true, skipped: false, error: exec: "rm": executable file not found in $PATH

Any suggestions?

/var/run/docker.sock: connect: no such file or directory

Tried my first cron yesterday:
/var/run/docker.sock: connect: no such file or directory... any tips??:

2016/02/05 05:12:10 scheduler.go:34  NOTICE New job registered "job-executed-on-new-container" - "/mnt/docker/google-domains-ddns-updater/init.d/update-ddns start" - "@hourly"
2016/02/05 05:12:10 scheduler.go:54  DEBUG Starting scheduler with 1 jobs
2016/02/05 06:00:00 scheduler.go:103  DEBUG job-executed-on-new-container - Job started "a26af2b652b6" - "/mnt/docker/google-domains-ddns-updater/init.d/update-ddns start"
2016/02/05 06:00:00 scheduler.go:120  ERROR job-executed-on-new-container - Job finished "a26af2b652b6" in "263.004s", failed: true, skipped: false, error: error pulling image "jh/ubuntu-cron": dial unix /var/run/docker.sock: connect: no such file or directory

config.ini looks like this:

[job-run "job-executed-on-new-container"]
schedule = @hourly
image = jh/ubuntu-cron
command = /mnt/docker/google-domains-ddns-updater/init.d/update-ddns start

edit: title clarified

Posting to slack webhook fails

Hi, I recently found Ofelia and it looks great! Seems very simple to use. I'm testing it with a simple config.ini:

[global]
slack-webhook = https://hooks.slack.com/services/xxxxxxxxx/yyyyyyyyy/zzzzzzzzzzzzzzzzzzzzzzzzzzz

[job-run "job-executed-on-new-container"]
schedule = * * * * *
image = ubuntu:latest
command = touch /tmp/example

Yet, when I look into the logs I see:
2018/09/06 13:39:55 slack.go:64 ▶ ERROR Slack error calling "https://hooks.slack.com/services/xxxxxxxxx/yyyyyyyyy/zzzzzzzzzzzzzzzzzzzzzzzzzzz" error: "Post https://hooks.slack.com/services/xxxxxxxxx/yyyyyyyyy/zzzzzzzzzzzzzzzzzzzzzzzzzzz: x509: failed to load system roots and no roots provided"

I tried to google the error and it seems to be related with missing CA certificates. Is there anything wrong in the config file?

failed: true, skipped: false, error: error pulling image

2017/03/07 16:11:42 scheduler.go:120 ▶ ERROR job-executed-on-new-container - Job finished "17cbafb1763f" in "1.024575ms", failed: true, skipped: false, error: error pulling image "node:4": dial unix /var/run/docker.sock: connect: no such file or directory

docker-compose / scaled container support

Would be nice to be nice to be able to trigger a script on all instances of a container. Using docker-compose you can end up with docker_nginx_1, docker_nginx_2, ..., docker_nginx_N after scaling nginx up.

Proposal:

Add in the use of

[job-exec "job-executed-on-running-container"]
schedule = @hourly
compose-project= docker
container = nginx
command = /etc/init.d/nginx reload

As a dynamic shortcut for:

[job-exec "job-executed-on-running-container"]
schedule = @hourly
container = docker_nginx_1
command = /etc/init.d/nginx reload

[job-exec "job-executed-on-running-container"]
schedule = @hourly
container = docker_nginx_2
command = /etc/init.d/nginx reload

Where compose-project is -p, --project-name NAME Specify an alternate project name (default: directory name).

Thought?

Live jobs discovery from docker labels

It would be nice to discover jobs looking at container and services tags (à la traefik)

eg, in a docker-compose file :

labels:
  - ofelia.schedule: "@hourly"
  - ofelia.command: "touch /tmp/example"

it could support job-exec and job-service-run (once #49 is merged) or maybe a job-service-exec style

this would make it much easier to work with as you wouldn't have to give access to the container running ofelia or its configuration to the user running app containers or stacks

the idea come from traefik and https://github.com/vfarcic/docker-flow-cron but this project is not ready nor very active

which kind of `command` can I use ?

I run ofelia in a container, and just have configures like this:

[job-local "a-job"]
schedule = @every 10s
command = date >> /tmp/date.log

traced log using docker log:

2016/12/29 05:44:37 scheduler.go:124 ▶ NOTICE work-in-workstation - Job finished "30b8dc1f1104" in "116.700666ms", failed: false, skipped: false, error: none

But, I do not find the /tmp/date.log in the container.

If I change the move the command to a shell script something like /tmp/run.sh,
then change the config command line to /tmp/run.sh, It works fine.

The doc or readme have no description about the command,
Would you give out some usages ?

Run from Local Image?

I have built an image locally local/manage_uploads:v1 and it seems like it's not being found.

[job-run "hello-world"]
schedule = */1 * * * *
image = local/manage_uploads:v1
command =  echo "hello world!"

When I start ofelia with the above configuration, I get:

task_scheduler01  | 2018/04/16 21:17:52 scheduler.go:103 ▶ DEBUG hello-world - Job started "0c3c7368fa4d" - "echo hello world!"
task_scheduler01  | 2018/04/16 21:17:52 scheduler.go:120 ▶ ERROR hello-world - Job finished "0c3c7368fa4d" in "586.370188ms", failed: true, skipped: false, error: error pulling image "local/manage_uploads:v1": API error (404): {"message":"pull access denied for local/manage_uploads, repository does not exist or may require 'docker login'"}

Is there a way for me to allow ofelia to use a local image?

Fails for me on local host and creating containers, questions about targeting

How can I target a container on another host?

I tried the option where it tries to create a container, but it fails to pull the image.
5/6/2016 10:16:14 PM2016/05/07 05:16:14 �[31mscheduler.go:120 ▶ ERROR�[0m job-executed-on-new-container - Job finished "6c0daef33999" in "206.425µs", failed: true, skipped: false, error: error pulling image "ubuntu:latest": dial unix /var/run/docker.sock: connect: no such file or directory

I tried the one where it target the local host using the touch /tmp/example but that failed as well.
5/6/2016 10:25:35 PM2016/05/07 05:25:35 �[32mscheduler.go:124 ▶ NOTICE�[0m job-executed-on-current-host - Job finished "52a4541f03bc" in "1.226501ms", failed: false, skipped: false, error: none

Hot reload of the config

Using inotify or maybe with a signal, allow to reload the configuration without the requirement of restart the process

Attach new container to a network

For the job executed on a new container we need to have the possibility to attach this container to a network.

For example :
[job-run "job-executed-on-new-container"]
schedule = @hourly
image = ubuntu:latest
command = touch /tmp/example
network = my-net

Multiple `email-to`

It would be great to have a possibility to send logs to multiple emails. I mean replace EmailTo with array of strings in configs instead of string.

How about this feature?

web ui

a very simple web ui could be interesting, I'd like to be able to list configured jobs (would help trusting #5 )

and later more features could be added if useful, eg: job run history, logs, etc.

cannot unmarshal array into Go value of type docker.Container

I might be missing something but I can't get any commands to run.
Here's how to recreate my scenario -

ofelia.ini:

[job-run "Test"]
schedule = @every 2s
command = echo "hello"

and I run
docker run -it -v ${PWD}/ofelia.ini:/etc/ofelia/config.ini -v /var/run/docker.sock:/var/run/docker.sock mcuadros/ofelia:latest

The result is
scheduler.go:120 ▶ ERROR Test - Job finished "ea3c6191a546" in "3.9365ms", failed: true, skipped: false, error: json: cannot unmarshal array into Go value of type docker.Container

Volume support?

Trying to run a node script and was trying to mount a volume? Is this possible? I think I'm confused.
I'm trying to run a casperJS script and then a node script. Both of these scripts are on the host.

[job-run "job-executed-on-running-container"]
schedule = @every 30s
container = zopanix/casperjs --rm -v /etc/ofelia/scripts:/data
command = casper.js --web-security=no --ssl-protocol=any --ignore-ssl-errors=yes --city=AR --output=XML --filename=latest.xml

[job-run "job-executed-on-new-container"]
schedule = @every 60s
image = node:4 -v /etc/ofelia/scripts:/usr/src/app -w /usr/src/app
command = node xmltojson.js

Smaller image

Hi!

I noticed that the docker image is quite large and tried to fix it using this approach: https://blog.codeship.com/building-minimal-docker-containers-for-go-applications/

See here: https://github.com/bonndan/ofelia

I messed around with the makefile and go 1.7.4. and I HAVE NO IDEA WHAT I AM DOING! Could you take a quick look?

The image is now 11MB and starts properly reading to job configurations, but the I don't see any activity although I added minutely jobs. Is that correct?

Last question: Do you still use ofelia today?

New job type `run`

Use run instead of exec as alternative to a always running container. Delete it after executed.

Logs are not taking the leap second into account

The log messages are not strictly ordered with time. E.g.

2017/01/01 00:00:00 scheduler.go:103 ▶ DEBUG github-stars - Job started "0d97c4adebbf" - "/bin/sh -c ..."
2016/12/31 23:59:59 scheduler.go:120 ▶ ERROR github-stars - Job finished "0d97c4adebbf" in "-992.473897ms", failed: true, skipped: false, error: error starting exec: Unrecognized input header

I believe this happened because of the leap second.

Cannot build master

The build gets an error on the step 5/7

The command '/bin/sh -c go get -v ./... && go install -v ./... && rm -rf $GOPATH/src/' returned a non-zero code: 2

Config.ini Documentation

As I'm not sure of the config.ini syntax, how do I use the mail, save, and slack logging features? I don't see it clearly explained in the documentation. Could you provide a full example of a config.ini that uses the various forms of logging? Thanks!

New job type `local`

Classic local process execution, maybe is useful for maintenance tasks inside of the container.

daisy chain jobs

Say I have:

# Step 1. create new cert
[job-run "job-executed-on-new-container"]
schedule = * * * * *
image = willfarrell/letsencrypt
command = dehydrated \
            --cron \
            --domain certbot.willfarrell.ca \
            --out /etc/ssl \
            --challenge http-01

# Step 2. reload cert into nginx
[job-exec "job-executed-on-running-container"]
schedule = * * * * *
container = docker_nginx_1
command = /etc/scripts/make_hpkp \
            && /etc/init.d/nginx reload

Would it be possible to have step two triggered when step one is complete.

Proposal:

[job-run "job-executed-on-new-container"]
name = letsencrypt-renew
schedule = * * * * *
image = willfarrell/letsencrypt
command = dehydrated \
            --cron \
            --domain certbot.willfarrell.ca \
            --out /etc/ssl \
            --challenge http-01

[job-exec "job-executed-on-running-container"]
trigger = letsencrypt-renew
container = docker_nginx_1
command = /etc/scripts/make_hpkp \
            && /etc/init.d/nginx reload

Where you can set the name of a job and have a trigger that runs a new job once the trigger job is complete.

Thought?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.