Giter Club home page Giter Club logo

easy-jobs's Introduction


Easy Jobs
The simple, stupid job server for Java™

MIT license Downloads Release Project status


Latest news

  • 24/07/2018: Version 0.4 is out to fix a small issue with the startup of the admin UI.
  • 05/11/2017: Version 0.3 is out with a new administration web interface. See all changes in details here.
  • 29/06/2017: Version 0.2 is released with some bug fixes and few enhancements. See all changes in details here.
  • 22/06/2017: Version 0.1 is out! See what this first version brings to the table here.

What is Easy Jobs?

Easy Jobs is a simple job server for Java. It allows you to define jobs and request their executions through a RESTful API.

How does it work?

Easy Jobs stores meta-data of jobs in a relational database. Three tables are used: job, job_execution and job_execution_request. The job server polls the job_execution_request table regularly looking for pending job execution requests. When a job execution request comes in, the job server creates a job instance of the requested job and executes it:

The job server uses a pool of worker threads to execute jobs. Job execution requests are submitted through a RESTful API. For more details about these design choices, please refer to the technical notes.

How to use it ?

Download the latest release and unzip it. You should get a directory with the following content:

$>cd easy-jobs-dist-0.4
$>tree -d
├── conf
├── drivers
│   ├── h2
│   └── mysql
├── jobs
└── lib
    ├── admin
    └── server

Run the job server with the following command:

java -cp "drivers/h2/*:lib/server/*" \
 -Deasy.jobs.database.config.file=$(pwd)/conf/database.properties \
 -Deasy.jobs.database.config.init=true \
 -Deasy.jobs.server.config.jobs.directory=$(pwd)/jobs \
 -Deasy.jobs.server.config.jobs.descriptor=$(pwd)/conf/jobs.yml \
 org.jeasy.jobs.server.JobServer

If you are on windows, use the following command:

java -cp "drivers/h2/*;lib/server/*" ^
 -Deasy.jobs.database.config.file=%cd%\conf\database.properties ^
 -Deasy.jobs.database.config.init=true ^
 -Deasy.jobs.server.config.jobs.directory=%cd%\jobs ^
 -Deasy.jobs.server.config.jobs.descriptor=%cd%\conf\jobs.yml ^
 org.jeasy.jobs.server.JobServer

That's it! The job server should be up and running waiting for you to submit job execution requests. To submit a job execution request, you can use either the REST API or the Admin Web Interface.

REST API

By default, the job server will be started on localhost:8080. You can change the port as well as other parameter as described in the wiki. In the previous command, we used H2 database which is fine for testing but not recommended for production. You can use another supported database if you want.

The distribution comes with a sample job called HelloWorldJob located in the jobs directory. Here is its source code:

public class HelloWorldJob {

    private String name;

    public void doWork() {
        System.out.println("Hello " + name);
    }

    // getter and setter for name
}

Jobs in Easy Jobs are regular Java classes. There is no annotation to add, no interface to implement or class to extend. Your jobs are simple POJOs. Easy Jobs is not intrusive! But you have to tell it where to find your job using a job descriptor:

---
id: 1
name: hello world job
class: HelloWorldJob
method: doWork

This job descriptor jobs.yml can be found in conf directory. It gives Easy Jobs all required information to identify your job and execute it when requested. Let's first check if the HelloWorldJob is registered:

$>curl localhost:8080/jobs
[
 {
  "id": 1,
  "name": "Hello World Job",
  "description" : "description: A job that says 'hello' to the given name"
 }
]

The job server has successfully loaded the job. Now, we can submit a job execution request:

$>curl \
  --request POST \
  --header "Content-Type: application/json" \
  --data '{"jobId":"1", "name":"world"}' \
  localhost:8080/requests

The job server will pick up this request in the next polling run, create a job instance of the HelloWorldJob and execute it with parameter name=world. Let's check job executions on the /executions endpoint:

$>curl localhost:8080/executions
[
 {
  "id": 1,
  "requestId": 1,
  "jobExecutionStatus": "FINISHED",
  "jobExitStatus": "SUCCEEDED",
  "startDate": [
      2017, 6, 23, 9, 25, 13, 939000000
  ],
  "endDate": [
      2017, 6, 23, 9, 25, 13, 959000000
  ]
 }
]

Great! the job has been executed and finished successfully. You should have seen this in the server's log:

INFO: Received a new job execution request for job 1 with parameters {jobId=1, name=world}
INFO: Found 1 pending job execution request(s)
INFO: Creating a new job for request n° 1 with parameters [{"jobId":"1", "name":"world"}]
INFO: Submitted a new job for request n° 1
INFO: Processing job execution request with id 1
Hello world
INFO: Successfully processed job execution request with id 1

That's all! You can find more details on how to configure the server in the wiki.

You can also download a pre-configured insomnia or postman workspace with all endpoints to help you play around with Easy Jobs server through its REST API.

Admin Web Interface

Easy Jobs comes with an administration web interface:

This interface gives you some insights on a running job server and allows you to submit job execution requests. To run the application, use the following command:

java -cp "drivers/h2/*:lib/admin/*" \
 -Deasy.jobs.database.config.file=$(pwd)/conf/database.properties \
 -Dserver.port=9000 \
 org.jeasy.jobs.admin.Application

On windows, you can use the following command:

java -cp "drivers/h2/*;lib/admin/*" ^
 -Deasy.jobs.database.config.file=%cd%\conf\database.properties ^
 -Dserver.port=9000 ^
 org.jeasy.jobs.admin.Application

For demonstration purpose, you can login to the application using admin/admin credentials. Make sure the server and the admin interface are running on different ports. In this example, the server is running on port 8080 (by default) and the admin interface on port 9000.

Contribution

You are welcome to contribute to the project with pull requests on GitHub.

If you find a bug or want to request a feature, please use the issue tracker.

Awesome contributors

Thank you for your contributions!

License

Easy Jobs is released under the terms of the MIT license:

The MIT License (MIT)

Copyright (c) 2020 Mahmoud Ben Hassine ([email protected])

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

easy-jobs's People

Contributors

dependabot[bot] avatar fmbenhassine avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

easy-jobs's Issues

Add ability to stop a job execution

Jobs are submitted to a pool of threads.
Stopping a thread in Java is not the easiest thing to do 😄

It should be possible through the admin interface to stop an execution.
The server should keep track of the execution's Future objects and call cancel on the requested one.

This will set the interrupted flag on the execution thread which should be detected by the user's job using Thread.currentThread().isInterrupted(). This is possible out of the box if the job is an Easy Batch job.

Add job details page

The job descriptor should support describing which job parameters (and their types) are expected.

Then, the GUI can introspect the descriptor (or better the job description in the DB) to assist the user in creating job execution requests.

These details can be shown in a Job details page.

Add config file templates in the distribution

Just like issue #9 , the goal is to add templates files for job descriptor and database properties that can be directly modified by the user.

This will avoid guessing database properties keys or job descriptor attributes.

Another credit goes to @nabilov for this suggestion 👍

Add ej_ prefix to tables

Table names may interfere with existing tables.

The idea is to add a prefix specific to easy jobs:

  • job -> ej_job
  • job_execution -> ej_ job_execution
  • job_execution_request -> ej_ job_execution_request

Add startup scripts

There should be a shell/batch script to start the server/admin apps for unix/windows systems.

A work in progress is in branch startup-script

Add a job example in the distribution

In order to make it easy to get started with Easy Jobs, it would be interesting to add a "hello world" job in the distribution file.

This will allow the user to start playing with the job server right away after downloading the distribution instead of creating/compiling/packaging the job himself.

Many thanks to @nabilov for this suggestion!

Add the reason of a rejected request in the admin UI

As of v0.3, when a request is rejected, the reason is not shown in the UI. Hence there is a need to check the server's log in order to understand the issue. It should be possible to check the reason why the request was rejected in the UI (new column with the exception stack trace in the Job execution requests page)

Inconsistent job execution state

As of v0.1, when a job execution is fast enough, the job server cannot find a job execution for the executed job. Here is the exception:

org.springframework.dao.EmptyResultDataAccessException: No entity found for query; nested exception is javax.persistence.NoResultException: No entity found for query
	at org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:392) ~[spring-orm-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.orm.hibernate5.HibernateExceptionTranslator.translateExceptionIfPossible(HibernateExceptionTranslator.java:55) ~[spring-orm-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:59) ~[spring-tx-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:213) ~[spring-tx-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:147) ~[spring-tx-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) ~[spring-tx-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.jeasy.jobs.execution.JobExecutionRepository$$EnhancerBySpringCGLIB$$529c1f6b.findByJobRequestId(<generated>) ~[classes/:na]
	at org.jeasy.jobs.job.JobService.updateJobExecutionAndItsCorrespondingRequest(JobService.java:62) ~[classes/:na]
	at org.jeasy.jobs.job.DefaultJob.call(DefaultJob.java:30) [classes/:na]
	at org.jeasy.jobs.job.DefaultJob.call(DefaultJob.java:9) [classes/:na]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_73]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_73]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_73]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]
Caused by: javax.persistence.NoResultException: No entity found for query
	at org.hibernate.query.internal.AbstractProducedQuery.getSingleResult(AbstractProducedQuery.java:1465) ~[hibernate-core-5.2.10.Final.jar:5.2.10.Final]
	at org.jeasy.jobs.execution.JobExecutionRepository.findByJobRequestId(JobExecutionRepository.java:25) ~[classes/:na]
	at org.jeasy.jobs.execution.JobExecutionRepository$$FastClassBySpringCGLIB$$2068fec7.invoke(<generated>) ~[classes/:na]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) ~[spring-tx-4.3.9.RELEASE.jar:4.3.9.RELEASE]
	... 14 common frames omitted

Add throttling parameter

Let's consider the following server configuration:

  • worker pool size: 10
  • polling interval: 1h
  • number of requests received since last polling run: 1000

As of version 0.1, the server will query for pending requests. It will get all the 1000 requests and submit them to the executor service. With only 10 worker threads, this may exhaust resources on the server.

It should be possible to limit the number of pending requests to take in each polling run.
This should provide throttling/back pressure on the server side.

Add REJECTED status in JobExecutionRequestStatus

When a job execution request is not correct (malformed json), it still in the 'PENDING' status and the job server will continuously try to parse it and submit it. This has a useless overhead in execution, may pollute the logs with repetitive information and may mislead the user.

A new status REJECTED is needed to prevent this overhead.

Isolate jobs classpath from server's classpath

As of v0.1, the classpath of user's jobs is the same as the server's classpath.

This may result in classes clash at runtime.

Easy Jobs should use a separate classloader for user's jobs.

Add allowConcurrent parameter

Sometimes, two instances of the same job should not be executed concurrently.

The goal is to add a allowConcurrent parameter in the job definition to allow/disallow concurrent job executions.

Whenever a job request comes in and there is already an execution running for the requested job, then the request should be postponed (or may be ignored, this should be configurable).

Add job execution details

A job execution may complete successfully or fail.

  • If it succeeds, the return type of the execution method can be serialized and shown in the GUI (it can be a calculation result or a job report as in Easy Batch).
  • If it fails, it would be great to be able to access the exception through the admin interface.

Error while launching the admin web interface

Hi,

When I try to run
java -cp "drivers/h2/*;lib/*" ^ -Deasy.jobs.database.config.file=%cd%\conf\database.properties ^ -Dserver.port=9000 ^ org.jeasy.jobs.admin.Application
I got
java.lang.IllegalArgumentException: Could not resolve placeholder 'application.version' in value "${application.version}"
The server works fine but it's impossible to run the admin web interface.
I didn't make any modifications. I'm just trying to run the "demo" that you provide.
Did I miss something ?
BTW, java version is openjdk 1.8.0_171 on a debian 9.

Thanks

Use YAML in jobs descriptor

YAML is easier for humans to write and less error prone than JSON.

The goal is to write job descriptors in YAML instead of JSON. For example:

---
id: 1
name: my first job
class: org.mycompany.jobs.MyFirstJob
method: doWork
---
id: 2
name: my second job
class: org.mycompany.jobs.MySecondJob
method: doWork

Runtime dependency

What is the easy-jobs runtime dependency? It seems that the server has its own main(), therefore, there is no dependency of, for example, netty http lib?

Add request priority parameter

It should be possible to submit job request with higher priorities to be executed first.

The goal of this issue is to add the possibility to specify a request priority.

Easy Jobs should execute high priority requests first.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.