holunda-io / camunda-bpm-cloud Goto Github PK

Discontinued, look for camunda-bpm-taskpool

Home Page: https://github.com/holunda-io/camunda-bpm-taskpool

License: Apache License 2.0

Java 90.70% Shell 7.53% Dockerfile 1.77%

camunda cloud microservice axon rabbitmq

camunda-bpm-cloud's Introduction

This project is discontinued. New ideas are contributed to Camunda BPM Taskpool. https://github.com/holunda-io/camunda-bpm-taskpool

Camunda BPM Cloud

Problem Statement

Process applications are following the BPM/SOA orchestration pattern invented many years ago. This promotes the separation of orchestration and the business functions. Since the existence of BPMN 2.0 standard, the orchestration is performed by the business process engine executing the BPMN 2.0 directly - which fosters the business-IT alignment.

The bad part of the story is, that this pattern is implemented in the same way, as 20 years ago: as a monolith. In the early 90-ties the workflow engine (the predecessor of business process engines) were large installations hosted on big servers (fostered by big manufacturers). The were two reasonable runtime strategies: hosting business functions on the same machine as the business process engine, or deploying the business functions to other machines and strong distributed transactions to guarantee the consistency of the data. Both strategies result in a system thats runtime behaviour corresponds to those of a monolith, with all drawbacks and pains related to it operations.

In the ara of cloud computing, Microservices and self-contained systems those runtime strategies are not state of the art.

Mission Statement

The goal of Camunda BPM Cloud project is to provide a implementation strategy of the BPM/SOA orchstration pattern that fosters louse coupling, dynamic scaling and resilience of the resulting system. In doing so, the following principles are applied:

The orchestration layer (process engine with BPMN 2.0 model) must not have runtime dependencies to business/domain services (Usage of External Service Tasks)
The high availability (HA) of the process engine can be reached by creating a homogenous cluster (clustered BPE, clustered shared DB, same deployments).
The co-existence of several process engines must not rely on shared database.
The user needs a single workload view, independing from the amount of process engine.

Implementation Strategy

Configuration and Registry

The engines are using Camunda BPM Spring in order to achieve maximum flexibility in ad-hoc deployment. In order to ease the management and configuration of the engines, a registry and a config server is used (from Spring Cloud project).

Workload Service

In order to achive maximum decoupling from the engines to the business services and the task list application, a workload service should collect user and external service task from multiple engines.

Edge Service

In order to provide a single point of integration for a common task list (displaying the united workload of all process engines) an edge service should be provided.

Current state of implementation

Runtime and Deployment strategy

In order to automate the runtime deployment docker shoould be used as a container for any component.

Requirements

You would need the following software in your environment to run the example:

Java 8
Maven 3.3
Docker 1.11 (native or via VirtualBox)
Docker Compose 1.7.1
npm >= 3.9.6
node 4.4.3

Start-up

We run all services in docker containers. To build the images, use the dockerBuild profile:

mvn -PdockerBuild clean install

This will create docker images (prefix camunda-bpm-cloud) that you can check with docker images. (Hint old images will be "dangling", you can delete them via docker images -f "dangling=true" -q | xargs docker rmi).

The containers are configured using docker-compose, but they have to be started in a certain order. In order to simplify the start, the run script is provided. Just call from console:

./run.sh

This will start the services:

discovery
config
edge
workloadservice
simpleprocess
trivialprocess

In order to start the Cloud Tasklist please run:

 ./tasklist.sh

Runing with Docker

If you use native docker hosts on our machines, no need for additional VBox configurations is required. If you run docker based on VirtualBox, please make sure to expose the IP address of the Docker host by exeucting the following line of code: export DOCKER_IP=$(docker-machine ip $(docker-machine active))

Start all containers with

-P map all ports
--net="host" use hosts network device directly, so localhost works fine, found here

Restarting a single container

docker-compose stop camundabpmcloud_workloadservice_1
mvn clean install -f extension/workload-service/ -PdockerBuild
docker-compose up -d workloadservice
docker logs -f camundabpmcloud_workloadservice_1

Analyse logs using ELK

In order to have a more detailed insight on what is happening inside the private cloud, you can use

./runs.sh -e

to start-up an additional Docker container hosting

Elasticsearch
Logstash
Kibana

All other containers are configured to send their logs via tcp to the configured tcp input channel of Logstash, if it is available. Logstash outputs them to the Elasticsearch installation within the same Docker container to index them. These indexes can be analysed using the Kibana dashboard.

Components

Service-discovery / Registry (eureka)

The EurekaServer starts up and serves as service-registry.

Config-service (configuration)

When the ConfigServer is started, it registers itself as CONFIGSERVER at EurekaServer.

Workload service

The WorkloadService registers itself as WORKLOADSERVICE at EurekaServer and provides

a REST endpoint for the EventBroadcasters used in ProcessApplications,
a H2 database to store the tasks for query access (TaskQueryObjectRepository) and
a REST endpoint for the external task list (stripped down camunda REST-API having one additional field engineUrl).

Internal structure of the workload service

The workload service is implemented using the Axon framework.

Edge Service (zuul)

The edge service acts as a reverse proxy to access the workload service from outside of the cloud.

Cloud Tasklist

The Cloud task list is a SpringBoot Application containing a task list and a component which connects to the Camunda cloud vias edge service.

Example Scenarios

If an instance of a process is created, the SimpleProcess and/or TrivialProcess broadcast TaskEvents for every task that is created, completed or deleted to the WorkloadService.
The WorkloadService caches the Tasks using its database.
The Cloud tasklist queries the WorkloadService for all cached Tasks.
By clicking on a task, details of the task are shown and a complete button is present.
When using the complete button, the task list sends a request to complete the task directly to the engine identified by the task.engineUrl field.

Resources

ASCII banner generator, font: standard
Spring Cloud Config Server

camunda-bpm-cloud's People

Contributors

Stargazers

Watchers

camunda-bpm-cloud's Issues

Trivial process throws exception "task supervisor rejected the task"

2016-06-30 16:25:30.220 ERROR 1 --- [ver-bootstrap-0] c.netflix.discovery.TimedSupervisorTask  : task supervisor rejected the task

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3323acbd rejected from java.util.concurrent.ThreadPoolExecutor@3de454a9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[na:1.8.0_92]
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) [na:1.8.0_92]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) [na:1.8.0_92]
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) ~[na:1.8.0_92]
    at com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:62) ~[eureka-client-1.4.6.jar!/:1.4.6]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_92]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_92]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_92]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_92]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_92]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_92]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92]

seems to be related to spring-dev-tools (unconfirmed)

configserver: allow change of configuration yaml without restart

currently, our yaml files are in src/main/resources/config ... that's not nice, because we have to rebuild the docker image for every config change. We need to configure an url where to load the files.

move non-extension-services to "infrastructure"

the extension modukle should only contain libs and services that are "core" extensions for bpm-cloud. Move config, discovery and edge to separate directory

remove discovery-example-server spike

Create the edge service to access process engines for direct task manipulations.

The edge service should be used as a facade to the process engines which store the user tasks.

use camunda-bpm-reactor 2.0.0 in broadcaster

new version is out - changes to full java 8 functional support

remove "old" tasklist

the examples/tasklist is no longer required

use spring-cloud bom dependency-management

We should use the Brixton bom

 <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-dependencies</artifactId>
        <version>${spring-cloud.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>

instead of manual netflix deps

 <dependencies>
      <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-netflix</artifactId>
        <version>1.1.0.RELEASE</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>

support "local" configuration to run services outside docker-compose

The idea is, that during development, we have to run the infrastructure components via docker-compose

these are

mysql
rabbitmq
discovery
configserver
edgeserver

But we can decide via active-profile switch, if we want to run the process-engines and the workload services directly from the console/IDE so we do not have to rebuild and restart the docker containers anymore.

Motivation: reduction of roundtrip time

For this to work we need to:

provide bootstrap-local.yml for all components in question where we define a localhost configserver
provide "*-local.yml" configurations for all components that use "localhost" references instead of docker-network internal names.

Challenges: This should work without completely copying the yaml files, it should be enough to just provide a delta in the -local files, preferably just the different hostnames.

Note: to make this work on machines where docker-host is on a vm, we need to use ${DOCKER_IP:localhost} instead of localhost so when a docker-ip is present, it is used instead of the localhost.

remove "docs"

additional docs in the gh-pages branch add an extra level of complexity that we can as well deal with later. For now, using README.md files is enough.

fix travis npm build

we are getting an error building the cloud-tasklist:

Done. Phantomjs binary available at /home/travis/build/holisticon/camunda-bpm-cloud/examples/cloud-tasklist/node_modules/karma-phantomjs-launcher/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs
npm ERR! peerinvalid The package grunt does not satisfy its siblings' peerDependencies requirements!
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.2
npm ERR! peerinvalid Peer [email protected] wants grunt@~0.4.1
npm ERR! peerinvalid Peer [email protected] wants grunt@>= 0.4.5
npm ERR! peerinvalid Peer [email protected] wants grunt@>= 0.4.5
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.0
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.1
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.0
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.5
npm ERR! peerinvalid Peer [email protected] wants grunt@~0.4.0
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.0
npm ERR! System Linux 3.13.0-40-generic
npm ERR! command "/home/travis/.nvm/v0.10.36/bin/node" "/home/travis/.nvm/v0.10.36/bin/npm" "install"
npm ERR! cwd /home/travis/build/holisticon/camunda-bpm-cloud/examples/cloud-tasklist
npm ERR! node -v v0.10.36
npm ERR! npm -v 1.4.28
npm ERR! code EPEERINVALID
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /home/travis/build/holisticon/camunda-bpm-cloud/examples/cloud-tasklist/npm-debug.log
npm ERR! not ok code 0

Seems to be related to an outdated npm/grunt version, see drewzboto/grunt-connect-proxy#139

error on task list maven build

[INFO] 
[INFO] Running "postcss:dist" (postcss) task
[ERROR] Container#eachAtRule is deprecated. Use Container#walkAtRules instead.
[ERROR] Container#eachRule is deprecated. Use Container#walkRules instead.
[ERROR] Container#eachDecl is deprecated. Use Container#walkDecls instead.
[INFO] >> 1 processed stylesheet created.
[INFO]

seems to work anyway ...

tasklist/frontend - use target/generated-resources

Currently, grunt builds to src/main/resources/static which is excluded via gitignore.
It would be better to use target/generated-resources so we do not polute the src-space.

Implement the external service task component

Similar to the user workload component, the service task component should be implemented.

Complete with command/event

Complete tasks based on commands (not REST calls)

clean up pom

spring boot 1.3.5
camunda 7.5.0
remove unsed dep/dep-mgt
disable camunda ee

adding banners to applications

we should have banners to identify apps.

I already started with config-server and eureka.

Provide a Camunda(-like) API on the edge service

camunda rest-like task api for task query (camunda tasklist as client)

switch to camunda spring boot 2.1.0

with 2.1 we will have:

camunda 7.6
spring boot 1.5.3

and, most important: a prefixed uuidGenerator ... so we can transport the app Name with the taskId.

Create an integration scenario for a user accessing different BPEs

Build a identity service which is used by all process engines, and allow the user to see in his task list user tasks from different process engines.

Replay based on shared eventstore

Implement aggregate restore from event store (replay).

use mysql as axon-eventstore

currently, axon uses an h2 in memory eventstore, so when we restart the workloadservice, we can not recreate the query-view from an existing eventstream, all tasks are lost.

use mysql (current db) as event store.

events are not processed by query service

though events are received, they cannot be processed ...

2017-06-19 21:25:55.071  INFO 78567 --- [cTaskExecutor-1] .c.w.q.s.WorkloadQueryServiceApplication : receiving event: (Body:'[B@1548971e(byte[551])' MessageProperties [headers={axon-message-id=8120845e-4533-4b19-8158-a889d281d918, axon-message-aggregate-seq=0, axon-metadata-correlationId=2bc64c7f-d358-4aa8-8e75-4c5fc42a08db, axon-message-aggregate-type=TaskAggregate, axon-message-revision=null, axon-message-timestamp=2017-06-19T19:25:55.066Z, axon-message-type=org.camunda.bpm.extension.cloud.workload.event.TaskCreatedEvent, axon-metadata-traceId=2bc64c7f-d358-4aa8-8e75-4c5fc42a08db, axon-message-aggregate-id=simpleprocess-1d3a9a00-5525-11e7-a954-acde48001122}, timestamp=null, messageId=null, userId=null, receivedUserId=null, appId=null, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=null, contentEncoding=null, contentLength=0, deliveryMode=null, receivedDeliveryMode=PERSISTENT, expiration=null, priority=null, redelivered=false, receivedExchange=camunda-task-event-exchange, receivedRoutingKey=org.camunda.bpm.extension.cloud.workload.event, receivedDelay=null, deliveryTag=32, messageCount=0, consumerTag=amq.ctag-jF3lQoTeovJm05FRo28lxw, consumerQueue=camunda-task-event])#Cached Rabbit Channel: AMQChannel(amqp://[email protected]:5672/,1), conn: Proxy@214334c8 Shared Rabbit Connection: SimpleConnection@5aa25f98 [delegate=amqp://[email protected]:5672/, localPort= 57935]
2017-06-19 21:26:35.072  WARN 78567 --- [cTaskExecutor-1] o.s.a.s.c.Jackson2JsonMessageConverter   : Could not convert incoming message with content-type [null]

application version does not show in banner


Application:  
Spring-Boot:  (v1.5.3.RELEASE)

refactor maven modules

use groupId org.camunda.bpm.extension.cloud - all extensions with all submodules otherwise are on the same level in repo
use short names for module dirs (discovery-service instead of camunda-bpm-cloud-discovery-service) - easier on the eyes when browsing through the project

activate full uuid for task ids

both processes should generate full uuids

Implement decoupling queue between REST call to the event service and the REST call to the process engine.

Scenario

 Given task in tasklist
 when user complets task in tasklist
 then complete task event is created in eventservice store
 and the task is no longer in tasklist

 Given active task in event service store
 when a complete task event arrives
 then a rest call "complete" is send to coresponding process engine
 and the task is completed

Look at API called by the Camunda TaskList and create the Endpoint serving it.

every rest api we provide on eventservice should match the camunda signature

replace custom config server with docker core image

We do not really need to build our own config server, it is part of the infrastructure and should be replaced by a standard image, see https://hub.docker.com/r/hyness/spring-cloud-config-server/

let broadcaster publish to rabbitmq instead of rest

when the process engine uses broadcaster to notify the workloadservice via rest and the workloadservice is not available, the message can not be sent. This leads to either: lost tasks or roll back of the process instance.

We should use messaging (via rabbitmq) instead, so we can configure a persistant queue that can be processes by the workloadservice once it is back up.

use travis ci

activate travis ci for the project.

Done:

create travis.yml
add badge to README.md

Fails:

npm build of tasklict-cloud

Done. Phantomjs binary available at /home/travis/build/holisticon/camunda-bpm-cloud/examples/cloud-tasklist/node_modules/karma-phantomjs-launcher/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs
npm ERR! peerinvalid The package grunt does not satisfy its siblings' peerDependencies requirements!
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.2
npm ERR! peerinvalid Peer [email protected] wants grunt@~0.4.1
npm ERR! peerinvalid Peer [email protected] wants grunt@>= 0.4.5
npm ERR! peerinvalid Peer [email protected] wants grunt@>= 0.4.5
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.0
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.1
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.0
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.5
npm ERR! peerinvalid Peer [email protected] wants grunt@~0.4.0
npm ERR! peerinvalid Peer [email protected] wants grunt@>=0.4.0
npm ERR! System Linux 3.13.0-40-generic
npm ERR! command "/home/travis/.nvm/v0.10.36/bin/node" "/home/travis/.nvm/v0.10.36/bin/npm" "install"
npm ERR! cwd /home/travis/build/holisticon/camunda-bpm-cloud/examples/cloud-tasklist
npm ERR! node -v v0.10.36
npm ERR! npm -v 1.4.28
npm ERR! code EPEERINVALID
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /home/travis/build/holisticon/camunda-bpm-cloud/examples/cloud-tasklist/npm-debug.log
npm ERR! not ok code 0

Wrap external tasklist with spring-boot

I did not manage to run the node.js app. Maybe we should wrap it with jhipster?

use configserver banner in hyness-image

we can mount and define a banner location, so our configserver looks like before.

switch to camunda-springboot 1.3

new release is out

provide common database(s) for process examples

So far, each process node (trivial/simple) launches an h2 in mem db. So when we start more nodes of an app, they will only know there own data (tasks), thus we cannot use loadbalancing.

We need a db config with camunda-schema (h2-server?, mysql-docker?) that can be shared among process apps.

Hint: see mysql node setup in https://github.com/holisticon/cughh-springboot-example/tree/master/database

split workloadservice in query and command

currently, we have the command and the query component inside the one workloadservice microservice. We should split them into

workload-command-service
workload-query-service

Motivation

clean separation of concerns according to CQRS/ES architecture
scalability: we can decide to run multiple read or multiple write nodes depending on load

Enhance cloud-tasklist ui

Make it look good.

Complete with variables

minimal form
send variables with complete request
store variables on task in eventservice

remove mysql - switch to h2-file based

less overhead, use file based h2.

note: SERVER_MODE for multi connection!

// Application 1:
DriverManager.getConnection("jdbc:h2:/data/test;AUTO_SERVER=TRUE");

// Application 2:
DriverManager.getConnection("jdbc:h2:/data/test;AUTO_SERVER=TRUE");

Implement the user workload component

The user workload component should store the workload from different process engines.

simple-process: NoSuchMethodError when completing a task in cloud-tasklist

branch: workload-service-axon

2017-03-31 15:20:15.746  ERROR 1 --- [o-auto-1-exec-7] c.b.s.b.s.r.CamundaJerseyResourceConfig] : Allocate exception for servlet org.camunda.bpm.spring.boot.starter.rest.CamundaJerseyResourceConfig

java.lang.NoSuchMethodError: javax.ws.rs.core.Application.getProperties()Ljava/util/Map;
at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:331) ~[jersey-server-2.23.2.jar!/:na]
at org.glassfish.jersey.servlet.WebComponent.<init>(WebComponent.java:392) ~[jersey-container-servlet-core-2.23.2.jar!/:na]
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:177) ~[jersey-container-servlet-core-2.23.2.jar!/:na]
...