LocalTaskLauncher limit number of concurrent tasks about spring-cloud-deployer-local HOT 13 CLOSED

@sabbyanandan @markpollack
Can you suggest any alternative for now,
Actually we are heading towards developing a feature using spring-cloud-task and our POC get stuck in this scenario.

from spring-cloud-deployer-local.

Hi, @udaraliyanage @bhuvnesh91 @rajprasadtarget. Thanks for the continued interest in this topic. Based on the internal discussion, here's how we are planning to address it.

In SCDF: (see spring-cloud/spring-cloud-dataflow#2140)

• A Local-Server user would be able to configure a "maximum task launches at a time" as a server-level setting.
• The Local-server would persist it in the DB (some form of a counter)
• The Local-server would then be able to query the DB for the counter and as well as the no. of currently running tasks to determine whether any new tasks can be launched or not. If the maximum threshold is reached, Local-server would gracefully handle the scenario by logging an exception and letting the user know about the limit.

In Local Deployer:

• A cleanup routine would be in place to make sure the Task PIDs in the running Map doesn't grow indefinitely. We would have a separate thread to double check whether the PID is still alive; if not, we will remove it from the Map.
• Also, when a cleanup/destroy routine is invoked from SCDF, we will make sure to cleanup the PID from running Map. This is already in place for Streams; we would implement it for Tasks now.

from spring-cloud-deployer-local.

We are addressing this via spring-cloud/spring-cloud-task#81.

from spring-cloud-deployer-local.

@sabbyanandan I feel that it addresses a different scenario. It is to have only one instance of a perticular task.
But my requirement is to limit the number of concurrent processors. For instance if max is 3, there can be only three tasks/processors running. If more task requests are received, they will have to wait till one out of three already running tasks exits.

from spring-cloud-deployer-local.

Hi, @udaraliyanage. What we are addressing in spring-cloud/spring-cloud-task#81 is indeed to prevent the simultaneous launch of the same task. It appears you see stability issues when launching "n" number of different types of task applications - did I get that right? If yes, could you describe the use-case with a concrete example maybe? Also, what is the actual behavior when it hits the unstable state?

from spring-cloud-deployer-local.

Hi @sabbyanandan ,

We are creating a in house function as a service platform using task launcher. Developers implement tasks (maven) and upload to maven repo. Then send a task request when they want to get the task executed. There are many tasks(maven1, maven2....) implemented by devs. When devs send task launch requests via Kafka, the task launcher receives them and start executing them as a subprocess.

The problem is when there are many task requests are received, the launcher start executing them all in parallel, resulting in many subprocess are executing at the same time. This is becacuse TaskLauncherSink.launchTask() returns as soon as subprocess started, not when completed. In our case the launcher is running as a Docker instance. When there are many subprocess started, it slows down all processors and sometime Docker kills the instance if it detects high memory/cpu usage.

The suggested solution is to introduce max number of tasks that can be executed simultaneously. The task launcher hold consuming task launch requests when there are already max number of tasks running. Next tasks launch requests are executed when one of running tasks exited. Basically first come first served.

I already implemented it locally. Issue 76 is also connected to this.

What do you think of te relevancy of the requirement ?

from spring-cloud-deployer-local.

Hi @sabbyanandan,

Hope you understand the requirement I explained.

from spring-cloud-deployer-local.

Hi @udaraliyanage I am having the same issue as yours, basically to limit the number of concurrent tasks.
Please let me know how did you resolve this.
We are trying to deploy the spring batch into Spring cloud data server

from spring-cloud-deployer-local.

Hi Raj,

Sorry for the late reply. Yes I was able to resolve this issue.

I defined a parameter which defines maximum number of jobs that can be run in parallel. The launcher keep track of tasks/jobs running and wait accepting new job requests if there are already running jobs equal to defined max.

from spring-cloud-deployer-local.

Hi @udaraliyanage Do you have the code sample for the same. Where did you put this code for validating this. Please let me know

from spring-cloud-deployer-local.

I think for local deployer, this is a good approach. It would limit concurrent executing tasks that were invoked via some synchronous invocation mechanism, e.g. a REST endpoint in spring cloud dataflow.

Note, that there is a related issue we are solving regarding the Stream TaskLauncher. The use case there is that a message is what triggers the invocation of a task. We introduced the ability for a spring cloud stream application to poll for new messages, this way it can poll when the number of tasks goes below a certain threshold.

from spring-cloud-deployer-local.

@sabbyanandan Kindly update.

from spring-cloud-deployer-local.

Hi.

So I think all the functionality you want is in place and will be released shortly.

• The number of running tasks is provided by this commit
• Data Flow concurrent task execution limit is provided in this commit
• A sink that checks the task execution limits is provided here

There is a separate issue relating to cleaning up the task PIDs in the local deployer but that isn't critical to the functionality you were requesting. I'll close the issue now. Feel free to open up another one if this approach is not meeting your needs/use-case.

from spring-cloud-deployer-local.