codaxy / conductor-sharp Goto Github PK

These could be a very useful endpoints to use.

Following expression is not supported:

InputProperty = $"{wf.Task.Output.PrimitiveProperty}"

where PrimitiveProperty has primitive C# type(bool, int, ...)

In the form of additional builders.

Some users might want to use System.Text.json and some might want to use Newtonsoft.Json. We should support both and have it be configurable by the user.

Instead of having to register workflows and all their tasks manually (error-prone for large workflows), it could be useful to add support for automatic worker and sub-workflow registration. The user would then register only the workflows they need, and all of its workers and sub-workflows would be discovered via reflection and registered behind the scenes.

This feature can be optional, so something like this:

builder.RegisterWorkflow<MyWorkflow>(opts => opts.RegisterTasksAutomatically());

As reported by @markor23 the semaphore Wait call in ExecutionManager.cs is missing (removed in a hurried refactor at some point) and therefore the intended limit on the number of concurrent workers is not being applied.

Noting here and waiting for PR.

There are endpoints that allow us to see polling information for tasks. This is useful to check if tasks are not being polled (workers might be offline).

The less libraries we depend on the better.

Quoting exising TODO:

// TODO: Implement load balancing strategy so we can avoid // task starvation. One way is to create some sort of task priority // which will dynamically change based on how many of those tasks we served // When a task is polled we reduce its priorty and when increse the priorty of all others

The intent here is to implement some smarter way of polling tasks. The way polling currently works is:

1. Check queue to see how many of which task is queued
2. Select task names implemented by microservice which have queue count > 0
3. Execute them concurrently (up to a limit)

This should equate to round robin polling by task name. Maybe we can assign priorities to tasks and use that, or select them based on how many are in queue, or select them at random, or something smarter.

There are no apparent issues with the current approach so this TODO can also just be cleared.

There are many scenarios where we only want to list registered workflows by name and version. It would be nice to have a quick and lightweight response for these.

Often there are workflows that differ only in some Workers and/or inputs but the general structure remains the same.

CSharpLambdaTaskModel is really annoying to use especially when it usually does mundane things such as string interpolation

Add support for retry, restart and rerun.

Conductor Sharp MetadataService UpdateWorkflowDefinition method doesn't match the conductor API signature.

This could be useful.

When there are conductor outages, this service will crash and workers will no longer respond. It would be helpful to be able to configure some sort of recovery strategy so that it can restart if conductor comes back up.

Quoting existing TODO from code:

// TODO: Implement polling strategy so that if there // are no requests incoming we poll less, and when queues are full // we poll more often

This basically means that when there is a high load we poll more often and when there is no load we extend the polling interval to be kind to conductor (as there are many microservices individually polling it).

I am not 100% sure this would make much of a difference as we cannot allot a too long polling interval (5s+) because new tasks would end up waiting a lot. We did notice in production scenarios that conductor is affected by heavy polling.

Expose more of the client library functionalities through the endpoints. For workflow task manipulation these are skip, resume and pause tasks.

At the moment workflow descriptions are stringified json. This allows us to make more complex descriptions, use labels and such, but it was not the intended use of the description field. It should by default be a string field. However, we should allow the user to retain the previous behavior somehow if they want.

It would be nice to have an injectable logger that writes logs to conductor task endpoint, in case we want to have Conductor as the main logging system.

UpdateWorkflowDefinition method parameter should be array of WorkflowDefinition

The http task is very useful and could lead to less writing of code. For starters we should add a simple builder which allows us to add these to workflows. More advanced features and use cases can be implemented later (as there can be quite a lot of complexity in dealing with requests and responses).