danielgerlag / conductor Goto Github PK

In workflows that allow the SelectNextStep key (the Decide step type below for example):

Issue:
One can see that BSON has an issue with the key names that aren't str or int types.

I've only briefly looked at the documentation, but noticed that it seems like all the examples of workflows have a very linear flow...

Does there already exist the ability to say wait until we get a decision... and if yes do this chain of steps, otherwise do that chain of steps?

We refuse to use MongoDB. I understand that a document store is probably a natural fit for this but there are other options, such as PostgreSQL JSONB field.

Any chance at using other data provider such as NPGSQL?

Hi,

I have the following workflow definition:

{
    "Id": "AutomaticReply",
    "Steps": [
        {
            "Id": "OnMoReceived",
            "StepType": "While",
            "Inputs": {
                "Condition": "1"
            },
            "Do": [[
                {
                    "Id": "WaitForMo",
                    "StepType": "WaitFor",
                    "Inputs": {
                        "EventName": "\"MoReceived\"",
                        "EventKey": "\"MOIN\"",
                        "EffectiveDate": "DateTime.Now"
                    },
                    "Outputs": {
                        "Mo": "step.Data"
                    },
                    "NextStepId": "SendMt"
                },
                {
                    "Id": "SendMt",
                    "StepType": "HttpRequest",
                    "Inputs": {
                        "BaseUrl": "http://localhost:5006/api/v1/test/",
                        "Resource": "message",
                        "Method": "POST",
                        "Headers":{
                            "@Authorization": "Bearer xxx",
                            "@Content-Type": "application/json"
                        },
                        "Body": {
                            "destination": "data.Mo.Source",
                            "text": "data.Mo.Text"
                        }
                    }
                }
            ]]
        }
    ]
}

When I start this workflow (I'm using Conductor) using the http request:

curl --location --request POST 'http://localhost:5000/api/workflow/AutomaticReply' \
--header 'Content-Type: application/json' \
--data-raw '{
    
}'

I get the following exception

fail: WorkflowCore.Services.WorkflowExecutor[0]
      Workflow 5f5889a91cf3171bf042a51b raised error on step 1 Message: name 'DateTime' is not defined
IronPython.Runtime.UnboundNameException: name 'DateTime' is not defined
   at IronPython.Runtime.Operations.PythonOps.GetVariable(CodeContext context, String name, Boolean isGlobal, Boolean lightThrow)
   at IronPython.Compiler.LookupGlobalInstruction.Run(InterpretedFrame frame)
   at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame)
   at Microsoft.Scripting.Interpreter.LightLambda.Run2[T0,T1,TRet](T0 arg0, T1 arg1)
   at IronPython.Compiler.PythonScriptCode.RunWorker(CodeContext ctx)
   at IronPython.Compiler.PythonScriptCode.EvalWrapper(CodeContext ctx)
   at IronPython.Compiler.PythonScriptCode.Run(Scope scope)
   at IronPython.Compiler.RuntimeScriptCode.InvokeTarget(Scope scope)
   at IronPython.Compiler.RuntimeScriptCode.Run(Scope scope)
   at Microsoft.Scripting.SourceUnit.Execute(Scope scope, ErrorSink errorSink)
   at Microsoft.Scripting.SourceUnit.Execute(Scope scope)
   at Microsoft.Scripting.Hosting.ScriptSource.Execute(ScriptScope scope)
   at Conductor.Domain.Scripting.ScriptEngineHost.EvaluateExpression(String expression, IDictionary`2 inputs) in C:\Users\conductor\src\Conductor.Domain.Scripting\ScriptEngineHost.cs:line 33
   at Conductor.Domain.Services.ExpressionEvaluator.EvaluateExpression(String sourceExpr, Object pData, IStepExecutionContext pContext) in C:\Users\conductor\src\Conductor.Domain\Services\ExpressionEvaluator.cs:line 22
   at Conductor.Domain.Services.WorkflowLoader.<>c__DisplayClass12_0.<BuildScalarInputAction>g__acn|0(IStepBody pStep, Object pData, IStepExecutionContext pContext) in C:\Users\conductor\src\Conductor.Domain\Services\WorkflowLoader.cs:line 258
   at WorkflowCore.Models.ActionParameter`2.Assign(Object data, IStepBody step, IStepExecutionContext context)
   at WorkflowCore.Models.ActionParameter`2.AssignInput(Object data, IStepBody body, IStepExecutionContext context)
   at WorkflowCore.Services.WorkflowExecutor.ExecuteStep(WorkflowInstance workflow, WorkflowStep step, ExecutionPointer pointer, WorkflowExecutorResult wfResult, WorkflowDefinition def)
   at WorkflowCore.Services.WorkflowExecutor.Execute(WorkflowInstance workflow)

How do I set DateTime.Now in json? I'm following the syntax given here.

There may be multiple datetime formats (even within ISO 8601) that other processes operating in other languages pass to these workflows.

Notably, Javascript typically returns something like: 2020-03-24T00:47:49.8090000Z for a datetime object.

The docs
(https://docs.python.org/3/library/datetime.html#datetime.datetime.fromisoformat)
suggest dateutil to gain the functionality of arbitrary ISO 8601 datetime parsing from dateutil.parser.isoparse

Hi, Daniel!

I didn't know how to message you this request, that's why I opened it as an issue. Feel free to close it if this is not the right place.

Would you consider opening a Discord channel for people to discuss/ask Conductor and Workflow-Core questions? Or it is too much/you don't have time? It may be a nice addition to Github page, and give Workflow-Core community a place for asking silly questions / get feedback quicker :P.

Thank you!
Razvan

Love your work, We are planning to develop something in the like of 5k different workflows, just like to if conductor will support this, keeping in mind that 5K workflow might be running with 50 instance of each workflow at any given time, so 5kx50=250000 instances ?

Moreover, just like to know if conductor support Multi-node clusters.

Hello,
I really appreciate your work and you have a great project.

I read the documentation and I can't find a solution on how to call external micro-service (worker) from the conductor. Yes, I can check "my-activity" but if I have a lot of workers? In this case I think it was a performance issue.
I would like to understand the correct architecture before diving in.
Thank you in advance!

Hi, I'm beginner and I'd like to ask a question.
Whether there is a way to send data from one step to another?
For example, in one step I do HttpRequest and I get JSON record. I want to pass JSON data to next step, and do some kind of data manipulation inside workflow, without REST service. Is this possible?
Gratefully, Stefan

I noted that workflow.core supports error handling: https://workflow-core.readthedocs.io/en/latest/error-handling/

However from conductor docs under Error Handling it says "TODO": https://conductor-core.readthedocs.io/en/latest/primitives/

Would you be able to clarify whether conductor is capable of supporting error handling?

Thanks for this Daniel, will build some really cool stuff on it!
In order to do that though, see subject: can we call our own custom scripts that may need their own Python package imports?

Following this tutorial https://conductor-core.readthedocs.io/en/latest/custom-steps/

and got

System.ArgumentException: Step type Lambda not found

"StepType: Lambda" should be "StepType: add" I guess

On the issue of using getpendingactivity in activitycontroller, I can't get the activity. I refer the source code to debug and find that the source code is passed in Event.EventTypeActivity This value is always WorkflowCore.Activity , rather than the activity I want to query

How does one set the schedule interval? I've tried multiple formats available here: https://docs.microsoft.com/en-us/dotnet/api/system.timespan?view=netframework-4.8#code-try-4

and still no luck. What exactly am I doing wrong?

@danielgerlag @willsmith9182

Hi Daniel, Just like to say thank you for creating such an awesome package.

We have built a UI that wraps around the workflow core engine (basically replicating what Conductor does). However, we don't persist any of our definitions to a store instead we just build them using the Fluent API and they are stored in memory at runtime.

We're hoping that we can do the opposite of what this method does and map a definition from the registry to the domain model.

I've started doing a POC but were just wondering if you have already done this in a feature branch somewhere.

Any help will be greatly appreciated

Thanks,
Paul

Hey,

Does Conductor comunicate with another workflow servers to just build a workflow instance?
From what I understood Conductor needs to know every step that is going to be executed by a workflow (the step must be coded inside the server). This leads me to every new step to re-deploy the server.

Thank you.

Describe the bug
For a "Schedule" step, the expression for "Interval" created through Json/Yaml cannot be evaluated.

To Reproduce
An exception is thrown when we use "datetime.timedelta(seconds=300)" as the "Interval" input.

Additional context
The scenario is using the REST API for creating a workflow definition by passing a JSON representation; afterwards, the "Workflow" API is used to start/create a workflow. The exception is thrown when the input expression for a schedule step is evaluated.

If I am wrong on defining the "Interval" expression, please correct me. Thanks for your great work.

Quick question regarding how to set the recur interval. In workflowcore we used timespan which is defined in .net and seems to be the object type workflowcore expects. How does this work with conductor and iron python script engine?

@danielgerlag
Attempt:

Issue:
I've tried both syntax in both StepTypes. Neither works, they both return the error below.

Requested Behavior:

Some documented way to bind ActivityName ,EventData, and Parameters to data properties.

Love the project and am building out a simple workflow right now and need to use the Parallel task in the docs. Every time I just and upload the definition form your samples on the Parallel task I get the message that Parallel step is not found. Any known issues with this?

Hi, I created a workflow definition using the following call:

curl --location --request POST 'http://localhost:5000/api/Definition' \
--header 'Content-Type: application/json' \
--data-raw '{
    "Id": "AutomaticReply",
    "Steps": [
        {
            "Id": "OnMoReceived",
            "StepType": "While",
            "Do": [[
                {
                    "Id": "WaitForMo",
                    "StepType": "WaitFor",
                    "Inputs": {
                        "EventName": "MoReceived",
                        "EventKey": "MOIN",
                        "EffectiveDate": "DateTime.Now"
                    },
                    "Outputs": {
                        "Mo": "step.Data"
                    },
                    "NextStepId": "SendMt"
                },
                {
                    "Id": "SendMt",
                    "StepType": "HttpRequest",
                    "Inputs": {
                        "BaseUrl": "http://localhost:5006/api/v1/send/",
                        "Resource": "message",
                        "Method": "POST",
                        "Headers":{
                            "@Authorization": "Bearer XXX",
                            "@Content-Type": "application/json"
                        },
                        "Body": {
                            "destination": "data.Mo.Source",
                            "text": "data.Mo.Text"
                        }
                    }
                }
            ]]
        }
    ]
}'

And I received the expected 204 response. I can see the workflow definition saved to the database. Then I started the workflow using

curl --location --request POST 'http://localhost:5000/api/workflow/AutomaticReply' \
--header 'Content-Type: application/json' \
--data-raw '{
    
}'

I received the response as expected:

Data: {}
DefinitionId: AutomaticReply
EndTime: null
Reference: null
StartTime: 2020-09-09T07:21:31.8100000Z
Status: Runnable
Version: 1
WorkflowId: 5f58827ba91412682c7ba41b

Then, I wanted to check the status of this workflow. Here, something strange happens. I expected the workflow to be running, but the status I got back was that it was completed. My curl request:

curl --location --request GET 'http://localhost:5000/api/workflow/5f58827ba91412682c7ba41b'

And the response I received was

Data: {}
DefinitionId: AutomaticReply
EndTime: 2020-09-09T07:21:31.9070000Z
Reference: null
StartTime: 2020-09-09T07:21:31.8100000Z
Status: Complete
Version: 1
WorkflowId: 5f58827ba91412682c7ba41b

In any case, I tried raising the event using the http request:

curl --location --request POST 'http://localhost:5000/api/event/MoReceived/MOIN' \
--header 'Content-Type: application/json' \
--data-raw '{
    "Destination": "xxxxxxxxx",
    "Source": "xxxxxxxx",
    "Text": "Hello, world!"
}'

To which I got back the 204 response.

However, I did not receive any request as expected to http://localhost:5006/api/v1/send/message. I don't see any errors on the console. In the database, I see the entry:

{
    "_id": "5f5882a7a91412682c7ba41c",
    "EventName": "MoReceived",
    "EventKey": "MOIN",
    "EventData": {
        "Destination": "xxxxxxx",
        "Source": "xxxxx",
        "Text": "Hello, world!"
    },
    "EventTime": {
        "$date": {
            "$numberLong": "1599636135337"
        }
    },
    "IsProcessed": true
}

What am I doing wrong? Thanks.

I am curious whether this project was inspired by https://github.com/Netflix/conductor ?
If not, you might want to visit that project as it shares a few things in common but seems more evolved on the UI side (given netflix's vast resources).

However I prefer the look of this project as it uses a simple tool chain (and libraries) more familiar to me as a dotnet developer which is a big bonus.

Hi,

I like your workflow core project, But i want to extend to execute an API actions , and that workflow can listen to the fs, ampq, db or emails triggers.

Any ideas how can I begin, I have fork the workflow core. But I am thinking starting ideas.

Daniel,

Thanks a lot for this system, very informative!

I'm following the Quick Start tutorial, in examples one and two, I don't see any Emit Message Responses. How does one pipe outputs to API responses and stdout while we're at it?

Is there a way for Conductor to obtain the IP address of the server running its container?

This is not the IP of the container but the server running it.

just wondering are the any ideas building a frontend?

I am thinking about building something with https://retejs.org/examples

hi @danielgerlag , I'm building a workflow which has it's own custom definitions, I was able to test my workflow using workflow core directly using a console app. I see that conductor has its own script engine to validate the workflow and expecting the expression to be in python.

When I post my workflow definition to conductor Post api, it is not able to find my custom step types, hence I have made the below changes and it worked. Is there any issue in the code? or Is this feature is not supported yet?