Workflows often throw errors and get stuck about dapr HOT 13 OPEN

@odzhurik

Right now Dapr Workflows is not stable when running the Workflow with more than 1 instance of the App, as it causes dead-locking on the underlying Actor Reminder sub system. This is likely what you're hitting against.

The good news is that this is a well known issue, and maintainers & contributors have been actively working for several months on a new sub-system which will replace the current poor performing Actor Reminder subsystem. The preview release of the new sub-system will be in the next dapr version, which will likely release next month (June) - Keep an eye out for the 1.14 release :)

from dapr.

I tried with 1.14.0 rc2 but the error still occurs.
I tested with 2 and 3 application instances and with Redis and Microsoft SQL Server state stores but without success.
With Redis I get
error saving reminders partition and metadata: ERR Error running script (.....): @user_script:14: user_script:14: failed to set key actors

and with MS SQL I get

error saving reminders partition and metadata: mssql: Error number 2601 in the THROW statement is outside the valid range.

Does the error occur with any state store type?
Does anyone know when this will be fixed?

from dapr.

@NunzioCar

Have you enabled the new SchedulerReminders preview feature which forces the Actor reminder system to use the new scheduler control plane service? You must do this if you want to try horizontal scaling of workflow App instances.

https://v1-14.docs.dapr.io/operations/support/support-preview-features/

from dapr.

@olitomlinson thanks for the suggestion.
I hadn't enabled the new SchedulerReminders preview feature. I'll try it.

Have you an example of the scheduler configuration in a self-hosted enviroment (docker-compose).
I suppose that i should add the daprio/scheduler container and configure the sidecar.

from dapr.

@olitomlinson thanks for the suggestion. I hadn't enabled the new SchedulerReminders preview feature. I'll try it.

Have you an example of the scheduler configuration in a self-hosted enviroment (docker-compose). I suppose that i should add the daprio/scheduler container and configure the sidecar.

We are working on enabling scheduler in the latest 1.14 CLI, I'll keep you posted here once that's done so you can try it out

from dapr.

@olitomlinson thanks for the suggestion.

I hadn't enabled the new SchedulerReminders preview feature. I'll try it.

Have you an example of the scheduler configuration in a self-hosted enviroment (docker-compose).

I suppose that i should add the daprio/scheduler container and configure the sidecar.

Yep, I've got a compose file with 3 instances which works fine :)

Check it out here

https://github.com/olitomlinson/dapr-workflow-examples/blob/main/compose-3-instances.yml

from dapr.

@yaron2 @olitomlinson great, it works fine.
Thanks a lot, you saved me!

There are no errors and the workflows was completed successfully but I see a loss of performance:

with a workfow that call other child workflows and with 1000 parallel workfow executions
the system complete the workflows in

• ~2min with 1 application instance and without scheduler
• ~5min with 1 application instance and with scheduler
• ~5min with 2 application instance and with scheduler
• ~5min with 3 application instance and with scheduler

So with the scheduler it is slower and the application doesn't scale.
Should I scale the scheduler?

from dapr.

@yaron2 @olitomlinson with 1.14 rc5 there are no errors and there is a performance improvement; it's like the version without scheduler, ~2min instead of ~5min.
Great job,
thanks

from dapr.

@NunzioCar Thanks for confirming!

This matches my testing as well (RC5 has increased the speed greatly, compared to previous RCx)

from dapr.

@NunzioCar

I see you were testing with 1000 parrallel workflows? Can you roughly describe what each workflow instance is doing? i.e. is it fanning out to multiple Activities and then doing a whenall ? if so how many activities, and how long does each activity take to produce a result?

Also, what environment were you testing this in? Docker compose? Kubernetes on your dev machine? Kubernetes on a real cluster? If a real cluster, how many nodes?

Also, what Actor state store were you using? i.e Redis? Postgres? and was that running locally on your machine? or a cloud service like AWS RDS / Azure SQL etc?

from dapr.

@olitomlinson it's a real business case (I'm taking a risk by using dapr workflow that is in beta version), the workflow contains 3 child-workflow levels with activities that manage files, activities that read & write into DB, activities that invoke remote services etc. It's not a workflow for a benchmark purpose.

Anyway there are 2 child workflow that execute 100 child workflows in parallel (await Task.WhenAll(parallelTasks)), for 10 times.

I tested this workflow with Visual Studio 2022 with docker-compose. Dapr was not configured in HA mode and I used Redis as state store, configured into the docker-compose.

from dapr.

Hey @NunzioCar - Can you test again now that 1.14.1 is out? There's been a lot of positive change since rc5 :)

from dapr.

@olitomlinson thanks, I will test it as soon as possible.

from dapr.