Add DAG support and fix inconsistent naming about obsei HOT 6 CLOSED

@shahrukhx01 When I created this ticket and now I encountered many different ways to solve it. Each have some pros and some cons. Ideally I would like to build this feature such that we can plug any framework, because it directly linked to production deployment. Currently our target to build good user base who try Obsei and once it build then we can think about it that is the reason for lowest priority.
Few things we considered -

1. Ray
2. temporal
3. airflow
4. networkx
5. containers (similar to what Jina does)
6. Loky for local parallelism
7. Pathos for local level parallelism
8. Apache Beam (Pipeline IO concept can be borrowed) https://beam.apache.org/documentation/io/built-in/

If you have any suggestion regarding how we can abstract it then it would be great.
I am adding @GirishPatel to this conversation as well.

from obsei.

I see, I will explore orthe options and add to the alternatives you mentioned. Maybe before that, I can first finish #75 and focus on medium/high priority issues first then.

from obsei.

Our aim to build NLP based open source workflows like https://github.com/n8n-io/n8n
Currently we supporting text but there is plan to support image and audio based workflows.
We are very far from it. But hoping one day we can achieve this target 😅🐢

from obsei.

@lalitpagaria for this point "Introduce DAG based workflow. Need to finalise between networkx or airflow" please do let me know when you plan to start this implementation, as I'm seriously interested in working on this feature, as it'd also help me understand Haystack's codebase. thanks!

from obsei.

eventually, we'll get there!

from obsei.

Closing this and moving discussion to #145

from obsei.