Questions about the input vector about decima-sim HOT 2 OPEN

Thanks for your questions.

1. The scheduler needs to know how many available executors are at disposal when making the scheduling decision. In other words, when the number of available executors are different, the optimal scheduling decision might be different. For example, when there is only 1 executor available, decima will have to prioritize the executor to the most important node. But if there are 50 free executors, decima might schedule executors to two parent stages in a job (i.e., when executors are too few, only running one parent stage can be suboptimal).

2. We associate the parallelism to DAG mainly because we want to reduce the problem complexity. Our paper section 5.2, parallelism limit section, paragraph "Decima’s action specifies job-level parallelism, as opposed fine-grained stage-level parallelism...." explains this point in more details.

Hope these help!

from decima-sim.

Well explained.

I just have a question regarding #1, You said Decima will have to prioritize the executor to the most important node => My question here is, How does it really prioritize the executor to do such a job? For example: I have multiple jobs in the DAG and some are free or no dependency, now how I can able to teach the patterns of the node to communicate with the executor to complete the job and in an optimized way (I guess here ML concept is involved). Is there any special parameter you would like to add so I can have a good understanding of it? Also which part of the code actually allows DAG to do such operation with the GNN, how it's actually triggered here in the GNN part from the DAG job state.

Another thing, so far what I have found and understand, I just want to share and get your opinion, Is it possible to create DAG without RDD in the spark env, I was trying to figure it out but did not get any better explanation here, would you like add some points here.

Again thank you so much and look forward to getting some valuable information from you.

from decima-sim.