What about cross-server data transmission overhead? about decima-sim HOT 2 OPEN

I agree that data locality is an important aspect to optimize. Our simulator didn't capture it explicitly because the particular workload we run on Spark did not show much difference (all VMs are in a single datacenter, where the large network throughput makes this locality issue minimum).

However, I would say it shouldn't be hard to add the transmission time in the simulator. You can create a multiplier on the task run time based on parent and child node.

Also, for RL, you might want to still optimize directly for the end-objective as opposed to divide the goal into sub-goals and optimize them individually. It might be difficult to hand-tune the balance between execution time and cross-node communication overhead.

Hope these help!

from decima-sim.

Thanks! This helps a lot!

from decima-sim.