Backends for Platform LSF, Oracle Grid Engine about distributed-process HOT 3 CLOSED

Hi David,

The core Cloud Haskell infrastructure (distributed-process) does not provide any functionality for node discovery or node creation. You can spawn processes to, and monitor, nodes with arbitrary node IDs (NIDs). As such, distributed-process certainly makes no assumptions about the dynamics of the network topology.

The origin of those node IDs is the province of Cloud Haskell backends. So it is conceivable that some backends might assume a static network topology, whereas others might work with dynamic network topologies. At the moment there is only a single backend available, called distributed-process-simplelocalnet (and we are working on an Azure backend). This provides a findPeers function which uses a UDP broadcast to discover peer nodes -- hence, it too does not assume that the network is static. I am therefore a bit surprised that you got the impression Cloud Haskell assumes a network topology. If you can tell me what part of the documentation gave you this impression, I'd appreciate it so that I can fix it.

A major reason for separating the Cloud Haskell core from the Cloud Haskell backend on one side and from the underlying network transport (as abstracted by network-transport) is precisely so that it would be possible to design backends for specific infrastructures such as the one you sketch above.

So although Cloud Haskell as it stands doesn't provide specific functionality for the kind of things you enumerate, it won't get in your way either, and the architecture is expressly designed to support the design of such backends. I would suggest to have a look at the distributed-process-simplelocalnet backend. It is only small, and it might serve as a good starting point. If you have questions, feel free to ask me -- here, at [email protected] or on #haskell.

from distributed-process.

Hi Edsko,

Thanks for the response. I think I was thrown off by the type signature

startMaster :: Backend -> ([NodeId] -> Process ()) -> IO ()

where the NodeId list is determined once and for all just before starting the master process. This, combined with my recollections from reading the original Cloud Haskell paper, made me think that the set of slaves could not be added to during the running of the program.

After reading the source for distributed-process-simplelocalnet, it's clear that this is not the case. startMaster is not right for my purposes, but it looks like the more general features of distributed-process-simplelocalnet are already pretty close to what I want. In particular, the list of slaves can grow at any time as the discoveryDaemon receives new messages. I guess I need to add some functionality for pruning the list of slaves when processes get killed or suspended. I think I can figure this out.

Thanks,
David

from distributed-process.

I see. I have added the following comment to 'startMaster':

-- If you start more slave nodes after having started the master node, you can
-- discover them with later calls to 'findSlaves', but be aware that you will
-- need to call 'redirectLogHere' to redirect their logs to the master node.  

Hopefully that will avoid confusion. I'm glad that you think distributed-process-simplelocalnet might be a good starting point; as I mentioned before, if you get stuck somewhere, feel free to ask me any questions you might have.

from distributed-process.