HPC about targets HOT 6 CLOSED

Should wrap schedulers in classes so external folks can write their own.

from targets.

Ideally we should use the same master process, just have an hpc abstract class to submit and conclude targets.

from targets.

First order of business: add an HPC option to settings to let users decide whether to run on workers or master.

from targets.

• Make a new "preclustermq" algorithm to pre-run all the local targets before deployment to hpc. Nope. Just repurpose local algo. Implement an option to throw a custom condition to quit early from run_target() if the deployment setting is "remote".
• In the clustermq algorithm itself, implement a local_first argument of run() to optionally invoke the above. Test skipping targets with and without that option.
• In the first test (all local targets) do not set any clustermq options.
• Test on a plan where the first few targets are up to date and the last ones are not. (Get going first with local algo.)
• Test on a plan with nontrivial common data (e.g global functions)
• Repeat existing clustermq tests on a real cluster. Test with tests in tests/hpc/clustermq.R. Includes Keras model serialization.

from targets.

Okay, just need to run the last couple tests on SGE.

from targets.

So close yet so far. It's hanging on receive_data() when there is more than one worker.

from targets.