Pass through dask-worker configuration options about dask-drmaa HOT 3 OPEN

I'll answer the last question first.

Like with #4 , this can also get a bit site-specific. By default, every scheduler I know of will only request one core by default. Sometimes that will be restricted to one actual core via cgroups or taskset and other times it will just be a gentleman's agreement that if you're requesting a core, you'll only use a core and nothing will actually prevent you from using all the cores on a box.

With Grid Engine and its derivatives, you request more than one core with a "parallel environment". The common ones are named, by convention (but by no means universally) "smp" and "mpi". These are also requested with template.nativeSpecification (nativeSpecification = "-pe smp 10" will for instance give you 10 cores on the same host. -pe mpi 10 will give you 10 cores, but not guaranteed to be on the same host and the process must be started with MPI. that's likely out of scope for this project. :)

Given that 1-core non-parallel-environment jobs will likely be scheduled fastest, the easiest thing to do is probably to schedule as many of those as possible in an array job (as DRMAACluster seems to be doing) and only use a single thread/core.

from dask-drmaa.

OK, using a single core by default seems like the sane default choice, perhaps coupled with some release valve to let people pass through their own specific options.

from dask-drmaa.

Have opened issue ( #66 ) to discuss more how resources could be configured by checking the scheduler implementation with DRMAA and thus passing through the appropriate flags with such settings. We can also forward this information to the dask-worker running in that job so that it is aware what the scheduler job was configured with.

from dask-drmaa.