PBS scheduler errors with qsub submission: && in bash command about flowr HOT 7 CLOSED

I checked, for some reason the memory argument was empty:

#PBS -l nodes=1:ppn=1
#PBS -l walltime=00:05:00
#PBS -l mem=

fobj=readRDS("flow_details.rds")
fobj@jobs[[1]]@memory
character(0)

Can you check the flow_definition file it it has memory in the right format.

from flowr.

flow_def needs the following column names:

memory_reserved: amount of memory reserved
cpu_reserved: number of CPU reserved

It is a attempt to differentiate this from:

cpu_used

For example, one may reserve a node with 12 cores (cpu_reserved), but would like to use only 6 (cpu_used) due to memory restrictions.

In future, would like to have both columns.

from flowr.

Yea, may be memory is shorter than memory_reserved.

I have added this:
513263c

from flowr.

Ok, so the second issue was with rendering.

• Update the package
• replace {{ with {{{ as shown in: f7bf959 in your torque.sh

I have tested this:
https://github.com/sahilseth/flowr/blob/master/inst/extdata/example1_flow_mat.txt

from flowr.

Thanks and commit: f7bf959 is helpful to set up job scheduler config for more than one compute clusters.

Two minor issues:

• shell environment seems not passing correctly to compute nodes.

This requires me to hard code my individual commands and prefix module load <module> to make sure env has path to all required binaries and libraries. I see that my torque.sh has #PBS -r y -V flag. Not sure if -V can be supplied with array index or should be a new line.

• At line 3 of torque.sh, does #PBS -l nodes={{{NODES}}}: takes character vector? Although not intuitive, one of our compute nodes needs character and not numeric vector for number of nodes! If I supply character vector in flow_def.txt under column: nodes, I see following in flowr submitted commands:

./002.make_config/make_config_cmd_1.sh:3:#PBS -l nodes=1:ppn=2
./003.chr/chr_cmd_16.sh:3:#PBS -l nodes=1:ppn=2
...

Thanks

from flowr.

Yes character vector is now supported. Can you try this with a column nodes instead of nodes_reserved.

I am going to make this simpler, for now this is what we have:

*: These are generated on the fly
**: This is gathered from flow_mat

from flowr.

Hi @dyndna, I personally was not using/changing nodes variable, and seems had dropped it out of to_flow() function.

I can confirm that it works now using this example:

exdata = file.path(system.file(package = "flowr"), "extdata")
flow_mat = read_sheet(file.path(exdata, "example1_flow_mat.txt"))
flow_def = read_sheet(file.path(exdata, "example1_flow_def2.txt"))
fobj = suppressMessages(to_flow(flow_mat, def = flow_def, platform = "torque"))
fobj[[1]]@jobs[[1]]@nodes
fobj = submit_flow(fobj)

Sorry for the confusion.

links:
def: https://github.com/sahilseth/flowr/blob/master/inst/extdata/example1_flow_def2.txt
change in to_flow: db0d3c6?diff=split

from flowr.