Comments (7)
leepc12
commented on August 23, 2024
Can you make a TAR packing contents of this directory ./atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/?
Please upload the following items. This will be very helpful for debugging.
- TAR ball of
./atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/ - Full
caper runcommand line. - Your
~/.caper/default.conf. - Your input JSON file (
-i). - Full
caper runscreen output (STDOUT).
from caper.
tanpuekai
commented on August 23, 2024
1, See attached tar-ball
for.jin.lee.tar.gz
2, The full caper run command is as this:
caper run \
-i input.json \
--tmp-dir /project1/TMPDIR4 \
--pbs-queue legacy \
../atac-seq-pipeline/atac.wdl
3, The ~/.caper/default.conf is:
$ cat ~/.caper/default.conf
backend=pbs
4, See attached tar-ball: for.jin.lee.tar.gz
5, See attached tar-ball: Message_encod4_335233.omics.txt for.jin.lee.tar.gz
Thanks.
Best
Chan
from caper.
leepc12
commented on August 23, 2024
Pipeline failed at the first task read_genome_tsv.
The following is STDERR from the failed task but I couldn't find anything related to the pipeline code or Caper.
-bash: inspectXeption: line 0: syntax error near unexpected token `;'
-bash: inspectXeption: line 0: `inspectXeption () { echo "***checking exceptions***"; for i in `ls *reeting*`; do; cnt=`cat $i|grep ception|wc -l`; if [ $cnt -gt 0 ]; then; echo $i ' ' $cnt; fi; done; echo ""; echo "***checking unfinished jobs***"; for i in `ls *reeting*`; do; cnt=`cat $i|grep "Total time"|wc -l`; if [ $cnt -eq 0 ]; then; echo $i ' ' $cnt; fi; done; }'
-bash: error importing function definition for `BASH_FUNC_inspectXeption'
-bash: waitqueue: line 0: syntax error near unexpected token `;'
-bash: waitqueue: line 0: `waitqueue () { while true; do; sleep 10; cnt1=`qstat -rn1|grep USER1|wc -l`; if [ $cnt1 -gt 0 ]; then; echo $cnt1; else; break; fi; done; }'
-bash: error importing function definition for `BASH_FUNC_waitqueue'
-bash: sedline: line 0: syntax error near unexpected token `;'
-bash: sedline: line 0: `sedline () { for i in $(seq 1 10); do; sed -i 's/ / /g' $1; done; sed -i 's/^ //g' $1; sed -i ':a;N;$!ba;s/\n /\n/g' $1; sed -i ':a;N;$!ba;s/ \n/\n/g' $1; sed -i 's/ /\t/g' $1; sed -i 's/\tCHR/CHR/g' $1; sed -i 's/^\t//g' $1; }'
-bash: error importing function definition for `BASH_FUNC_sedline'
Can you run the following command and see what happens? This is a raw command line used by the pipeline to qsub this bash script /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/script for the failed task.
echo "/bin/bash /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/script" | qsub -N cromwell_21385ee3_read_genome_tsv -o /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/stdout -e /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/stderr -lselect=1:ncpus=1:mem=4000mb -lwalltime=1:0:0 -q legacy -V
from caper.
tanpuekai
commented on August 23, 2024
Hi Jin,
Ok. This what I did:
$cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/
$echo "/bin/bash script" | qsub -N cromwell_21385ee3_read_genome_tsv -o STDOUT -e STDERR -lselect=1:ncpus=1:mem=4000mb -lwalltime=1:0:0 -q small -V
$ ls -lt|head
total 96
-rw------- 1 USER1 group 1276 Nov 3 13:01 STDERR
-rw------- 1 USER1 group 0 Nov 3 13:01 STDOUT
-rw-r--r-- 1 USER1 group 2 Nov 3 12:58 rc.tmp
-rw-r--r-- 1 USER1 group 150 Nov 3 12:58 roadmap_meta
$ cat STDERR
-bash: inspectXeption: line 0: syntax error near unexpected token `;'
-bash: inspectXeption: line 0: `inspectXeption () { echo "***checking exceptions***"; for i in `ls *reeting*`; do; cnt=`cat $i|grep ception|wc -l`; if [ $cnt -gt 0 ]; then; echo $i ' ' $cnt; fi; done; echo ""; echo "***checking unfinished jobs***"; for i in `ls *reeting*`; do; cnt=`cat $i|grep "Total time"|wc -l`; if [ $cnt -eq 0 ]; then; echo $i ' ' $cnt; fi; done; }'
-bash: error importing function definition for `BASH_FUNC_inspectXeption'
-bash: waitqueue: line 0: syntax error near unexpected token `;'
-bash: waitqueue: line 0: `waitqueue () { while true; do; sleep 10; cnt1=`qstat -rn1|grep USER1|wc -l`; if [ $cnt1 -gt 0 ]; then; echo $cnt1; else; break; fi; done; }'
-bash: error importing function definition for `BASH_FUNC_waitqueue'
-bash: sedline: line 0: syntax error near unexpected token `;'
-bash: sedline: line 0: `sedline () { for i in $(seq 1 10); do; sed -i 's/ / /g' $1; done; sed -i 's/^ //g' $1; sed -i ':a;N;$!ba;s/\n /\n/g' $1; sed -i ':a;N;$!ba;s/ \n/\n/g' $1; sed -i 's/ /\t/g' $1; sed -i 's/\tCHR/CHR/g' $1; sed -i 's/^\t//g' $1; }'
-bash: error importing function definition for `BASH_FUNC_sedline'
/usr/bin/script: /usr/bin/script: cannot execute binary file
$
BTW, the contents of the file script is as follows:
$cat script
#!/bin/bash
cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution
tmpDir=$(mkdir -p "/project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/tmp.4b17423c" && echo "/project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/tmp.4b17423c")
export _JAVA_OPTIONS=-Djava.io.tmpdir="$tmpDir"
export TMPDIR="$tmpDir"
export HOME="$HOME"
(
cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution
)
out53879f59="${tmpDir}/out.$$" err53879f59="${tmpDir}/err.$$"
mkfifo "$out53879f59" "$err53879f59"
trap 'rm "$out53879f59" "$err53879f59"' EXIT
tee '/project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/stdout' < "$out53879f59" &
tee '/project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/stderr' < "$err53879f59" >&2 &
(
cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution
# create empty files for all entries
touch genome_name
touch ref_fa bowtie2_idx_tar bwa_idx_tar chrsz gensz blacklist blacklist2
touch custom_aligner_idx_tar
touch ref_mito_fa
touch bowtie2_mito_idx_tar bwa_mito_idx_tar custom_aligner_mito_idx_tar
touch tss tss_enrich # for backward compatibility
touch dnase prom enh reg2map reg2map_bed roadmap_meta
touch mito_chr_name
touch regex_bfilt_peak_chr_name
python <<CODE
import os
with open('/project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/inputs/1245739974/mm10_caper.local.tsv','r') as fp:
for line in fp:
arr = line.strip('\n').split('\t')
if arr:
key, val = arr
with open(key,'w') as fp2:
fp2.write(val)
CODE
) > "$out53879f59" 2> "$err53879f59"
echo $? > /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/rc.tmp
(
# add a .file in every empty directory to facilitate directory delocalization on the cloud
cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution
find . -type d -exec sh -c '[ -z "$(ls -A '"'"'{}'"'"')" ] && touch '"'"'{}'"'"'/.file' \;
)
(
cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution
sleep 30 && sync
)
mv /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/rc.tmp /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/rc
$
Thanks.
Best
Chan
from caper.
tanpuekai
commented on August 23, 2024
Hi Jin,
Seems the problem is in the directive -V in your command:
"/bin/bash script" | qsub -N cromwell_21385ee3_read_genome_tsv -o STDOUT -e STDERR -lselect=1:ncpus=1:mem=4000mb -lwalltime=1:0:0 -q small -V
I changed the file script to:
#!/bin/bash
#PBS -N cromwell_21385ee3_read_genome_tsv
#PBS -o STDOUT
#PBS -e STDERR
#PBS -lselect=1:ncpus=12:mem=40000mb
#PBS -lwalltime=24:0:0
#PBS -q medium
#PBS -V
#PBS -j oe
date
echo "Step 1"
cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution
tmpDir=$(mkdir -p "/project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/tmp.4b17423c" && echo "/project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/tmp.4b17423c")
export _JAVA_OPTIONS=-Djava.io.tmpdir="$tmpDir"
export TMPDIR="$tmpDir"
export HOME="$HOME"
(
cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution
)
out53879f59="${tmpDir}/out.$$" err53879f59="${tmpDir}/err.$$"
mkfifo "$out53879f59" "$err53879f59"
trap 'rm "$out53879f59" "$err53879f59"' EXIT
tee '/project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/stdout' < "$out53879f59" &
tee '/project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/stderr' < "$err53879f59" >&2 &
(
cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution
echo "Step 2"
# create empty files for all entries
touch genome_name
touch ref_fa bowtie2_idx_tar bwa_idx_tar chrsz gensz blacklist blacklist2
touch custom_aligner_idx_tar
touch ref_mito_fa
touch bowtie2_mito_idx_tar bwa_mito_idx_tar custom_aligner_mito_idx_tar
touch tss tss_enrich # for backward compatibility
touch dnase prom enh reg2map reg2map_bed roadmap_meta
touch mito_chr_name
touch regex_bfilt_peak_chr_name
echo "Step 3"
python <<CODE
import os
with open('/project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/inputs/1245739974/mm10_caper.local.tsv','r') as fp:
for line in fp:
arr = line.strip('\n').split('\t')
if arr:
key, val = arr
with open(key,'w') as fp2:
fp2.write(val)
CODE
) > "$out53879f59" 2> "$err53879f59"
echo "Step 4"
echo $? > /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/rc.tmp
(
# add a .file in every empty directory to facilitate directory delocalization on the cloud
cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution
find . -type d -exec sh -c '[ -z "$(ls -A '"'"'{}'"'"')" ] && touch '"'"'{}'"'"'/.file' \;
)
(
cd /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution
sleep 30 && sync
)
mv /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/rc.tmp /project1/second-bat-one-go/atac/21385ee3-a703-4a97-8a87-855e8941fa03/call-read_genome_tsv/execution/rc
date
echo finished
and then performed this:
$ qsub script
and in 1-2 minutes it is finished:
$ ls -lt [Ss]*|head
-rw------- 1 USER1 group 3714 Nov 3 13:51 STDOUT
-rw-r--r-- 1 USER1 group 2418 Nov 3 13:51 stdout
-rw-r--r-- 1 USER1 group 0 Nov 3 13:51 stderr
-rw-r--r-- 1 USER1 group 3642 Nov 3 13:50 script
$ cat STDOUT
-bash: inspectXeption: line 0: syntax error near unexpected token `;'
-bash: inspectXeption: line 0: `inspectXeption () { echo "***checking exceptions***"; for i in `ls *reeting*`; do; cnt=`cat $i|grep ception|wc -l`; if [ $cnt -gt 0 ]; then; echo $i ' ' $cnt; fi; done; echo ""; echo "***checking
-bash: error importing function definition for `BASH_FUNC_inspectXeption'
-bash: waitqueue: line 0: syntax error near unexpected token `;'
-bash: waitqueue: line 0: `waitqueue () { while true; do; sleep 10; cnt1=`qstat -rn1|grep USER1|wc -l`; if [ $cnt1 -gt 0 ]; then; echo $cnt1; else; break; fi; done; }'
-bash: error importing function definition for `BASH_FUNC_waitqueue'
-bash: sedline: line 0: syntax error near unexpected token `;'
-bash: sedline: line 0: `sedline () { for i in $(seq 1 10); do; sed -i 's/ / /g' $1; done; sed -i 's/^ //g' $1; sed -i ':a;N;$!ba;s/\n /\n/g' $1; sed -i ':a;N;$!ba;s/ \n/\n/g' $1; sed -i 's/ /\t/g' $1; sed -i 's/\tCHR/CHR/g' $1; sed -
-bash: error importing function definition for `BASH_FUNC_sedline'
Sun Nov 3 13:51:02 UTC 2019
Step 1
Step 2
Step 2
Step 3
Sun Nov 3 13:51:32 UTC 2019
finished
the file stdout has the same contents as STDOUT, except the -bash: inspectXeption: part.
Thanks.
Best
Chan
from caper.
tanpuekai
commented on August 23, 2024
Hi Jin,
I guess I know where the problem is.
waitqueue (), sedline (), and inspectXeption () are three self-defined functions in my ~/.bash_profile, wherein I also have these lines:
export -f sedline
export -f waitqueue inspectXeption
and your -V is basically asking our PBS server to take my self-defined functions as well. Somehow, our PBS server has a different environment as its login node, i.e. where my home folder is. So that I can log in and use these self-defined functions, but in the PBS server mode, they can't be defined.
I commented these 3 functions, and it is all ok now.
Thanks.
Best
Chan
from caper.
tanpuekai
commented on August 23, 2024
problem solved:
$ find .|grep stderr|xargs ls -lt
-rw-r--r-- 1 USER1 group 0 Nov 3 15:09 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-4/execution/stderr.check
-rw-r--r-- 1 USER1 group 0 Nov 3 15:09 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-1/execution/stderr.check
-rw-r--r-- 1 USER1 group 0 Nov 3 15:09 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-3/execution/stderr.check
-rw-r--r-- 1 USER1 group 0 Nov 3 15:06 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-2/execution/stderr.check
-rw-r--r-- 1 USER1 group 0 Nov 3 15:06 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-5/execution/stderr.check
-rw-r--r-- 1 USER1 group 0 Nov 3 15:06 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-0/execution/stderr.check
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-2/execution/stderr
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-2/execution/stderr.submit
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-0/execution/stderr
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-0/execution/stderr.submit
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-4/execution/stderr
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-4/execution/stderr.submit
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-5/execution/stderr
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-5/execution/stderr.submit
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-3/execution/stderr
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-3/execution/stderr.submit
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-1/execution/stderr
-rw-r--r-- 1 USER1 group 0 Nov 3 15:02 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-align/shard-1/execution/stderr.submit
-rw------- 1 USER1 group 0 Nov 3 15:01 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-read_genome_tsv/execution/stderr
-rw-r--r-- 1 USER1 group 0 Nov 3 15:01 ./atac/efd3d433-78df-41c8-93ad-d180c753a733/call-read_genome_tsv/execution/stderr.submit
Thanks.
Best
Chan
from caper.
Related Issues (20)
- caper init slurm not working HOT 6
- Newer cromwell versions HOT 2
- changes to sge_resource_param in config file don't make it through to backend.conf HOT 2
- Missing `engine` in gcs configuration HOT 1
- cromwell works, caper doesn't
- Relative paths to data files are not accessible by Singularity containers HOT 4
- Normalization
- Error when specifying --slurm-leader-job-resource-param HOT 1
- apptainer HOT 1
- LSF job parameter error HOT 3
- .caper_tmp slowly eating up space HOT 1
- caper halted
- Cannot run on lsf system, not sure if exit code 127 is the reason HOT 6
- Consider increasing minimum requirement of PyYAML to 6.0.1 HOT 2
- Caper conda forge
- Caper run with scatter runs out of memory HOT 2
- --slurm-extra-param: expected one argument HOT 1
- Setting $SINGULARITY_TMPDIR environment variable
- using slurm-extra-param and the leader job
- Compatibility Update for Caper with Google Batch Following Deprecation of Cloud Life Sciences
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from caper.