Does it need to create the fastq-index files? about rsem HOT 6 CLOSED

Hi @Hecate08 ,

I have to admit that there is no need to extract reads from BAM file. Unfortunately, this is how RSEM is programmed. At the first 10 EM iterations, RSEM need to load these files for estimating sequencing error parameters.

However, I am surprised that the indexing step took that long time. In theory, this step should be as fast as parsing and writing the BAM file. If your BAM file is not as large as several TB, the only reason I can guess is if the server ran out of disk space. Can you check this possibility?

Thanks,
Bo

from rsem.

Hello,

The server has 20 threads and about 60GB memory. For this job I gave rsem 5 threads and in sum 12GB memory. So this should be ok.
I tried rsem now with a smaller sample and with the rsem star alignment. Below are the temp files. It took about 5 hours to build the fastq-files for a bam-file of 3GB. The whole STAR-alignment step tooks 2 hours. What can be the problem that it takes so long?

-rw-rw-r-- 1 lneums jclab 3001763491 Apr 27 09:23 JZ-3.FCC8R8HACXX_L6_IGCCAAT.bam
-rw-rw-r-- 1 lneums jclab 7090 Apr 27 09:23 JZ-3.FCC8R8HACXX_L6_IGCCAATLog.progress.out
-rw-rw-r-- 1 lneums jclab 52225 Apr 27 09:23 JZ-3.FCC8R8HACXX_L6_IGCCAATLog.out
-rw------- 1 jchien jclab 7107497 Apr 27 09:24 JZ-3.FCC8R8HACXX_L6_IGCCAATSJ.out.tab
-rw------- 1 jchien jclab 1867 Apr 27 09:24 JZ-3.FCC8R8HACXX_L6_IGCCAATLog.final.out
-rw------- 1 jchien jclab 0 Apr 27 09:24 JZ-3.FCC8R8HACXX_L6_IGCCAAT.omit
-rw------- 1 jchien jclab 457810116 Apr 27 14:36 JZ-3.FCC8R8HACXX_L6_IGCCAAT_un_2.fq
-rw------- 1 jchien jclab 457810116 Apr 27 14:36 JZ-3.FCC8R8HACXX_L6_IGCCAAT_un_1.fq
-rw------- 1 jchien jclab 277061770 Apr 27 14:36 JZ-3.FCC8R8HACXX_L6_IGCCAAT.dat
-rw------- 1 jchien jclab 2997789796 Apr 27 14:36 JZ-3.FCC8R8HACXX_L6_IGCCAAT_alignable_2.fq
-rw------- 1 jchien jclab 2997789796 Apr 27 14:36 JZ-3.FCC8R8HACXX_L6_IGCCAAT_alignable_1.fq
-rw------- 1 jchien jclab 6032032 Apr 27 14:39 JZ-3.FCC8R8HACXX_L6_IGCCAAT_alignable_1.fq.ridx
-rw------- 1 jchien jclab 6032032 Apr 27 14:41 JZ-3.FCC8R8HACXX_L6_IGCCAAT_alignable_2.fq.ridx
-rw------- 1 jchien jclab 31 Apr 27 14:41 JZ-3.FCC8R8HACXX_L6_IGCCAAT.mparams
-rw------- 1 jchien jclab 739293536 Apr 27 15:53 JZ-3.FCC8R8HACXX_L6_IGCCAAT.ofg
-rw------- 1 jchien jclab 3704242 Apr 27 15:53 JZ-3.FCC8R8HACXX_L6_IGCCAAT.iso_res
-rw------- 1 jchien jclab 3169194 Apr 27 15:53 JZ-3.FCC8R8HACXX_L6_IGCCAAT.gene_res

Thanks
Hecate08

from rsem.

How much time does it take to finish RSEM once you have extracted read files?

from rsem.

@Hecate08 That's so weird. Is it possible that in your server, it is very slow to write things to the disk?

P.S., for the space, I refer to disk space instead of memory.

from rsem.

Sorry, there are 6 TB of diskspace on this server.
The speed of writing to the disk may be the problem. Is there a way to specify a local temp folder instead of writing to the file distribution?
Thanks

from rsem.

@Hecate08, yes, see option --temporary-folder

from rsem.