Error when starting singularity exec iMOKA preprocess.sh -i test about imoka HOT 13 CLOSED

I'm glad it was helpful. I just added the link to the CLI documentation in the main README file, thanks for the suggestion.
Cheers,
Claudio

from imoka.

Hello,
first of all, thanks for trying iMOKA, user feedbacks are always very useful.
The issue should be due to Singularity: by default, it mounts only "canonical" Linux paths, such as /home and /tmp.
If your test text file is in /mnt/projects_tn04/KMER_analysis, for example, it will be not visible by the image.
Try exporting the environmental variable SINGULARITY_BINDPATH with the path(s) you want to mount before running the command.

export SINGULARITY_BINDPATH="/mnt"

Let me know if it helps.
Cheers,
Claudio

from imoka.

Hi,
Hello that was that, thanks!
I knew the singularity behavior but I forgot it!

Now I have to figure out why it tells me :

ERROR! File /mnt/projects_tn04/KMER_analysis/sequences/concatenated_fastq/samplesnovaseq_reads_iss_simulation_01_concatenated.fastq not found!

since I put the zipped fastq for my input path. Maybe it does not support zipped file ?

from imoka.

Hello,
great!
It supports gzipped files, are you sure that in the input file there is the path to the .gz one?

from imoka.

Yes I'm sure the path is correct.
what is strange is my fastq file it written without the ".gz" in the error message :

ERROR! File /mnt/projects_tn04/KMER_analysis/sequences/concatenated_fastq/samplesnovaseq_reads_iss_simulation_01_concatenated.fastq not found!

Here is the screenshot of my input file:

It looks like a problem of singularity path accession again, but I tried :

export SINGULARITY_BINDPATH="/mnt/projects_tn04/KMER_analysis/"

=> which cover all the paths

and with the singularity --bind option

singularity exec --bind "/mnt/projects_tn04/KMER_analysis" tools/iMOKA preprocess.sh -i tools/bio_imoka.tsv -m 1 -o results/results_imoka/ -t 40 -r 500

None of the solutions worked.

from imoka.

It's indeed the name of the file that doesn't convince: the error is raised at line 188 of the preprocess.sh script ( https://github.com/RitchieLabIGH/iMOKA/blob/master/iMOKA_core/scripts/preprocess.sh) and it never truncate the ending part of the string, that's why I asked you to check again the the input file ( and if you are actually giving the correct one as input and not maybe an older one ).
But if the screenshot you are showing is the head of tools/bio_imoka.tsv, I'm not sure what's going on.
Are you using the 1.0 or 1.1 singularity image?

from imoka.

Ok I take a look on your script to understand why I can't see my entire file name

s_links=$(cat tools/bio_imoka.tsv |head -n 1| awk -F "\t" '{gsub(";", " ", $3); print $3}')
echo $s_links

/mnt/projects_tn04/KMER_analysis/sequences/concatenated_fastq/samplesnovaseq_reads_iss_simulation_01_concatenated.fastq.gz

=> that give me that correct path name, but if I try to use it for whatever functions like that:

ls -hail $s_links

ls: impossible d'accéder à /mnt/projects_tn04/KMER_analysis/sequences/concatenated_fastq/samplesnovaseq_reads_iss_simulation_01_con: Aucun fichier ou dossier de ce type

=> the name is truncated of "catenated.fastq.gz". If I do the same command for the entire real path name:

ls -hail /mnt/projects_tn04/KMER_analysis/sequences/concatenated_fastq/samplesnovaseq_reads_iss_simulation_01_concatenated.fastq.gz

152466 -rwxrwxrwx. 1 xxxxxx wheel 1,3G 23 mar 09:44 /mnt/projects_tn04/KMER_analysis/sequences/concatenated_fastq/samplesnovaseq_reads_iss_simulation_01_concatenated.fastq.gz

I don't understand, and I am using the 1.1 version.

from imoka.

Hi there, I am having the exact same issue despite the suggested changes, any help would be greatly appreciated

from imoka.

That's really strange, it's the first time I have seen this happening: it seems that bash is truncating the variable.
I'm not even able to reproduce the error, could you send me your tools/bio_imoka.tsv?
Does the following command give you error as well?

ls -hail ${s_links}

I'm really sorry for this issue, I hope we'll be able to find the solution soon.
@elcdrmj do you have the same issue meaning that it's truncating the name of the file?
Thanks for your patience.
Cheers,
Claudio

from imoka.

Hello, first of all, thanks for trying iMOKA, user feedbacks are always very useful. The issue should be due to Singularity: by default, it mounts only "canonical" Linux paths, such as /home and /tmp. If your test text file is in /mnt/projects_tn04/KMER_analysis, for example, it will be not visible by the image. Try exporting the environmental variable SINGULARITY_BINDPATH with the path(s) you want to mount before running the command.

export SINGULARITY_BINDPATH="/mnt"

Let me know if it helps. Cheers, Claudio

Thanks for the solution. Maybe it shold be in the documentation.

from imoka.

Hello ! I work on the same files as novitch. Isolved a first issue : the input .tsv file had hidden character, probably inserted by an old MacOS (I guess). I remove the carriage return ^M so now the command

s_links=$(cat tools/temp |head -n 1| awk -F "\t" '{gsub(";", " ", $3); print $3}')
echo $s_links

outputs the file name correctly.
But the singularity image

singularity run /mnt/software/singularity/iMOKA create --input /mnt/projects_tn04/KMER_analysis/DATA/imoka_files/temp --output /mnt/projects_tn04/KMER_analysis/DATA/imoka_files/output_imoka.json

still outputs:

Creating a binary file from /mnt/projects_tn04/KMER_analysis/DATA/imoka_files/temp
Rescale Factor: 1e+09
Prefix size: -1
0 - Sample_01
Reading the sorted tsv kmer count text file.
Error! Problem opening Sample_01

Anything I could test ?
PS: there is not \n at the end of the error message, that would be more comfortable to have.

from imoka.

Hello Louis,
I think you are using the tsv file with the input for the preprocess.sh script as input for the iMOKA_core create method.
The order of execution should be:

• preprocess.sh: input a tsv file with file ID, group and file location ( or SRR/ERR entry ). This process will:
  • download the raw data
  • optionally convert the bam into fastq
  • decompose the raw data into k-mer counts (using KMC3)
  • generate the iMOKA binary k-mer counts database
• iMOKA_core create: input a tsv file with the k-mer counts file ( already binary or a tsv file with two columns: the alphabetically sorted k-mers and in the second column the corresponding counts), the file ID and the group. If you used the preprocess.sh to generate the k-mer counts, it's the create_matrix.tsv file. This process will:
  • gather the k-mer counts file, generating the iMOKA binary k-mer counts databases when needed
  • output a json file that can be used in the following steps
• iMOKA_core reduce: input the matrix.json and output a tsv file containing the k-mers able to classify the two or more groups with an accuracy higher than the given threshold
• iMOKA_core aggregate: the input is the reduce output, the matrix.json ( optionally ) and the configuration to blat the k-mer derived sequences ( group of overlapping k-mers ), also this is optional. The output is a json file that can be visualize with the GUI and some tsv files containing the "winning" k-mers ( those with the highest accuracy within a group of overlapping k-mers ).

Let me know if this was helpful and if you meet other issues.

from imoka.

Thank you very much, that was indeed what I was trying. I missed the documentation in iMOKA core. I think it would be better to put it or at least link it in the docs/ or main readme. I only saw the link to the videos.

from imoka.