andersen-lab / freyja Goto Github PK

View Code? Open in Web Editor NEW

101.0 101.0 29.0 74.51 MB

Depth-weighted De-Mixing

License: BSD 2-Clause "Simplified" License

Python 4.84% Makefile 0.02% HTML 95.15%

freyja's People

Contributors

Stargazers

Watchers

freyja's Issues

Demix files not adding up to 1

Hi,

Thank you for making Freyja available. I am plotting lineages for a few wastewater samples, but, for some samples, the lineages in the demix file and the relative summarized barplot do not add up to 100%; the sum in some cases is less than 90%.

Here is an example:

resultsFreyja/1150-UF-WW-8_S286.vars.out.tsv
summarized [('Other', 0.6522580699986724), ('Delta', 0.2237909999997744)]
lineages "['B.1.1.38' 'AY.47' 'AY.67' 'AY.57' 'AY.58' 'AY.59' 'AY.61' 'B.1.1.92'
'AH.2' 'B.31' 'B.1.1.339']"
abundances "[0.644444 0.0372985 0.0372985 0.0372985 0.0372985 0.0372985
0.0372985 0.00304878 0.00218103 0.00143747 0.00114679]"
resid 2.1811085621964645

I run the demix with default options (freyja v. 1.3.1). I will probably install the latest version; in the meantime, can you please help clarify these results?

Thank you,
Max

No module named 'pandas.concat'

Hi Freyja's team,
Thanks for this amazing tool :-)

I updated freyja using the conda command (Linux, Cinnamon). Since then, I get the following error:
Traceback (most recent call last): File "/home/anaconda3/envs/SARSCoV2/bin/freyja", line 6, in <module> from freyja._cli import cli File "/home/anaconda3/envs/SARSCoV2/lib/python3.8/site-packages/freyja/_cli.py", line 3, in <module> from freyja.convert_paths2barcodes import parse_tree_paths,\ File "/home/anaconda3/envs/SARSCoV2/lib/python3.8/site-packages/freyja/convert_paths2barcodes.py", line 1, in <module> import pandas.concat as pd ModuleNotFoundError: No module named 'pandas.concat'
I updated pandas, but it didn't help.
Thanks for your help,

Adelaide

Deprecated warning messages

Hi wonderful Freyja's team,

When I update the database (freyja update), I get the warnings below.
/anaconda3/envs/freyja/lib/python3.10/site-packages/freyja/convert_paths2barcodes.py:27: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. df_barcodes = df_barcodes.append(cladeSeries)

I also get this error, which I don't know if I should worry about:
/anaconda3/envs/freyja/lib/python3.10/site-packages/freyja/convert_paths2barcodes.py:41: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of callingframe.insertmany times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, usenewframe = frame.copy()temp[mt] = df_barcodes[c]

When using the demix function, I get the warning:
/anaconda3/envs/SARSCoV2/lib/python3.8/site-packages/freyja/sample_deconv.py:42: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead. mix = df.loc[keptInds, 'ALT_FREQ'].astype(float)

Thanks
Adelaide

Demix for low genome coverage samples

Hi! Thanks for the great tool.

I have a few questions about the demix step. Based on my understanding, freyja incorporates a unique barcode for every lineage. In the case of low genome coverage samples that do not have amplicons covering all the characteristic mutations (missing a few mutation sites) belonging to a specific lineage, does it get assigned 'Others'?

We have several samples that have genome coverage between 10-20% and freyja demix assigned lineages to them. I am curious how confident these assignments can be.

Thanks!

Custom confing.yaml

Hello,

I have noticed on recent runs that the lineage grouping rules I have in my config.yaml file are not consistenly followed when running freyja dashboard. (I am still using version 1.3.10)
For example I have the following:

Lineages:
  grp_1:
    name: "BA.2"
    members:
      - "BA.2*"
    color: "orange"
  grp_2:
    name: "BA.4.6"
    members:
      - "BA.4.6*"
    color: "olive"
  grp_3:
    name: "BA.4"
    members:
      - "BA.4*"
    color: "green"   
  grp_4:
    name: "BA.5"
    members:
      - "BA.5*"      
    color: "red"

In this case the BA.5 sublineages BA.5.2.3 and BA.5.3.2 are grouped correctly, but BA.5.2.17, BA.5.1.10, and BA.5.5.3 are still showing up individually.
Additionally, how can I write these rules so that BE and BF can also be grouped under BA.5?

Thanks,
Wes

Omicron = A ?

Hi,

thanks for making such an easily accessible and functional tool!

I just installed Freyja through bioconda and got for my sample this output:

summarized	[('Omicron', 0.6661200000004268), ('Delta', 0.3116281285838772)]
lineages	['B.1.1.529' 'AY.43.3' 'AY.1']
abundances	[0.66612    0.2799447  0.03168343]
resid	14.57234387623232

after running freyja update today, the output for the same sample is

summarized	[('A', 0.6699959999989149), ('Delta', 0.31067812395089456)]
lineages	['BA.1.1' 'AY.43.3' 'AY.1']
abundances	[0.669996   0.27980382 0.03087431]
resid	15.751763606543287

So the summary in the first line changed Omicron to A.
Is this correct and what does the A mean?

Freyja update and custom files clarificaiton

Hi Josh!

Could you please confirm my understanding of how I can use the freyja update and then demix commands together? My setup is I'm using the conda package and I would like to run the update command once per day, then grab the output files and send those files to a bunch of parallel processes to use without all those workers having to call freyja update as well.

My understanding is I could call "freyja update" and specify an --outdir. Then I could take the usher_barcodes.csv and lineages.yml that get written there and then call freyja demix { path to freyja.variants.tsv} { path to freyja.depths} --output {path to output freyja.demix} --barcodes usher_barcodes.csv --meta lineages.yml

Could you confirm this is the case? This will fully update all needed reference files for freyja?

Thanks!!

Missing lineagePaths.txt file for update command

When I run the freyja update command I get the error:

FileNotFoundError: [Errno 2] No such file or directory: './lineagePaths.txt'.

I see in the update.py script that
varCmd = f"matUtils extract -i {treePath} -C lineagePaths.txt"
There is no lineagePaths.txt, do I need to supply "lineagePaths.txt'?

freyja demix output inconsistent for abundances

I keep running into inconsistencies in the format for the abundances line in the freyja demix tsv output.

For example, one sample I will get the following output:
sample1.variants.tsv
summarized [('Delta', 0.9891650089965712)]
lineages ['AY.44' 'AY.106' 'AY.39.1' 'AY.3.1']
abundances [0.388514 0.26584011 0.25 0.0848109 ]
resid 5.793999402624937
coverage 81.75099488345651

And the next sample I'll get quotes and a new line introduced breaking up the abundance line:
sample2.variants.tsv
summarized [('Delta', 0.7286465058355318), ('Omicron', 0.14188191374126788), ('Other', 0.11252699999574227)]
lineages ['AY.39' 'AY.126' 'AY.119.2' 'BA.1' 'B.1.1.378' 'AY.117' 'AY.25' 'AY.3']
abundances "[0.16961648 0.169355 0.166667 0.14188191 0.112527 0.10395
0.0726979 0.04636013]"
resid 13.656308232291073
coverage 96.48195833193994

Does a new line get introduced after so many abundance values listed? Can this be kept to one line? I'm trying to parse the file into a final report, and I keep running into issues trying to grab all the values for each sample.

Thanks for your help,
Sarah

Can not plot lineages

I am plotting with and without the --lineages option. Plotting freyja plot AM//AM.aggregate.txt --lineages --output AM//AM.aggregate.lineages.pdf the lineages creates the error message below.

AM.aggregate.txt

  File "/usr/local/bin/freyja", line 33, in <module>
    sys.exit(load_entry_point('freyja==2021.10', 'console_scripts', 'freyja')())
  File "/usr/local/lib/python3.10/site-packages/click-8.0.3-py3.10.egg/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click-8.0.3-py3.10.egg/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click-8.0.3-py3.10.egg/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click-8.0.3-py3.10.egg/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click-8.0.3-py3.10.egg/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/freyja-2021.10-py3.10.egg/freyja/_cli.py", line 170, in plot
    makePlot_simple(agg_df, lineages, output, colors0)
  File "/usr/local/lib/python3.10/site-packages/freyja-2021.10-py3.10.egg/freyja/utils.py", line 109, in makePlot_simple
    ax.bar(k, agg_df.iloc[k][queryType][label],
  File "/usr/local/lib/python3.10/site-packages/matplotlib/__init__.py", line 1412, in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
  File "/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py", line 2395, in bar
    r = mpatches.Rectangle(
  File "/usr/local/lib/python3.10/site-packages/matplotlib/patches.py", line 733, in __init__
    self._convert_units()  # Validate the inputs.
  File "/usr/local/lib/python3.10/site-packages/matplotlib/patches.py", line 744, in _convert_units
    y1 = self.convert_yunits(self._y0 + self._height)
TypeError: unsupported operand type(s) for +: 'int' and 'dict'

freyja aggregate sending IndexError

im trying to call:
freyja aggregate --output output.tsv DEMIX/
where DEMIX is the directory with output files from freyja demix but i keep getting an IndexError

  File "/scicomp/home-pure/ofx5/miniconda3/envs/freyja-env2.0/bin/freyja", line 10, in <module>
    sys.exit(cli())
  File "/scicomp/home-pure/ofx5/miniconda3/envs/freyja-env2.0/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/scicomp/home-pure/ofx5/miniconda3/envs/freyja-env2.0/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/scicomp/home-pure/ofx5/miniconda3/envs/freyja-env2.0/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/scicomp/home-pure/ofx5/miniconda3/envs/freyja-env2.0/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/scicomp/home-pure/ofx5/miniconda3/envs/freyja-env2.0/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/scicomp/home-pure/ofx5/miniconda3/envs/freyja-env2.0/lib/python3.7/site-packages/freyja/_cli.py", line 100, in aggregate
    df_demix = agg(results)
  File "/scicomp/home-pure/ofx5/miniconda3/envs/freyja-env2.0/lib/python3.7/site-packages/freyja/utils.py", line 13, in agg
    df_demix.index = [x.split('/')[1] for x in df_demix.index]
  File "/scicomp/home-pure/ofx5/miniconda3/envs/freyja-env2.0/lib/python3.7/site-packages/freyja/utils.py", line 13, in <listcomp>
    df_demix.index = [x.split('/')[1] for x in df_demix.index]
IndexError: list index out of range

any help?

freyja boot command has error message "Error: No such command 'boot'."

Steps to reproduce:

process files with freyja
use update command
$ freyja update
use specified syntax:
$ freyja boot [variants-file] [depth-file] --nt [number-of-cpus] --nb [number-of-bootstraps] --output_basename [base-name] with freyja generated files

Result:
Usage: freyja [OPTIONS] COMMAND [ARGS]...
Try 'freyja --help' for help.
Error: No such command 'boot'.

Environment:

conda installation & environment
macOS Monterey
bash shell

Maybe this function isn't ready for primetime? Either way, great suite of tools! What a gem!

Freyja dash KeyError

I am running a runinnig into an issue with "freyja dash" when I include "--eps 0.01" in the upstream "freyja demix" command.
I see the following error:

/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py:248: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
df_ab_lin = df_ab_lin.append(
/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py:248: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
df_ab_lin = df_ab_lin.append(
/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py:248: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
df_ab_lin = df_ab_lin.append(
/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py:248: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
df_ab_lin = df_ab_lin.append(
/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py:248: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
df_ab_lin = df_ab_lin.append(
/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py:248: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
df_ab_lin = df_ab_lin.append(
/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py:248: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
df_ab_lin = df_ab_lin.append(
Traceback (most recent call last):
File "/opt/conda/envs/freyja-env/bin/freyja", line 10, in
sys.exit(cli())
File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/_cli.py", line 242, in dash
make_dashboard(agg_df, meta_df, thresh, titleText, introText,
File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py", line 262, in make_dashboard
df_ab_lin = df_ab_lin[cols0]
File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/pandas/core/frame.py", line 3511, in getitem
indexer = self.columns._get_indexer_strict(key, "columns")[1]
File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 5782, in _get_indexer_strict
self._raise_if_missing(keyarr, indexer, axis_name)
File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 5845, in _raise_if_missing
raise KeyError(f"{not_found} not in index")
KeyError: "['Other'] not in index"

I do not get this error and the dashboard is generated as expected when the "--eps" flag is not raised in the demix step.
Please find attached the two aggregate files, one that works and the other that produces the above error. Also attached is the metadata file used to run freyja dash for these.

metadata.csv
aggregate_works.tsv.txt
aggregate_fail.tsv.txt

Understanding Bootstraps estimates

Hi,

Thanks alot for creating Frejya! It works superbly for our waste-water samples.

I simply want to know, how we can be confident that what we are observing in samples is truly there or if we can increase the confidence score somehow (read abundance-based filtering? number/proportion of mutations recovered to confidently assign a variant?). Also could you please explain the bootstrap estimates, for example; what it means to have 95% value of a lineage? or how we can use this information in the article?

Many thanks for your answers!!

Best,
Wasim

cvxpy.error.SolverError in freyja demix

Hello again,

Please see the output below that is halting the complete processing of some samples at the demix stage.
NIRE-00387e_S25.bam.txt
NIRE-00387e_S25_freyja_variants.tsv.txt
NIRE-00387e_S25_freyja_depth.tsv.txt

Thank you very much!
Evan

Commands and output traceback:
freyja variants NIRE-00387e_S25.bam --variants /media/evan/DATA2/freyja/NIRE-00387e_S25_freyja_variants.tsv --depths /media/evan/DATA2/freyja/NIRE-00387e_S25_freyja_depth.tsv --ref /home/evan/anaconda3/ipynb/SARS-COV-2/MN908947.3.fa
[mpileup] 1 samples in 1 input files
freyja demix /media/evan/DATA2/freyja/NIRE-00387e_S25_freyja_variants.tsv /media/evan/DATA2/freyja/NIRE-00387e_S25_freyja_depth.tsv --output /media/evan/DATA2/freyja/NIRE-00387e_S25_freyja_demix.tsv
building mix/depth matrices
demixing
Traceback (most recent call last):
File "/home/evan/anaconda3/envs/freyja-env/bin/freyja", line 10, in
sys.exit(cli())
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/freyja/_cli.py", line 49, in demix
sample_strains, abundances, error = solve_demixing_problem(df_barcodes,
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/freyja/sample_deconv.py", line 139, in solve_demixing_problem
prob.solve(verbose=False)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/cvxpy/problems/problem.py", line 481, in solve
return solve_func(self, *args, **kwargs)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/cvxpy/problems/problem.py", line 1016, in _solve
self.unpack_results(solution, solving_chain, inverse_data)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/cvxpy/problems/problem.py", line 1341, in unpack_results
raise error.SolverError(
cvxpy.error.SolverError: Solver 'ECOS' failed. Try another solver, or solve with verbose=True for more information.

I will also list the conda environment packages below

Freyja update maUtils issue with ouput path

I've found an issue when I run Freyja update command.

In the convert tree function, matUtils has trouble finding the output directory, it tries to output //lineagePaths.txt

When I tried running it with the -d option it fixes it
varCmd = f"matUtils extract -i {treePath} -C lineagePaths.txt -d {locDir}"
-d option is the option for output directory

Weirdly this only comes up when I run from a specific server. So probably not a big issue.

Freyja plot error

Hello,
I am using Freyja for wastewater samples analysis. Its a really good tool for the abundance of the lineages from mixed samples and also gave me the output for my previous analysis.

But now I have few samples for that I am running Freyja it is generating .demix and aggregated.tsv file but when I try to plot that it is throwing an error.
To check is there something wrong with the code I run the tool on previous data file it worked fine. I am not able to resolve the error.
Hope you can help me :)

Attaching the files below

Input fastq file:
barcode84.fastq.gz
demix file output:

barcode84_freyja.variants.tsv
summarized [('BA.2.75* [Omicron (BA.2.75.X)]', 0.9999999999999806)]
lineages BA.2.75
abundances 1.00000000
resid 13.565059442734382
coverage 1.7723974183192321

lineage file and summarise file
barcode84_freyja_boot_lineages.csv

barcode84_freyja_boot_summarized.csv

Screenshot of Output

"Solution may be inaccurate. Try another solver, "

Hello:

I am getting a warning when running Freyja demix, see the details below.
It simply says that the solution may be inaccurate (although when running in verbose mode, it does say it is close to optimal).
I was wondering how reliable the lineage predictions and frequency estimates are when this sort of warning is given, or whether there is any parameter(s) that can be used to judge how accurate Freyja's solutions are?
As to the reason this happens, in this case it might be due to patchy data, but would that be the only possible reason?

Thanks!

/home/user/anaconda3/envs/user_freyja/lib/python3.7/site-packages/cvxpy/problems/problem.py:1338: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
  "Solution may be inaccurate. Try another solver, "

I have run the solver with verbose=True, and get this instead:

EXAMPLE_C_S15_L001.mapped.primertrimmed.sorted.bam
[mpileup] 1 samples in 1 input files
building mix/depth matrices
demixing
===============================================================================
                                     CVXPY
                                     v1.2.1
===============================================================================
(CVXPY) Oct 28 01:24:54 PM: Your problem has 1736 variables, 2 constraints, and 0 parameters.
(CVXPY) Oct 28 01:24:54 PM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Oct 28 01:24:54 PM: (If you need to solve this problem multiple times, but with different data, consider using parameters.)
(CVXPY) Oct 28 01:24:54 PM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
-------------------------------------------------------------------------------
                                  Compilation
-------------------------------------------------------------------------------
(CVXPY) Oct 28 01:24:54 PM: Compiling problem (target solver=ECOS).
(CVXPY) Oct 28 01:24:54 PM: Reduction chain: Dcp2Cone -> CvxAttr2Constr -> ConeMatrixStuffing -> ECOS
(CVXPY) Oct 28 01:24:54 PM: Applying reduction Dcp2Cone
(CVXPY) Oct 28 01:24:54 PM: Applying reduction CvxAttr2Constr
(CVXPY) Oct 28 01:24:54 PM: Applying reduction ConeMatrixStuffing
(CVXPY) Oct 28 01:24:55 PM: Applying reduction ECOS
(CVXPY) Oct 28 01:24:55 PM: Finished problem compilation (took 2.422e-01 seconds).
-------------------------------------------------------------------------------
                                Numerical solver
-------------------------------------------------------------------------------
(CVXPY) Oct 28 01:24:55 PM: Invoking solver ECOS  to obtain a solution.

ECOS 2.0.10 - (C) embotech GmbH, Zurich Switzerland, 2012-15. Web: www.embotech.com/ECOS

It     pcost       dcost      gap   pres   dres    k/t    mu     step   sigma     IR    |   BT
 0  -3.591e-16  -1.144e-14  +2e+04  1e+00  7e-01  1e+00  2e+00    ---    ---    1  2  - |  -  -
 1  +4.056e+00  +5.772e+00  +1e+04  5e-01  2e-01  2e+00  1e+00  0.6412  3e-01   1  2  1 |  0  0
 2  +7.011e+00  +7.701e+00  +3e+03  2e-01  4e-02  7e-01  3e-01  0.7680  5e-02   1  1  1 |  0  0
 3  +8.622e+00  +8.848e+00  +2e+03  2e-01  2e-02  3e-01  2e-01  0.7092  3e-01   2  1  1 |  0  0
 4  +9.349e+00  +9.380e+00  +7e+02  7e-02  7e-03  4e-02  7e-02  0.8872  3e-01   2  1  1 |  0  0
 5  +9.663e+00  +9.676e+00  +4e+02  4e-02  4e-03  2e-02  4e-02  0.5862  2e-01   2  1  1 |  0  0
 6  +9.834e+00  +9.836e+00  +1e+02  1e-02  1e-03  3e-03  1e-02  0.8552  2e-01   2  1  1 |  0  0
 7  +9.914e+00  +9.915e+00  +6e+01  6e-03  5e-04  2e-03  5e-03  0.5083  7e-02   2  1  1 |  0  0
 8  +9.934e+00  +9.935e+00  +5e+01  5e-03  4e-04  1e-03  4e-03  0.2589  3e-01   2  1  2 |  0  0
 9  +9.951e+00  +9.951e+00  +3e+01  3e-03  3e-04  5e-04  3e-03  0.8964  6e-01   2  1  1 |  0  0
10  +9.981e+00  +9.981e+00  +8e+00  9e-04  8e-05  1e-04  8e-04  0.7562  4e-02   2  2  1 |  0  0
11  +9.998e+00  +9.998e+00  +3e+00  3e-04  3e-05  4e-05  3e-04  0.7559  2e-01   2  2  2 |  0  0
12  +1.001e+01  +1.001e+01  +1e+00  2e-04  1e-05  2e-05  1e-04  0.6367  2e-01   2  2  2 |  0  0
13  +1.001e+01  +1.001e+01  +7e-01  8e-05  7e-06  1e-05  7e-05  0.6382  2e-01   3  2  2 |  0  0
14  +1.001e+01  +1.001e+01  +4e-01  4e-05  4e-06  5e-06  3e-05  0.6355  3e-01   2  2  2 |  0  0
15  +1.001e+01  +1.001e+01  +3e-01  3e-05  3e-06  4e-06  3e-05  0.3356  3e-01   3  2  2 |  0  0
16  +1.001e+01  +1.001e+01  +1e-01  2e-05  1e-06  2e-06  1e-05  0.5722  1e-01   2  2  2 |  0  0
17  +1.001e+01  +1.001e+01  +8e-02  1e-05  8e-07  1e-06  8e-06  0.5590  3e-01   2  2  2 |  0  0
18  +1.001e+01  +1.001e+01  +7e-02  8e-06  7e-07  1e-06  6e-06  0.3799  5e-01   2  2  2 |  0  0
19  +1.001e+01  +1.001e+01  +5e-02  6e-06  5e-07  7e-07  5e-06  0.6506  6e-01   2  2  2 |  0  0
20  +1.001e+01  +1.001e+01  +4e-02  5e-06  4e-07  5e-07  4e-06  0.7978  7e-01   3  2  2 |  0  0
21  +1.001e+01  +1.001e+01  +1e-02  2e-06  2e-07  2e-07  1e-06  0.6577  7e-02   2  2  2 |  0  0
22  +1.001e+01  +1.001e+01  +9e-04  1e-07  9e-09  1e-08  8e-08  0.9701  3e-02   3  2  2 |  0  0
23  +1.001e+01  +1.001e+01  +2e-04  3e-08  2e-09  4e-09  2e-08  0.9291  2e-01   3  2  3 |  0  0
24  +1.001e+01  +1.001e+01  +2e-05  4e-09  2e-10  3e-10  2e-09  0.9890  9e-02   5  3  4 |  0  0
25  +1.001e+01  +1.001e+01  +1e-05  3e-09  1e-10  2e-10  1e-09  0.5557  9e-02   2  1  1 |  0  0
26  +1.001e+01  +1.001e+01  +1e-05  2e-09  1e-10  2e-10  1e-09  0.5166  8e-01   1  1  1 |  0  0
27  +1.001e+01  +1.001e+01  +5e-06  1e-09  5e-11  7e-11  5e-10  0.5757  4e-02   1  1  1 |  0  0
28  +1.001e+01  +1.001e+01  +2e-06  1e-09  2e-11  3e-11  2e-10  0.8604  3e-01   0  1  1 |  0  0
29  +1.001e+01  +1.001e+01  +1e-07  2e-10  1e-12  2e-12  1e-11  0.9550  3e-02   1  1  1 |  0  0
30   +nan   -nan  -nan  -nan  nan  -nan  -nan  0.9890  1e-04   1  0  0 |  0  0
Reached NAN dead end, recovering best iterate (29) and stopping.

Close to OPTIMAL (within feastol=2.5e-10, reltol=1.4e-08, abstol=1.4e-07).
Runtime: 1.090763 seconds.

/home/user/anaconda3/envs/user_freyja/lib/python3.7/site-packages/cvxpy/problems/problem.py:1338: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
  "Solution may be inaccurate. Try another solver, "
-------------------------------------------------------------------------------
                                    Summary
-------------------------------------------------------------------------------
(CVXPY) Oct 28 01:24:56 PM: Problem status: optimal_inaccurate
(CVXPY) Oct 28 01:24:56 PM: Optimal value: 1.001e+01
(CVXPY) Oct 28 01:24:56 PM: Compilation took 2.422e-01 seconds
(CVXPY) Oct 28 01:24:56 PM: Solver (including time spent in interface) took 1.091e+00 seconds

AttributeError: 'Series' object has no attribute 'log2'

Dear Joshua,
Thanks for developing this useful tools. I just install the version 1.3.11 and I get the following error when I am using the freyja demix function:

File "/home/s4519890/miniconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/generic.py", line 5907, in getattr
return object.getattribute(self, name)
AttributeError: 'Series' object has no attribute 'log2'
I try to converts the data type of the Series object from object to np.float64. But still not working.
I have using Evan's .bam file (Thanks for provide this file @evantroendle) as a test and it running well with no error. I also uploaded my bam file: https://drive.google.com/file/d/1_EUklUSMEPMxbnjz6cLSHioTfmOrIqbF/view?usp=sharing, could you please give some suggestions?

Thanks,
Yu.

Originally posted by @YuWang07 in #50 (comment)

Filter out recombinant lineages?

Would it be possible to filter out recombinant lineages when using --lineages? Especially with some of these new recombinants as a mix of various Omicron subvariants and some Delta lineages, the data can be a bit messy. Thanks!

Question about depth-weighted demixing

Hello,
I was curious if you could explain the math behind freyja's depth-weighted demixing.
It seems that you take the total value of the barcode value (A) subtracted by the SNV frequency (b)? I am having trouble understanding how you take depth of the SNV into account from your formula?

freyja update fails within conda environment with Snakemake

Downloading a new global tree
Getting outbreak data
Converting tree info to barcodes
/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/freyja/data
Loading input MAT file /data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/freyja/data/public-latest.all.masked.pb.gz.
Building barcodes from global phylogenetic tree
Traceback (most recent call last):
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/bin/freyja", line 10, in <module>
    sys.exit(cli())
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/freyja/_cli.py", line 102, in update
    df = pd.read_csv(lineagePath, sep='\t')
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/pandas/util/_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/pandas/util/_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 605, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1735, in _make_engine
    self.handles = get_handle(
  File "/data/gpfs/assoc/inbre/hansvg/home/miniconda3/envs/freyja/lib/python3.10/site-packages/pandas/io/common.py", line 856, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: './lineagePaths.txt'

The update command is able to successfully download the *.pb.gz file. However, when it tries to convert the file, it fails.

Looking at the source code, it looks like you can potentially update line 100 in __cli.py.

lineagePath = os.path.join(os.curdir, "lineagePaths.txt")

changed to

lineagePath = os.path.join(locDir, "lineagePaths.txt")

This should allow freyja to update using conda within a Snakemake environment.

samtools not working with freyja install

i keep getting an error when i run freyja:
samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

i get the same error when i run samtools alone in the freyja environment.

KeyError over many samples when using Freyja demix

Hello, I have been using Freyja for a research project and it has been very useful, interesting, and helpful, so thank you!

Regarding this issue:

I am batch processing 40252 *.bam files, but for some reason only 15364 of them have 'gotten through' the freyja demix step. All of the failures come from similar KeyErrors, and the mean alignment properties are below (showing most alignments are decent):
n_reads 107792
n_mapped 102044
percent_mapped 93.65

Example of a frequent error message:

freyja demix /media/evan/DATA2/freyja/NIRE-00304f_S45_freyja_variants.tsv /media/evan/DATA2/freyja/NIRE-00304f_S45_freyja_depth.tsv --output /media/evan/DATA2/freyja/NIRE-00304f_S45_freyja_demix.tsv
/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/freyja/sample_deconv.py:42: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
mix = df.loc[keptInds, 'ALT_FREQ'].astype(float)
building mix/depth matrices
Traceback (most recent call last):
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc
return self._engine.get_loc(casted_key)A
File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 2131, in pandas._libs.hashtable.Int64HashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 2140, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 22356

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/evan/anaconda3/envs/freyja-env/bin/freyja", line 10, in
sys.exit(cli())
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/freyja/cli.py", line 48, in demix
mix, depths = build_mix_and_depth_arrays(variants, depths, muts)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/freyja/sample_deconv.py", line 44, in build_mix_and_depth_arrays
b" depths = pd.Series({kI: df_depth.loc[int(re.findall(r'\d+', kI)[0]), 3]\n"
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/freyja/sample_deconv.py", line 44, in
b" depths = pd.Series({kI: df_depth.loc[int(re.findall(r'\d+', kI)[0]), 3]\n"
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/pandas/core/indexing.py", line 960, in getitem
return self.obj._get_value(*key, takeable=self._takeable)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/pandas/core/frame.py", line 3622, in _get_value
row = self.index.get_loc(index)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3623, in get_loc
raise KeyError(key) from err
KeyError: 22356\n

The variants and depth tsvs are attached (as txt)
NIRE-00304f_S45_freyja_depth.tsv.txt
NIRE-00304f_S45_freyja_variants.tsv.txt
in case this helps.

I will leave my conda list in case for replication or to otherwise spot a versioning issue:
packages in environment at /home/evan/anaconda3/envs/freyja-env:

Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
alsa-lib 1.2.3 h516909a_0 conda-forge
argon2-cffi 21.3.0 pyhd3eb1b0_0
argon2-cffi-bindings 21.2.0 py39h7f8727e_0
asttokens 2.0.5 pyhd3eb1b0_0
attrs 21.4.0 pyhd3eb1b0_0
backcall 0.2.0 pyhd3eb1b0_0
beautifulsoup4 4.10.0 pyh06a4308_0
blas 1.0 openblas
bleach 4.1.0 pyhd3eb1b0_0
boost-cpp 1.74.0 h6cacc03_7 conda-forge
bottleneck 1.3.4 py39hce1f21e_0
bowtie2 2.4.5 py39hd2f7db1_2 bioconda
brotli 1.0.9 he6710b0_2
bzip2 1.0.8 h7b6447c_0
c-ares 1.18.1 h7f8727e_0
ca-certificates 2022.3.29 h06a4308_0
certifi 2021.10.8 py39h06a4308_2
cffi 1.15.0 py39hd667e15_1
click 8.0.4 py39h06a4308_0
colorama 0.4.4 pyhd3eb1b0_0
cvxpy 1.2.0 py39hf3d152e_0 conda-forge
cvxpy-base 1.2.0 py39h1832856_0 conda-forge
cycler 0.11.0 pyhd3eb1b0_0
dbus 1.13.6 h5008d03_3 conda-forge
debugpy 1.5.1 py39h295c915_0
decorator 5.1.1 pyhd3eb1b0_0
defusedxml 0.7.1 pyhd3eb1b0_0
ecos 2.0.7.post1 py39h6323ea4_0
entrypoints 0.3 py39h06a4308_0
executing 0.8.3 pyhd3eb1b0_0
expat 2.4.8 h27087fc_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hd3eb1b0_0
font-ttf-inconsolata 2.001 hcb22688_0
font-ttf-source-code-pro 2.030 hd3eb1b0_0
font-ttf-ubuntu 0.83 h8b1ccd4_0
fontconfig 2.13.96 h8e229c2_2 conda-forge
fonts-anaconda 1 h8fa9717_0
fonts-conda-ecosystem 1 hd3eb1b0_0
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.11.0 h70c0345_0
freyja 1.3.4 pyhdfd78af_0 bioconda
gettext 0.21.0 hf68c758_0
giflib 5.2.1 h7b6447c_0
gst-plugins-base 1.20.1 hcf0ee16_1 conda-forge
gstreamer 1.20.1 hd4edc92_1 conda-forge
hisat2 2.1.0 h5005445_0 biobuilds
htslib 1.14 h9753748_2 bioconda
icu 69.1 h9c3ff4c_0 conda-forge
importlib-metadata 4.11.3 py39h06a4308_0
importlib_metadata 4.11.3 hd3eb1b0_0
ipykernel 6.9.1 py39h06a4308_0
ipython 8.2.0 py39h06a4308_0
ipython_genutils 0.2.0 pyhd3eb1b0_1
isa-l 2.30.0 ha770c72_4 conda-forge
ivar 1.3.1 hecb563c_3 bioconda
jbig 2.1 hdba287a_0
jedi 0.18.1 py39h06a4308_1
jinja2 3.0.3 pyhd3eb1b0_0
joblib 1.1.0 pyhd3eb1b0_0
jpeg 9e h7f98852_0 conda-forge
jsonschema 3.2.0 pyhd3eb1b0_2
jupyter_client 7.1.2 pyhd3eb1b0_0
jupyter_core 4.9.2 py39h06a4308_0
jupyterlab_pygments 0.1.2 py_0
kiwisolver 1.3.2 py39h295c915_0
krb5 1.19.2 hac12032_0
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
lerc 3.0 h295c915_0
libblas 3.9.0 8_openblas conda-forge
libcbor 0.9.0 h9c3ff4c_0 conda-forge
libclang 13.0.1 default_hc23dcda_0 conda-forge
libcurl 7.82.0 h0b77cf5_0
libdeflate 1.10 h7f98852_0 conda-forge
libedit 3.1.20210910 h7f8727e_0
libev 4.33 h7f8727e_1
libevent 2.1.10 h9b69904_4 conda-forge
libffi 3.4.2 h295c915_2
libfido2 1.10.0 h812cca2_0 conda-forge
libgcc-ng 11.2.0 h1d223b6_14 conda-forge
libgfortran-ng 7.5.0 ha8ba4b0_17
libgfortran4 7.5.0 ha8ba4b0_17
libglib 2.70.2 h174f98d_4 conda-forge
libgomp 11.2.0 h1d223b6_14 conda-forge
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 8_openblas conda-forge
libllvm13 13.0.1 hf817b99_2 conda-forge
libnghttp2 1.46.0 hce63b2e_0
libnsl 2.0.0 h7f98852_0 conda-forge
libogg 1.3.5 h27cfd23_1
libopenblas 0.3.12 pthreads_hb3c22a3_1 conda-forge
libopus 1.3.1 h7b6447c_0
libpng 1.6.37 hbc83047_0
libpq 14.2 hd57d9b9_0 conda-forge
libprotobuf 3.18.1 h780b84a_0 conda-forge
libsodium 1.0.18 h7b6447c_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 11.2.0 he4da1e4_14 conda-forge
libtiff 4.3.0 h542a066_3 conda-forge
libudev1 249 h7f98852_1 conda-forge
libuuid 2.33.1 h14c3975_0 biobuilds
libvorbis 1.3.7 h7b6447c_0
libwebp 1.2.2 h55f646e_0
libwebp-base 1.2.2 h7f8727e_0
libxcb 1.13 h7f98852_1004 conda-forge
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge
libxml2 2.9.12 h885dcf4_1 conda-forge
libzlib 1.2.11 h166bdaf_1014 conda-forge
lz4-c 1.9.3 h295c915_1
mafft 7.490 hec16e2b_1 bioconda
markupsafe 2.0.1 py39h27cfd23_0
matplotlib 3.5.1 py39h06a4308_1
matplotlib-base 3.5.1 py39ha18d171_1
matplotlib-inline 0.1.2 pyhd3eb1b0_2
mistune 0.8.4 py39h27cfd23_1000
mpi 1.0 openmpi
munkres 1.1.4 py_0
mysql-common 8.0.28 haf5c9bc_2 conda-forge
mysql-connector-c 6.1.11 h24aacaa_2
mysql-libs 8.0.28 h28c427c_2 conda-forge
nb_conda_kernels 2.3.1 py39h06a4308_0
nbclient 0.5.11 pyhd3eb1b0_0
nbconvert 6.4.4 py39h06a4308_0
nbformat 5.3.0 py39h06a4308_0
ncurses 6.3 h7f8727e_2
nest-asyncio 1.5.5 py39h06a4308_0
nextclade 1.11.0 h9ee0642_0 bioconda
notebook 6.4.8 py39h06a4308_0
nspr 4.33 h295c915_0
nss 3.74 h0370c37_0
numexpr 2.8.1 py39hecfb737_0
numpy 1.21.5 py39h7a5d4dd_1
numpy-base 1.21.5 py39hb8be1f0_1
openmpi 4.1.3 h414127c_0 conda-forge
openssh 8.9p1 hf695f80_0 conda-forge
openssl 1.1.1n h7f8727e_0
osqp 0.6.2.post0 py39hde0f152_3 conda-forge
packaging 21.3 pyhd3eb1b0_0
pandas 1.4.1 py39h295c915_1
pandocfilters 1.5.0 pyhd3eb1b0_0
parso 0.8.3 pyhd3eb1b0_0
pcre 8.45 h295c915_0
perl 5.26.2 h14c3975_0
pexpect 4.8.0 pyhd3eb1b0_3
pickleshare 0.7.5 pyhd3eb1b0_1003
pillow 9.0.1 py39h22f2fdc_0
pip 21.2.4 py39h06a4308_0
prometheus_client 0.13.1 pyhd3eb1b0_0
prompt-toolkit 3.0.20 pyhd3eb1b0_0
pthread-stubs 0.3 h0ce48e5_1
ptyprocess 0.7.0 pyhd3eb1b0_2
pure_eval 0.2.2 pyhd3eb1b0_0
pycparser 2.21 pyhd3eb1b0_0
pygments 2.11.2 pyhd3eb1b0_0
pyparsing 3.0.4 pyhd3eb1b0_0
pyqt 5.12.3 py39hf3d152e_8 conda-forge
pyqt-impl 5.12.3 py39hde8b62d_8 conda-forge
pyqt5-sip 4.19.18 py39he80948d_8 conda-forge
pyqtchart 5.12 py39h0fcd23e_8 conda-forge
pyqtwebengine 5.12.1 py39h0fcd23e_8 conda-forge
pyrsistent 0.18.0 py39heee7806_0
python 3.9.12 h9a8a25e_1_cpython conda-forge
python-dateutil 2.8.2 pyhd3eb1b0_0
python-fastjsonschema 2.15.1 pyhd3eb1b0_0
python_abi 3.9 2_cp39 conda-forge
pytz 2021.3 pyhd3eb1b0_0
pyzmq 22.3.0 py39h295c915_2
qdldl-python 0.1.5 py39hde0f152_2 conda-forge
qt 5.12.9 h1304e3e_6 conda-forge
readline 8.1.2 h7f8727e_1
samtools 0.1.19 h887b0a7_1 biobuilds
scipy 1.7.3 py39h492baa0_0
scs 3.2.0 py39h0ec8588_0 conda-forge
seaborn 0.11.2 pyhd3eb1b0_0
send2trash 1.8.0 pyhd3eb1b0_1
setuptools 61.2.0 py39h06a4308_0
six 1.16.0 pyhd3eb1b0_1
soupsieve 2.3.1 pyhd3eb1b0_0
sqlite 3.38.2 hc218d9a_0
stack_data 0.2.0 pyhd3eb1b0_0
tbb 2020.2 h4bd325d_4 conda-forge
terminado 0.13.1 py39h06a4308_0
testpath 0.5.0 pyhd3eb1b0_0
tk 8.6.12 h27826a3_0 conda-forge
tornado 6.1 py39h27cfd23_0
tqdm 4.63.0 pyhd3eb1b0_0
traitlets 5.1.1 pyhd3eb1b0_0
typing-extensions 4.1.1 hd3eb1b0_0
typing_extensions 4.1.1 pyh06a4308_0
tzdata 2022a hda174b7_0
ucsc-fatovcf 426 h954228d_0 bioconda
usher 0.5.3 hf1ae886_0 bioconda
wcwidth 0.2.5 pyhd3eb1b0_0
webencodings 0.5.1 py39h06a4308_1
wheel 0.37.1 pyhd3eb1b0_0
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.2.5 h7b6447c_0
zeromq 4.3.4 h2531618_0
zipp 3.7.0 pyhd3eb1b0_0
zlib 1.2.11 h166bdaf_1014 conda-forge
zstd 1.5.2 ha95c52a_0 conda-forge

All the best,
Evan

Recombinant detection

Hi Freyja's team,

Thanks again for your amazing tool!

Would it be possible to have the option to remove the recombinants from the analysis? I'm not questioning the existence of recombinants, I'm just skeptical of the relevance/accuracy of calling recombinants from wastewater using short-read sequencing.
Thanks,

Adelaide

BQ1.1 not present in UShER database upon updating

Big fan of Freyja. I have periodically been updating the tool using 'freyja update' to ensure I have the latest and greatest usher barcodes database. I have not yet seen BQ1.1 in the database, just BQ.1.

I am wondering if this suggests there is a code issue or if the database has simply not been updated yet by the folks that curate the UShER tree for SARS-CoV-2.

Thanks for any insights.

PS: I updated to Freyja 1.3.11 and called 'freyja update' last the morning of Oct 19, 2022.

Region option to use for targeted amplification

hi there, I'm working on some data that uses a targeted amplification approach vs whole genome (mostly to increase sensitivity and focus on the most informative regions) but there does not seem to be an option to specify a subset of the full genome for the variant and demix steps and this seems to be messing with the assignments/depth estimates. @joshuailevy is this something you have given thought to? mpileup etc take the -r command with a bed file which should reduce the depth and variant scope but I'm guessing you would need additional corrections in deconvolution...

pandas InvalidIndexError when performing freyja aggregate

When running freyja aggregate:
freyja aggregate /home/evan/anaconda3/ipynb/SARS-COV2/results/freyja/ --output freyja_aggregated_results.tsv

I get this output:

Traceback (most recent call last):
File "/home/evan/anaconda3/envs/freyja-env/bin/freyja", line 10, in
sys.exit(cli())
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/freyja/_cli.py", line 144, in aggregate
df_demix = agg(results)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py", line 12, in agg
df_demix = pd.concat(allResults, axis=1).T
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 359, in concat
return op.get_result()
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 588, in get_result
indexers[ax] = obj_labels.get_indexer(new_labels)
File "/home/evan/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3721, in get_indexer
raise InvalidIndexError(self._requires_unique_msg)
pandas.errors.InvalidIndexError: Reindexing only valid with uniquely valued Index objects

The folder in question solely contains the following files:
SAMPLE1_freyja_demix.tsv
SAMPLE1_freyja_depth.tsv
SAMPLE1_freyja_variants.tsv
...
SAMPLEN_freyja_demix.tsv
SAMPLEN_freyja_depth.tsv
SAMPLEN_freyja_variants.tsv

Add version

It would be helpful to me if there was a -v or -V or --version or even freyja version for an easy way to see if I'm current.

Suggestions for including deletion or insertion mutations into mutation barcode file

Hi there! Thanks for a nice tool.

It appears that the USHER tool does not support indel mutations (see the thread at this link). Is there a way to feed in indel data to Freyja in a custom barcode file?

Thanks!

Output unclear

Hi, Thank you for having this software open to the public. Could you document what the output of the demix step is? This is one of mine. I am unclear on what the summarized line gives apart from lineages, and I am unclear what resid means. Thank you!

        /tmp/10192355.1.all.q/tmp.VIFAQ9Hmvc10192355/SRR15383942.freyja/freyja.variants.tsv
summarized      [('Delta', 0.989275601990426), ('Other', 0.004566639999873597)]
lineages        ['B.1.617.2' 'AY.6' 'AY.20' 'B.1.1.434' 'B.1.177.2' 'AY.23' 'B.1.127']
abundances      "[0.97995719 0.00479616 0.00320588 0.00207684 0.00145138 0.00131637
 0.00103842]"
resid   2.1489096407905945

Aggregation issue

I installed Freyja yesterday afternoon using conda. The variants and demix steps worked fine.

I now have all of my output files in a single subdirectory called 'demixed'. When I go to aggregate them using the command 'freyja aggregate demixed --output ww-omi-demixed.tsv' (which looks like it is correctly formed per your documentation) I get an error that looks like it traces back to a '/' not being included when a file path was formed, because I do have a file 'demixed/112921-27.demix.out', but obviously ''demixed112921-27.demix.out' is not going to work:

FileNotFoundError: [Errno 2] No such file or directory: 'demixed112921-27.demix.out'

Weirdly this is not happening on the first file in the demixed directory, but the 7th.

Any suggestions?

Here's the tl/dr on the traceback:

Traceback (most recent call last):
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/bin/freyja", line 10, in <module>
    sys.exit(cli())
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/freyja/_cli.py", line 105, in aggregate
    df_demix = agg(results)
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/freyja/utils.py", line 11, in agg
    index_col=0) for fn in os.listdir(results)]
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/freyja/utils.py", line 11, in <listcomp>
    index_col=0) for fn in os.listdir(results)]
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 586, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 482, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 811, in __init__
    self._engine = self._make_engine(self.engine)
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 1040, in _make_engine
    return mapping[engine](self.f, **self.options)  # type: ignore[call-arg]
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 51, in __init__
    self._open_handles(src, kwds)
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/io/parsers/base_parser.py", line 229, in _open_handles
    errors=kwds.get("encoding_errors", "strict"),
  File "/Users/cgibas/opt/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/io/common.py", line 707, in get_handle
    newline="",
FileNotFoundError: [Errno 2] No such file or directory: 'demixed112921-27.demix.out'

Freyja variants invalid option -- 'r'

Hi,
I'm trying to run Freyja on bam file (aligned reads to covid reference, with trimmed primers - bam file after artic-ncov2019 pipeline):

freyja variants --variants "var_output" --depths "var_depth_output" --ref "nCoV-2019.reference.fasta" "primertrimmed.rg.sorted.bam"

and I always get error info "ivar: invalid option -- 'r' ". The same error is when running freyja variants without options like --variants, --depths or even --ref, so it is probably not the freyja arguments problem...

this topic: https://unix.stackexchange.com/questions/555360/bash-cat-invalid-option-r resolving similar problem with cat command, where cat is interpreting -rsomething like an additional argument '-r'. The problem was resolved by ending arguments section with standalone " -- ". Maybe here I have similar problem, but how to edit iVar command running by Freya?
[Freyja version 1.3.8, iVar version 1.0]

Problems running update

I am getting the following error when running freyja update.

Downloading a new global tree
Getting outbreak data
Converting tree info to barcodes
.
Loading input MAT file ./public-latest.all.masked.pb.gz.
Building barcodes from global phylogenetic tree
Traceback (most recent call last):
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/bin/freyja", line 10, in <module>
    sys.exit(cli())
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/freyja/_cli.py", line 99, in update
    df = pd.read_csv(lineagePath, sep='\t')
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 575, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 933, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1217, in _make_engine
    self.handles = get_handle(  # type: ignore[call-overload]
  File "/data/gpfs/assoc/inbre/projects/subhash_verma/covid_wastewater/snakemake_freyja_covidwastewater/.snakemake/conda/932d05727f530c39c2c142d5886fcbf9/lib/python3.10/site-packages/pandas/io/common.py", line 789, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: './lineagePaths.txt'

Looking at the source code, could it be from line 96 in _cli.py?

lineagePath = os.path.join(os.curdir, "lineagePaths.txt")

Should os.curdir be locDir instead?

Also, in the downstream commands, I don't see a way to configure the "update" directory with the data. How does freyja know where to look for this data?

Thank you,
-Hans

Freyja Variants creates unregistered bases

Hi everyone,
I'm currently using Freyja in a pipeline that processes a set of 414 COVID-19 samples, and I am able to get output TSV files that give positions on the single nucleotide variants. I noticed how the TSV files have positions with alternate base like -T or +C, but when I downloaded these as CSV files, their cells would only give errors saying "#NAME?". If someone could tell me why are there +/- bases, as well as how I can fix the #NAME? issue, that would be great!

The code and files should be attached too; I couldn't get the TSV file uploaded since Github doesn't support that format, but I pasted a screenshot of the file with some of the positions listed

TWC3_S126_variants.csv

`#!/bin/bash

refStrain=wuhan.fasta

FilesLoc=./ARGONNE_FILES/ARGONNE_FASTQ

PrimerLoc=qiaseqdirectsarscov2primersfinal.bed
CsvResults=./all_freyja/all_csv
#conda activate freyja

prefix=$1
prefixdir=all_freyja/all_temp/$prefix
mkdir $prefixdir

csvdir=$CsvResults/$prefix
mkdir $csvdir

Ivar_Dir=all_freyja/Ivar_Varians_output/$prefix
mkdir $Ivar_Dir

bwa-mem2 mem $refStrain $FilesLoc/${prefix}_R1_001.fastq $FilesLoc/${prefix}_R2_001.fastq > $prefixdir/$prefix.sam

samtools view -S -b $prefixdir/$prefix.sam > $prefixdir/$prefix.bam

samtools sort -o $prefixdir/$prefix.sorted.bam $prefixdir/$prefix.bam

samtools index $prefixdir/$prefix.sorted.bam

ivar trim -i $prefixdir/$prefix.sorted.bam -b $PrimerLoc -p $prefixdir/$prefix.trimmed

samtools sort -o $prefixdir/$prefix.trimmed.sorted.bam $prefixdir/$prefix.trimmed.bam

freyja variants $prefixdir/$prefix.trimmed.sorted.bam --variants $prefixdir/${prefix}_variants.tsv --depths 

freyja demix $prefixdir/${prefix}_variants.tsv $prefixdir/$prefix.depths --output $csvdir/$prefix.tsv`

Freyja Dashboard Unexpected Scaled Output

I was reviewing the "Load Sacled ..." tabs of a Freyja generated dashboard, and I was observing unexpected counts and relative proportions.
For example I used the following input files:
metadata_test.csv
aggregate_test.tsv.txt

In this contrived example there are two samples both from the same site on the same day. Sample 1 has 500k copies/L and Sample 2 has 1,000K copies/L. Sample 1 is 50% BA.2 and 50% BA.4. Sample 2 is 50% BA.2 and 50% BA.5.
Therefore I was expecting to the scaled lineages to be BA.2=750K/L (50%), BA.4=250K/L (16.7%), and BA.5=500K/L (33.3%)
However, the result from freyja dashboard seems to be based on the average of the overall lineage count of 750K resulting in BA.2=375K/L (50%), BA.4=187.5K/L (25%), BA.5=187.5K/L (25%).
Is this the intended behavior?

BC.1 in 'confirmed only' output

We're running in confirmed only mode and getting BC.1 classifications while we are fairly certain this is not around in our area

Convert demix to R-readable format

Hi Freyja devs,
Thanks for making a very useful software package. I have a question, that others may have too. Do you have a tool (or script for that matter) to easily export the demix files and/or the aggregated demix files into a table for importing into R? I think a lot of people who are not Python users would benefit from this.
Thanks!
Jason

Issue using --lineages flag with freyja plotting

Hi,

I am having an issue running freyja's plot function with the following command...

freyja plot --lineages ArgonneOverallPlot.tsv --output ArgonneOverallPlot5day.png --times sampleTimes.csv --interval D --windowsize 5

Whenever I attempt to plot using the --lineages flag, I encounter the following issue:

Traceback (most recent call last):
File "/home/bcg64/anaconda3/envs/freyja-env/bin/freyja", line 10, in
sys.exit(cli())
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/freyja/_cli.py", line 219, in plot
makePlot_time(agg_df, lineages, times_df, interval, output,
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py", line 166, in makePlot_time
agg_df = prepLineageDict(agg_df)
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py", line 21, in prepLineageDict
.apply(lambda x:
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/series.py", line 4433, in apply
return SeriesApply(self, func, convert_dtype, args, kwargs).apply()
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/apply.py", line 1082, in apply
return self.apply_standard()
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/apply.py", line 1137, in apply_standard
mapped = lib.map_infer(
File "pandas/_libs/lib.pyx", line 2870, in pandas._libs.lib.map_infer
File "/home/bcg64/anaconda3/envs/freyja-env/lib/python3.10/site-packages/freyja/utils.py", line 22, in
x.replace("'", "")
AttributeError: 'float' object has no attribute 'replace'

Please let me know if you have any recommendations on how to fix this issue!

Demix run error code

I am trying to run the demix code, however I'm running into this issue? Could it be due to the python version?

ERROR CODE:
Traceback (most recent call last):
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 2131, in pandas._libs.hashtable.Int64HashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 2140, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 10124

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/anaconda3/envs/freyja-env/bin/freyja", line 10, in
sys.exit(cli())
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/freyja/cli.py", line 55, in demix
mix, depths, cov = build_mix_and_depth_arrays(variants, depths, muts,
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/freyja/sample_deconv.py", line 51, in build_mix_and_depth_arrays
depths = pd.Series({kI: df_depth.loc[int(re.findall(r'\d+', kI)[0]), 3]
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/freyja/sample_deconv.py", line 51, in
depths = pd.Series({kI: df_depth.loc[int(re.findall(r'\d+', kI)[0]), 3]
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/indexing.py", line 960, in getitem
return self.obj._get_value(*key, takeable=self._takeable)
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/frame.py", line 3622, in _get_value
row = self.index.get_loc(index)
File "/anaconda3/envs/freyja-env/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3623, in get_loc
raise KeyError(key) from err
KeyError: 10124

Option to specify database directory?

Hello!

Super awesome tool! I was wondering if it would be possible to get an option to specify the location of the downloaded database.

For example

freyja update --database /home/rpetit3/freyjadb
freyja variants <OPTS> --database /home/rpetit3/freyjadb
freyja demix <OPTS> --database /home/rpetit3/freyjadb

I ask because I'd like to update the database while using the Docker container from Staph-B, but I'm running into write permissions

docker run --rm -u $(id -u):$(id -g) --entrypoint /bin/bash -v ${PWD}:/data staphb/freyja:latest -c "freyja update"
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-zm48iova because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Traceback (most recent call last):
  File "/opt/conda/envs/freyja-env/bin/freyja", line 10, in <module>
    sys.exit(cli())
  File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
Downloading a new global tree
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/_cli.py", line 68, in update
    download_tree()
  File "/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/updates.py", line 13, in download_tree
    urllib.request.urlretrieve(url, treePath)
  File "/opt/conda/envs/freyja-env/lib/python3.10/urllib/request.py", line 251, in urlretrieve
    tfp = open(filename, 'wb')
PermissionError: [Errno 13] Permission denied: '/opt/conda/envs/freyja-env/lib/python3.10/site-packages/freyja/data/public-latest.all.masked.pb.gz'

Thank you!

Deprecation warnings

Hi will this warning impact my results?

freyja demix SRR18541076.freyja_variants.txt.tsv SRR18541076.freyja_depth.txt --output freyja.txt
building mix/depth matrices
/home/lenore/.local/lib/python3.10/site-packages/freyja-2021.10-py3.10.egg/freyja/sample_deconv.py:42: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
  mix = df.loc[keptInds, 'ALT_FREQ'].astype(float)
demixing

Plotting lineages fails with ValueError: could not convert string to float:

Plotting lineages fails for aggregate files containing empty lists:

S23.variants.tsv	[]	[]	[]	1.2068454558706715e-12
S36.variants.tsv	[]	[]	[]	8.99193526131271e-13
S20.variants.tsv	[]	[]	[]	6.665485300422211e-13

Error:

Traceback (most recent call last):
  File "/usr/local/bin/freyja", line 33, in <module>
    sys.exit(load_entry_point('freyja==2021.10', 'console_scripts', 'freyja')())
  File "/usr/local/lib/python3.7/site-packages/click-8.0.3-py3.7.egg/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click-8.0.3-py3.7.egg/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click-8.0.3-py3.7.egg/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click-8.0.3-py3.7.egg/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click-8.0.3-py3.7.egg/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/freyja-2021.10-py3.7.egg/freyja/_cli.py", line 170, in plot
    makePlot_simple(agg_df, lineages, output, colors0)
  File "/usr/local/lib/python3.7/site-packages/freyja-2021.10-py3.7.egg/freyja/utils.py", line 94, in makePlot_simple
    agg_df['linDict'] = prepLineageDict(agg_df)
  File "/usr/local/lib/python3.7/site-packages/freyja-2021.10-py3.7.egg/freyja/utils.py", line 42, in prepLineageDict
    for samp in agg_d0.index]
  File "/usr/local/lib/python3.7/site-packages/freyja-2021.10-py3.7.egg/freyja/utils.py", line 42, in <listcomp>
    for samp in agg_d0.index]
  File "/usr/local/lib/python3.7/site-packages/freyja-2021.10-py3.7.egg/freyja/utils.py", line 39, in <dictcomp>
    agg_d0['linDict'] = [{lin: float(abund) for lin, abund in
ValueError: could not convert string to float:

key error while running demix

Hello,

I am getting a strange error while running demix for my wastewater samples. First I ran freyja variants with a bam file were the primers were supposed to be trimmed using Dagon at Illumina basespace, and them demix without any problems. Looking at fastq QC results, with a very similar noise pattern in base composition in the first 20-22 bp in several samples, I realized that maybe the primers were not trimmed, so I trimmed them and all go well until demix, when I stared getting the key error below. Freyja was installed through conda with python3.10 on a CentOS server. The files that worked (EGP7*, without ivar primers timming) and that caused the error (None*) are attached at the end.

(freyja-env) [ricardo@sgi ~/SARS-COV-2/EG-P7]$ freyja demix None_var.txt None.txt
building mix/depth matrices
Traceback (most recent call last):
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 3361, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 76, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 2131, in pandas._libs.hashtable.Int64HashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 2140, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 1320

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/bin/freyja", line 10, in
sys.exit(cli())
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/freyja/_cli.py", line 58, in demix
covcut)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/freyja/sample_deconv.py", line 52, in build_mix_and_depth_arrays
.astype(float) for kI in muts}, name=fn)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/freyja/sample_deconv.py", line 52, in
.astype(float) for kI in muts}, name=fn)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/core/indexing.py", line 925, in getitem
return self._getitem_tuple(key)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/core/indexing.py", line 1100, in _getitem_tuple
return self._getitem_lowerdim(tup)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/core/indexing.py", line 838, in _getitem_lowerdim
section = self._getitem_axis(key, axis=i)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/core/indexing.py", line 1164, in _getitem_axis
return self._get_label(key, axis=axis)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/core/indexing.py", line 1113, in _get_label
return self.obj.xs(label, axis=axis)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/core/generic.py", line 3776, in xs
loc = index.get_loc(key)
File "/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 3363, in get_loc
raise KeyError(key) from err
KeyError: 1320

I checked the sample_deconv.py script, and when executing the commands with the data form the None* files, when executing line 51 it produced the same KeyError: 1320.

import pandas as pd
import re
import numpy as np
df = pd.read_csv("EGP7_trimmed_vars.tsv", sep='\t')
df_depth = pd.read_csv("EGP7_trimmed_dep.txt", sep='\t', header=None, index_col=1)
df_barcodes = pd.read_csv("/STORAGE/ricardo/anaconda3/envs/freyja-env/lib/python3.10/site-packages/freyja/data/usher_barcodes.csv", index_col=0)
df['mutName'] = df['REF'] + df['POS'].astype(str) + df['ALT']
df = df.drop_duplicates(subset='mutName')
df.set_index('mutName', inplace=True)
keptInds = set(muts) & set(df.index)
mix = df.loc[list(keptInds), 'ALT_FREQ'].astype(float)
mix.name = "EGP7_trimmed"
depths = pd.Series({kI: df_depth.loc[int(re.findall(r'\d+', kI)[0]), 3]
... .astype(float) for kI in muts}, name=fn)
Traceback (most recent call last):
File "/STORAGE/ricardo/anaconda3/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 2131, in pandas._libs.hashtable.Int64HashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 2140, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 1320

Changing the line from:
depths = pd.Series({kI: df_depth.loc[int(re.findall(r'\d+', kI)[0]), 3]
.astype(float) for kI in muts}, name=fn)

to, as the mutations are already defined in the mix file) :
depths = pd.Series({kI: df_depth.loc[int(re.findall(r'\d+', kI)[0]), 3]
.astype(float) for kI in mix.index}, name=fn)

made it work for the None* files, but when I ran freyja demix with this modification in the files EGP7*, the results, although similar, were not exactly the same, so I am not sure if the results after the modification can be trusted.

These are the results without the modification:
EGP7_f.tsv
summarized [('Omicron', 0.4644832999962632), ('Gamma', 0.3888890000000889), ('BA.5* [Omicron (BA.5.X)]', 0.08097169999519067), ('Other', 0.06465799598341625)]
lineages P.1 BA.2.23 XAH BA.5 BA.5.2.26 BF.6 BA.5.2.1 B.1.401 B.1.67 B.1.395 B.1.609 B.1.1 B.1.399 B.1 B.1.1.189 B.1.1.148 B.1.1.210 B.1.416.1 B.1.1.154 B.1.1.125
abundances 0.38888900 0.35424400 0.11023930 0.03866400 0.02592590 0.00819090 0.00819090 0.00519302 0.00519302 0.00519302 0.00519302 0.00519302 0.00519302 0.00519302 0.00519302 0.00519302 0.00519302 0.00519302 0.00409836 0.00343643
resid 15.014286271912404
coverage 73.62137578169414

And these are the results after the modification:
EGP7_f.tsv
summarized [('Omicron', 0.5301393001361678), ('Gamma', 0.3888889998310014), ('BA.5* [Omicron (BA.5.X)]', 0.07309767998754463), ('BQ.1* [Omicron (BQ.1.X)]', 0.0026041700001017457)]
lineages P.1.7 P.1.7.1 XV XJ XAC XAE XBE BF.11.5 BF.11.3 BF.11.4 BF.11 BF.11.1 BF.11.2 BF.24 BQ.1.4
abundances 0.19444450 0.19444450 0.17830900 0.17830900 0.08676065 0.08676065 0.02592590 0.00705128 0.00705128 0.00705128 0.00705128 0.00705128 0.00705128 0.00486408 0.00260417
resid 12.304529675192697
coverage 73.62137578169414

Thank you very much for your time,

Ricardo

EGP7_dp.txt
EGP7_var.txt
None_var.txt
None.txt

problem plot freyja v1.3.10 python 3.7

Hi, problem plot
freyja plot archivo-demixed.tsv --output archivo-demixed.png --lineages
does not show graph, sample coverage 22.05464334682139

colors for plots with day interval

Hello, Thanks for making this software available.

I am just trying out Freyja while trying to figure out how to analyze our covid ww sequence data. I noticed the plots for daily intervals only have a 10 color pallet that is being used, and if there are >10 lineages the colors repeat. The plots for just lineage abundance or monthly intervals have more colors. I think it might be the same 10 color pallet but with 2 shades per color. Is it possible to add additional colors to the plot when its plotted by day intervals?

Example plot with daily interval with 14 day window, colors repeat:
freyja plot Freyja/freyja_aggregate.tsv --output Freyja/output_summary_plot_by_day.pdf --times Freyja/sample_dates.csv --interval D --windowsize 14
output_summary_plot_by_14day.pdf

Example plot with monthly interval, no repeating colors:
freyja plot Freyja/freyja_aggregate.tsv --output Freyja/output_lineages_plot_by_date.pdf --times Freyja/sample_dates.csv --interval MS --lineages
output_lineages_plot_by_month.pdf

Thanks!

No coverage information in output

Hi Josh,

I have just updated Freyja today (version 1.3.5; from https://github.com/andersen-lab/Freyja/releases) in order to get coverage information but somehow I am not getting coverage information in the output.

Could you please let me know where is the problem?

Thanks and regards,
Wasim

Running Freyja in Windows Subsystem for Linux: KeyError: 'REF'

I have been using Freyja on different sample sets, some single source and some from wastewater.

The single source sample data ran through Freyja with no issues.
The wastewater samples are giving me this error when running freyja demix

freyja demix /mnt/c/DataDrive/IRandD_ExactID_Covid/OCWN_fastq/Week_1/Wuhan_bam/SNP-Analyze_Conneaut_0411_Week1_SwiftSarsCoV2_BWA_SARS_Wuhan_sort_freyja_variants.tsv /mnt/c/DataDrive/IRandD_ExactID_Covid/
OCWN_fastq/Week_1/Wuhan_bam/SNP-Analyze_Conneaut_0411_Week1_SwiftSarsCoV2_BWA_SARS_Wuhan_sort_freyja_depths.txt --output /mnt/c/DataDrive/IRandD_ExactID_Covid/OCWN_fastq/Week_1/Wuhan_bam/SNP-Analyze_Conneaut_0411_Week1_SwiftSarsCoV2_BWA_
SARS_Wuhan_sort_freyja_demix.txt

building mix/depth matrices
Traceback (most recent call last):
  File "/home/minardsmitha/miniconda3/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'REF'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/minardsmitha/miniconda3/bin/freyja", line 10, in <module>
    sys.exit(cli())
  File "/home/minardsmitha/miniconda3/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/minardsmitha/miniconda3/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/minardsmitha/miniconda3/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/minardsmitha/miniconda3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/minardsmitha/miniconda3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/minardsmitha/miniconda3/lib/python3.9/site-packages/freyja/_cli.py", line 55, in demix
    mix, depths_, cov = build_mix_and_depth_arrays(variants, depths, muts,
  File "/home/minardsmitha/miniconda3/lib/python3.9/site-packages/freyja/sample_deconv.py", line 45, in build_mix_and_depth_arrays
    df['mutName'] = df['REF'] + df['POS'].astype(str) + df['ALT']
  File "/home/minardsmitha/miniconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3505, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/minardsmitha/miniconda3/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3623, in get_loc
    raise KeyError(key) from err
KeyError: 'REF'

I tried reinstalling Freyja and tried to determine if my issue is the same as issue #50 but I haven't been able to correct the problem.
Some help figuring this out would be great.

aggregate command exiting with UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte`

Running the aggregate command freyja aggregate AM/ --output AM/AM.aggregate.tsv on a folder containing out files created with freyja:

Traceback (most recent call last): File "/usr/local/bin/freyja", line 33, in <module> sys.exit(load_entry_point('freyja==2021.10', 'console_scripts', 'freyja')()) File "/usr/local/lib/python3.10/site-packages/click-8.0.3-py3.10.egg/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click-8.0.3-py3.10.egg/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.10/site-packages/click-8.0.3-py3.10.egg/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.10/site-packages/click-8.0.3-py3.10.egg/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.10/site-packages/click-8.0.3-py3.10.egg/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/freyja-2021.10-py3.10.egg/freyja/_cli.py", line 149, in aggregate df_demix = agg(results) File "/usr/local/lib/python3.10/site-packages/freyja-2021.10-py3.10.egg/freyja/utils.py", line 10, in agg allResults = [pd.read_csv(results+fn, skipinitialspace=True, sep='\t', File "/usr/local/lib/python3.10/site-packages/freyja-2021.10-py3.10.egg/freyja/utils.py", line 10, in <listcomp> allResults = [pd.read_csv(results+fn, skipinitialspace=True, sep='\t', File "/usr/local/lib/python3.10/site-packages/pandas/util/_decorators.py", line 311, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv return _read(filepath_or_buffer, kwds) File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 575, in _read parser = TextFileReader(filepath_or_buffer, **kwds) File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 933, in __init__ self._engine = self._make_engine(f, self.engine) File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1231, in _make_engine return mapping[engine](f, **self.options) File "/usr/local/lib/python3.10/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 75, in __init__ self._reader = parsers.TextReader(src, **kwds) File "pandas/_libs/parsers.pyx", line 544, in pandas._libs.parsers.TextReader.__cinit__ File "pandas/_libs/parsers.pyx", line 633, in pandas._libs.parsers.TextReader._get_header File "pandas/_libs/parsers.pyx", line 847, in pandas._libs.parsers.TextReader._tokenize_rows File "pandas/_libs/parsers.pyx", line 1952, in pandas._libs.parsers.raise_parser_error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte

Which lineages are in my current version?

I ran "freyja update" within the past few days, but I am curious to know which lineages/variants are currently within my database.

I recently ran some samples and had many variants called as "omicron", when I suspect these may instead be BA.4 or BA.5. However, I am not sure if BA.4 and/or BA.5 could be called by my current reference set.

Thanks so much!

andersen-lab / freyja Goto Github PK

freyja's People

Contributors

Stargazers

Watchers

Forkers

freyja's Issues

Name Version Build Channel

Recommend Projects

Recommend Topics

Recommend Org