Giter Club home page Giter Club logo

Comments (9)

ACEnglish avatar ACEnglish commented on May 27, 2024

Could you check to ensure that you have a SAMPLE field in your header (e.g. #CHROM POS ID...FORMAT SAMPLE)?

from truvari.

Tintest avatar Tintest commented on May 27, 2024

Hello,

Sorry for taking so long to answer, summer break and other projects ;)

You were right, my vcf file was lacking a sample name. But if I'm not mistaken, FORMAT and SAMPLE ID aren't in the vcf specification, they are supplementary fields for genotyped gvcf ? Does it mean Truvari works only for gvcf ? I'm trying to use it on a truth vcf and a vcf outputed by Sniffles, with default parameters, which means, it's not genotyped. Can it work anyway ?

Unfortunately, after fixing my sample_id problem, I got another error

truvari -b NA19240.GRCh38.variant_call.vcf.gz -c ERR3219853_minimap2_ont.vcf.gz -f ../hg38.fasta -o test
2019-09-03 16:28:25,577 [INFO] Params:
{
    "sizemax": 50000,
    "reference": "../hg38.fasta",
    "noprog": false,
    "multimatch": false,
    "pctsize": 0.7,
    "cSample": null,
    "includebed": null,
    "no_ref": false,
    "passonly": false,
    "pctsim": 0.7,
    "pctovl": 0.0,
    "comp": "ERR3219853_minimap2_ont.vcf.gz",
    "refdist": 500,
    "base": "NA19240.GRCh38.variant_call.vcf.gz",
    "giabreport": false,
    "sizefilt": 30,
    "typeignore": false,
    "gtcomp": false,
    "debug": false,
    "output": "test",
    "bSample": null,
    "sizemin": 50
}
2019-09-03 16:28:25,593 [INFO] Creating call interval tree for overlap search
Traceback (most recent call last):
  File "/home/tintest/miniconda3/bin/truvari", line 1047, in <module>
    run(sys.argv[1:])
  File "/home/tintest/miniconda3/bin/truvari", line 734, in run
    span_lookup, num_entries_b, cmp_entries = make_interval_tree(vcf_comp, args.sizefilt, args.sizemax, args.passonly)
  File "/home/tintest/miniconda3/bin/truvari", line 83, in make_interval_tree
    sz = get_vcf_entry_size(entry)
  File "/home/tintest/miniconda3/bin/truvari", line 269, in get_vcf_entry_size
    size = abs(entry.INFO["SVLEN"][0])
TypeError: bad operand type for abs(): 'str'

Such as indicated in #24, I removed every variant with no SVLEN (they were all CNV), but no improvement.

Can you help me ?

Regards.

from truvari.

ACEnglish avatar ACEnglish commented on May 27, 2024

While it's correct that FORMAT/SAMPLE aren't required for VCFs, Truvari assumes that at least a single sample is present in the VCF. You can simply 'fake' the presence of a sample by adding a FORMAT/SAMPLE line of "GT\t." to the end of every variant line.

As for your TypeError, the header of the comparison vcf probably doesn't have SVLEN defined as an Integer (e.g. ##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">) but instead a String.

from truvari.

Tintest avatar Tintest commented on May 27, 2024

Thank you !

I managed to get Truvari running by modifying the header and adding a fake format and genotyping to every variant line.

However, I am experiencing unusual behavior with your tool.

2019-09-04 18:15:13,337 [INFO] Running /home/tintest/miniconda3/bin/truvari -b NA19240.GRCh38.variant_call.vcf.gz -c NA19240.GRCh38.variant_call.vcf.gz -f ../hg38.fasta -o test5 -s 0 --sizemax 100000000000000000000000000000000000000000
2019-09-04 18:15:13,338 [INFO] Params:
{
    "sizemax": 100000000000000000000000000000000000000000,
    "reference": "../hg38.fasta",
    "noprog": false,
    "multimatch": false,
    "pctsize": 0.7,
    "cSample": null,
    "includebed": null,
    "no_ref": false,
    "passonly": false,
    "pctsim": 0.7,
    "pctovl": 0.0,
    "comp": "NA19240.GRCh38.variant_call.vcf.gz",
    "refdist": 500,
    "base": "NA19240.GRCh38.variant_call.vcf.gz",
    "giabreport": false,
    "sizefilt": 30,
    "typeignore": false,
    "gtcomp": false,
    "debug": false,
    "output": "test5",
    "bSample": null,
    "sizemin": 0
}
2019-09-04 18:15:13,355 [INFO] Creating call interval tree for overlap search
2019-09-04 18:15:21,877 [INFO] 38091 call variants in total
2019-09-04 18:15:21,878 [INFO] 38091 call variants within size range (30, 100000000000000000000000000000000000000000)
2019-09-04 18:15:26,405 [INFO] 38091 base variants
2019-09-04 18:15:26,441 [INFO] Matching base to calls
 [Elapsed Time: 0:00:44 38091/38091] |##################################################################################################################################################################################################################################################################################################################################################| (Time:  0:00:44)
2019-09-04 18:16:10,469 [INFO] Results peek: 35873 TP-base 2194 FN 94.24% Recall
2019-09-04 18:16:10,469 [INFO] Parsing FPs from calls
 [Elapsed Time: 0:00:05 38091/38091] |##################################################################################################################################################################################################################################################################################################################################################| (Time:  0:00:05)
{
    "TP-base": 35873,
    "TP-call": 35897,
    "FP": 0,
    "FN": 2194,
    "precision": 1.0,
    "recall": 0.9423647778916121,
    "f1": 0.9703272923992426,
    "base cnt": 38091,
    "call cnt": 35873,
    "base size filtered": 0,
    "call size filtered": 0,
    "base gt filtered": 0,
    "call gt filtered": 0,
    "TP-call_TP-gt": 35897,
    "TP-call_FP-gt": 0,
    "TP-base_TP-gt": 35873,
    "TP-base_FP-gt": 0,
    "gt_precision": 1.0,
    "gt_recall": 0.9423647778916121,
    "gt_f1": 0.9703272923992426
}
2019-09-04 18:16:15,771 [INFO] Stats: {
    "TP-base": 35873,
    "TP-call": 35897,
    "FP": 0,
    "FN": 2194,
    "precision": 1.0,
    "recall": 0.9423647778916121,
    "f1": 0.9703272923992426,
    "base cnt": 38091,
    "call cnt": 35873,
    "base size filtered": 0,
    "call size filtered": 0,
    "base gt filtered": 0,
    "call gt filtered": 0,
    "TP-call_TP-gt": 35897,
    "TP-call_FP-gt": 0,
    "TP-base_TP-gt": 35873,
    "TP-base_FP-gt": 0,
    "gt_precision": 1.0,
    "gt_recall": 0.9423647778916121,
    "gt_f1": 0.9703272923992426
}
2019-09-04 18:16:15,772 [INFO] Finished

Why when I am running my truth VCF against itself, it does not report recall equal to 1.0. Is it expected ? Also, even if I set the minimum SV calling size to 0, it does not process variants smaller than 30bp, but it seems they are taken into account for recall and precision calculation. By removing SV <30bp from my truth set, I achieved to get precision equal to 1.0, but I could'nt get more than 0.94 in recall.

Also, comparing my sample VCF to a truth set work only in a single "combinaison"

2019-09-04 18:20:24,633 [INFO] Running /home/tintest/miniconda3/bin/truvari -b NA19240.GRCh38.variant_call.vcf.gz -c ERR3219853_minimap2_ont.vcf.gz -f ../hg38.fasta -o test6 -s 0 --sizemax 100000000000000000000000000000000000000000
2019-09-04 18:20:24,634 [INFO] Params:
{
    "sizemax": 100000000000000000000000000000000000000000,
    "reference": "../hg38.fasta",
    "noprog": false,
    "multimatch": false,
    "pctsize": 0.7,
    "cSample": null,
    "includebed": null,
    "no_ref": false,
    "passonly": false,
    "pctsim": 0.7,
    "pctovl": 0.0,
    "comp": "ERR3219853_minimap2_ont.vcf.gz",
    "refdist": 500,
    "base": "NA19240.GRCh38.variant_call.vcf.gz",
    "giabreport": false,
    "sizefilt": 30,
    "typeignore": false,
    "gtcomp": false,
    "debug": false,
    "output": "test6",
    "bSample": null,
    "sizemin": 0
}
2019-09-04 18:20:24,742 [INFO] Creating call interval tree for overlap search
2019-09-04 18:20:29,379 [INFO] 18011 call variants in total
2019-09-04 18:20:29,379 [INFO] 17725 call variants within size range (30, 100000000000000000000000000000000000000000)
2019-09-04 18:20:35,749 [INFO] 38091 base variants
2019-09-04 18:20:35,795 [INFO] Matching base to calls
 [Elapsed Time: 0:03:30 38091/38091] |##################################################################################################################################################################################################################################################################################################################################################| (Time:  0:03:30)
2019-09-04 18:24:05,963 [WARNING] No TP or FP calls in base!
2019-09-04 18:24:05,963 [INFO] Parsing FPs from calls
 [Elapsed Time: 0:00:04 18011/18011] |##################################################################################################################################################################################################################################################################################################################################################| (Time:  0:00:04)
{
    "TP-base": 0,
    "TP-call": 0,
    "FP": 18011,
    "FN": 38091,
    "precision": 0,
    "recall": 0,
    "f1": "NaN",
    "base cnt": 38091,
    "call cnt": 18011,
    "base size filtered": 0,
    "call size filtered": 0,
    "base gt filtered": 0,
    "call gt filtered": 0,
    "TP-call_TP-gt": 0,
    "TP-call_FP-gt": 0,
    "TP-base_TP-gt": 0,
    "TP-base_FP-gt": 0,
    "gt_precision": 0,
    "gt_recall": 0,
    "gt_f1": "NaN"
}
2019-09-04 18:24:10,872 [INFO] Stats: {
    "TP-base": 0,
    "TP-call": 0,
    "FP": 18011,
    "FN": 38091,
    "precision": 0,
    "recall": 0,
    "f1": "NaN",
    "base cnt": 38091,
    "call cnt": 18011,
    "base size filtered": 0,
    "call size filtered": 0,
    "base gt filtered": 0,
    "call gt filtered": 0,
    "TP-call_TP-gt": 0,
    "TP-call_FP-gt": 0,
    "TP-base_TP-gt": 0,
    "TP-base_FP-gt": 0,
    "gt_precision": 0,
    "gt_recall": 0,
    "gt_f1": "NaN"
}
2019-09-04 18:24:10,873 [INFO] Finished

2019-09-04 18:24:43,769 [INFO] Running /home/tintest/miniconda3/bin/truvari -c NA19240.GRCh38.variant_call.vcf.gz -b ERR3219853_minimap2_ont.vcf.gz -f ../hg38.fasta -o test7 -s 0 --sizemax 100000000000000000000000000000000000000000
2019-09-04 18:24:43,770 [INFO] Params:
{
    "sizemax": 100000000000000000000000000000000000000000,
    "reference": "../hg38.fasta",
    "noprog": false,
    "multimatch": false,
    "pctsize": 0.7,
    "cSample": null,
    "includebed": null,
    "no_ref": false,
    "passonly": false,
    "pctsim": 0.7,
    "pctovl": 0.0,
    "comp": "NA19240.GRCh38.variant_call.vcf.gz",
    "refdist": 500,
    "base": "ERR3219853_minimap2_ont.vcf.gz",
    "giabreport": false,
    "sizefilt": 30,
    "typeignore": false,
    "gtcomp": false,
    "debug": false,
    "output": "test7",
    "bSample": null,
    "sizemin": 0
}
2019-09-04 18:24:43,788 [INFO] Creating call interval tree for overlap search
2019-09-04 18:24:49,809 [INFO] 38091 call variants in total
2019-09-04 18:24:49,809 [INFO] 38091 call variants within size range (30, 100000000000000000000000000000000000000000)
2019-09-04 18:24:52,336 [INFO] 18011 base variants
2019-09-04 18:24:52,370 [INFO] Matching base to calls
 [Elapsed Time: 0:09:33 18011/18011] |##################################################################################################################################################################################################################################################################################################################################################| (Time:  0:09:33)
2019-09-04 18:34:26,025 [INFO] Results peek: 1981 TP-base 16030 FN 11.00% Recall
2019-09-04 18:34:26,025 [INFO] Parsing FPs from calls
 [Elapsed Time: 0:01:00 38091/38091] |##################################################################################################################################################################################################################################################################################################################################################| (Time:  0:01:00)
{
    "TP-base": 1981,
    "TP-call": 1981,
    "FP": 36013,
    "FN": 16030,
    "precision": 0.05213981154919198,
    "recall": 0.10998834045860863,
    "f1": 0.0707436835996786,
    "base cnt": 18011,
    "call cnt": 37994,
    "base size filtered": 0,
    "call size filtered": 0,
    "base gt filtered": 0,
    "call gt filtered": 0,
    "TP-call_TP-gt": 1981,
    "TP-call_FP-gt": 0,
    "TP-base_TP-gt": 1981,
    "TP-base_FP-gt": 0,
    "gt_precision": 0.05213981154919198,
    "gt_recall": 0.10998834045860863,
    "gt_f1": 0.0707436835996786
}
2019-09-04 18:35:28,774 [INFO] Stats: {
    "TP-base": 1981,
    "TP-call": 1981,
    "FP": 36013,
    "FN": 16030,
    "precision": 0.05213981154919198,
    "recall": 0.10998834045860863,
    "f1": 0.0707436835996786,
    "base cnt": 18011,
    "call cnt": 37994,
    "base size filtered": 0,
    "call size filtered": 0,
    "base gt filtered": 0,
    "call gt filtered": 0,
    "TP-call_TP-gt": 1981,
    "TP-call_FP-gt": 0,
    "TP-base_TP-gt": 1981,
    "TP-base_FP-gt": 0,
    "gt_precision": 0.05213981154919198,
    "gt_recall": 0.10998834045860863,
    "gt_f1": 0.0707436835996786
}
2019-09-04 18:35:28,775 [INFO] Finished

I would have rather believed that base is the truth VCF, but it only works the other way.

Is this also an expected behavior ?

Thank you, regards.

from truvari.

ACEnglish avatar ACEnglish commented on May 27, 2024

For the size things you're seeing, please reference the main docs.
As for the different behavior when changing base/comp - could you check to make sure the tabix indexes for the two vcfs is present and correct.

from truvari.

Tintest avatar Tintest commented on May 27, 2024

I regenerated indexes with tabix, but nothing changed when I switched base and comp. It still works one way which is strange.

I reran my truth VCF against itself and I got a better recall, thanks to --sizefilt, but still not 1. I got every time a bunch of false negatives and I don't know why. I did the same experiment with my sample, and I only got 2 FN.

2019-09-05 16:58:15,971 [INFO] Running /home/tintest/miniconda3/bin/truvari -S 0 -s 30 --sizemax 100000000000000000 -c ERR3219853_minimap2_ont.vcf.gz -b ERR3219853_minimap2_ont.vcf.gz -f ../hg38.fasta -o test
2019-09-05 16:58:15,971 [INFO] Params:
{
    "base": "ERR3219853_minimap2_ont.vcf.gz",
    "comp": "ERR3219853_minimap2_ont.vcf.gz",
    "output": "test",
    "reference": "../hg38.fasta",
    "giabreport": false,
    "debug": false,
    "noprog": false,
    "refdist": 500,
    "pctsim": 0.7,
    "pctsize": 0.7,
    "pctovl": 0.0,
    "typeignore": false,
    "gtcomp": false,
    "bSample": null,
    "cSample": null,
    "sizemin": 30,
    "sizefilt": 0,
    "sizemax": 100000000000000000,
    "passonly": false,
    "no_ref": false,
    "includebed": null,
    "multimatch": false
}
2019-09-05 16:58:15,977 [INFO] Creating call interval tree for overlap search
2019-09-05 16:58:17,920 [INFO] 18011 call variants in total
2019-09-05 16:58:17,921 [INFO] 18011 call variants within size range (0, 100000000000000000)
2019-09-05 16:58:19,484 [INFO] 18011 base variants
2019-09-05 16:58:19,490 [INFO] Matching base to calls
 [Elapsed Time: 0:01:41 18011/18011] |########################################################################################################################| (Time:  0:01:41)
2019-09-05 17:00:01,387 [INFO] Results peek: 17723 TP-base 2 FN 99.99% Recall
2019-09-05 17:00:01,387 [INFO] Parsing FPs from calls
 [Elapsed Time: 0:00:01 18011/18011] |########################################################################################################################| (Time:  0:00:01)
{
    "TP-base": 17723,
    "TP-call": 17723,
    "FP": 0,
    "FN": 2,
    "precision": 1.0,
    "recall": 0.9998871650211566,
    "f1": 0.9999435793274657,
    "base cnt": 17725,
    "call cnt": 17723,
    "base size filtered": 286,
    "call size filtered": 286,
    "base gt filtered": 0,
    "call gt filtered": 0,
    "TP-call_TP-gt": 17723,
    "TP-call_FP-gt": 0,
    "TP-base_TP-gt": 17723,
    "TP-base_FP-gt": 0,
    "gt_precision": 1.0,
    "gt_recall": 0.9998871650211566,
    "gt_f1": 0.9999435793274657
}
2019-09-05 17:00:03,084 [INFO] Stats: {
    "TP-base": 17723,
    "TP-call": 17723,
    "FP": 0,
    "FN": 2,
    "precision": 1.0,
    "recall": 0.9998871650211566,
    "f1": 0.9999435793274657,
    "base cnt": 17725,
    "call cnt": 17723,
    "base size filtered": 286,
    "call size filtered": 286,
    "base gt filtered": 0,
    "call gt filtered": 0,
    "TP-call_TP-gt": 17723,
    "TP-call_FP-gt": 0,
    "TP-base_TP-gt": 17723,
    "TP-base_FP-gt": 0,
    "gt_precision": 1.0,
    "gt_recall": 0.9998871650211566,
    "gt_f1": 0.9999435793274657
}
2019-09-05 17:00:03,085 [INFO] Finished

Here are the 2 variants reported as FN :

1       143241950       935     N       <DEL>   .       PASS    CHR2=1;END=143242115;RE=331;IMPRECISE;SVLEN=165;SVMETHOD=Snifflesv1.0.11;SVTYPE=DEL;STD_quant_start=18.946016;STD_quant_stop=18.036663;Kurtosis_quant_start=3.152146;Kurtosis_quant_stop=1.392118;SUPTYPE=AL;STRANDS=+-;NumNeighbors=2;NumThresholdNeighbors=0;MatchId=979        GT:DR:DV        ./.:.:331
22      18421873        16638   N       <INV>   .       PASS    CHR2=22;END=18896991;RE=21;PRECISE;SVLEN=475118;SVMETHOD=Snifflesv1.0.11;SVTYPE=INV;STD_quant_start=0.0;STD_quant_stop=0.316228;Kurtosis_quant_start=1.892036;Kurtosis_quant_stop=5.868685;SUPTYPE=SR;STRANDS=++;NumNeighbors=1;NumThresholdNeighbors=0;MatchId=16924     GT:DR:DV        ./.:.:21

Have you any idea why ?

from truvari.

ACEnglish avatar ACEnglish commented on May 27, 2024

Weird. Could you send me the vcfs in question for debugging? Namely ERR3219853_minimap2_ont.vcf.gz and NA19240.GRCh38.variant_call.vcf.gz

from truvari.

Tintest avatar Tintest commented on May 27, 2024

Here they are :

https://filesender.renater.fr/?s=download&token=10db0378-46d6-c461-adf4-2459fed55e62&lang=en

Thank you.

from truvari.

ACEnglish avatar ACEnglish commented on May 27, 2024

For brevity - NA19240.GRCh38.variant_call.vcf.gz == 'N' and ERR3219853_minimap2_ont.vcf.gz == 'E'

  1. Comparing NtoE output against EtoN gives the same results with parameters:
{
    "base": "ERR3219853_minimap2_ont.vcf.gz",
    "comp": "NA19240.GRCh38.variant_call.vcf.gz",
    "output": "EtoN",
    "reference": "/reference/grch38_noalt_decoy/source.fasta",
    "giabreport": false,
    "debug": false,
    "noprog": false,
    "refdist": 500,
    "pctsim": 0.7,
    "pctsize": 0.7,
    "pctovl": 0.0,
    "typeignore": false,
    "gtcomp": false,
    "bSample": null,
    "cSample": null,
    "sizemin": 30,
    "sizefilt": 0,
    "sizemax": 100000000000000000,
    "passonly": false,
    "no_ref": false,
    "includebed": null,
    "multimatch": false
}

Even with switching the Base/Comp parameters, I saw:

    "TP-base": 0,
    "TP-call": 0,
    "FP": 38091,
    "FN": 17725,

With FP/FN being swapped, of course.

I tried again to replicate the NtoE != EtoN with your previous parameters:

truvari -c NA19240.GRCh38.variant_call.vcf.gz -b ERR3219853_minimap2_ont.vcf.gz -f ../hg38.fasta -o test7 -s 0 --sizemax 100000000000000000000000000000000000000000
truvari -b NA19240.GRCh38.variant_call.vcf.gz -c ERR3219853_minimap2_ont.vcf.gz -f ../hg38.fasta -o test6 -s 0 --sizemax 100000000000000000000000000000000000000000

The only thing that changed was the FP/FN counts.

    "TP-base": 0,
    "TP-call": 0,
    "FP": 38346,
    "FN": 18011,

Therefore I'm unable to replicate this problem. The last possibility I can think of is our versions of truvari may be different? I'm using the lastet v1.2.

  1. The calls in N.vcf are not sequence resolved. Your numbers are altered because the sequence '<DEL>' was being compared with real alt-sequences, thus creating lower sequence similarity below the default threshold of 0.7. I specified --pctsim 0 and was able to see in EtoN
    "TP-base": 9619,
    "TP-call": 9619,
    "FP": 28070,
    "FN": 8106,
    "precision": 0.25522035607206345,
    "recall": 0.5426798307475318,

Could you reanalyze your files again with these two commands?

truvari -S 0 -s 30 --sizemax 100000000000000000 -b ERR3219853_minimap2_ont.vcf.gz -c NA19240.GRCh38.variant_call.vcf.gz -f ../hg38.fasta -o EtoN_pctsim0 --pctsim 0
truvari -S 0 -s 30 --sizemax 100000000000000000 -c ERR3219853_minimap2_ont.vcf.gz -b NA19240.GRCh38.variant_call.vcf.gz -f ../hg38.fasta -o NtoE_pctsim0 --pctsim 0

(reference updated to paths provided from your logs)
Additionally, it appears you could possibly replace the ALTs with the INFO/SEQ if you wanted to check for sequence simiarlity.

  1. When running EtoE, the two 'FNs' arose because the events are 100% redundant. To keep track of matches, we keep a hash of keys representing vcf entries. Analyzing your E.vcf shows 2 copies of the chr1 and chr22 events that would collide. I'll think about a possible work around and if it is necessary for truvari to handle this edge case or the documentation should advise a pre-processing de-dupping of vcfs.
bcftools query -f "%CHROM %POS %INFO/END %REF %ALT\n" ERR3219853_minimap2_ont.vcf.gz | sort | uniq -c | sort -nr | head
      2 22 18421873 18896991 N <INV>
      2 1 143241950 143242115 N <DEL>
      1 Y 56888109 56888408 N TTTTTTTTTTGAGACGGAGTCTCGCTCTGTCGGCCAGGCTGAGTGCAGTGGCGGGATCTCGCTCACTGCAAGCTCCGCCTCCCGGGTTCACGCCATTCTCCTGCCTCAGCTTCCCAAGTAGCTGGGACTACAGGCACCCGCCACTACGCCTGGCTAATTTTTTTGTATTTTTAGTAGAGACGGGGTTTCACCGCTTTAGCCGGGATGGTCTCGATCTCCTGACCTCGTGATCCGCCTGCCTCGGCCTCCCAAAGTGCTGGGATTGCGGCGTGACACGCGCCTGCCTGATGGTAGTC
      ...

from truvari.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.