Linear regression about hail HOT 3 CLOSED

From @jbloom22 on October 14, 2015 1:26

core statistical test, though i reworked the formulas:
https://en.wikipedia.org/wiki/Simple_linear_regression#Normality_assumption

from hail.

From @jbloom22 on October 28, 2015 5:53

@cseed ready for your review

from hail.

From @jbloom22 on October 28, 2015 21:2

Hail runtime for linreg with 10 PCs on profile.vds: 56s, 59s, 58s.
Hail runtime for variantqc on profile.vds: 35s, 34s, 35s.
Plink runtime for linreg with 10 PCs on profile.vds: 13s, 13s, 13s.
Hail runtime (8 cores) for linreg with 10 PCs on profile.vds: 23s, 25s, 23s.

LINREG:
/Users/jbloom/k3/build/install/k3/bin/k3 read -i ~/data/profile.vds linreg -f ~/data/profile.fam -c ~/data/profile.cov -o ~/data/profile.linreg

read: 1407.415486
linreg: 58336.701622

VARIANTQC:
/Users/jbloom/k3/build/install/k3/bin/k3 read -i ~/data/profile.vds variantqc -o ~/data/profile.variantqc

read: 1417.763771
variantqc: 35466.355219

PLINK:
create bed/bim/fam:
./plink --vcf ~/data/profile.vcf.bgz

run regression:
time ./plink --bfile plink --double-id --pheno ~/data/profile.pheno
--allow-no-sex --covar ~/data/profile.covar --linear --out
~/data/plinkTest

PLINK v1.90b3w 64-bit (3 Sep 2015) https://www.cog-genomics.org/plink2
(C) 2005-2015 Shaun Purcell, Christopher Chang GNU General Public License v3
Logging to /Users/Jon/data/plinkTest.log.
Options in effect:
--allow-no-sex
--bfile plink
--covar /Users/Jon/data/profile.covar
--double-id
--linear
--out /Users/Jon/data/plinkTest
--pheno /Users/Jon/data/profile.pheno
16384 MB RAM detected; reserving 8192 MB for main workspace.
24885 variants loaded from .bim file.
2535 people (0 males, 0 females, 2535 ambiguous) loaded from .fam.
Ambiguous sex IDs written to /Users/Jon/data/plinkTest.nosex .
2535 phenotype values present after --pheno.
Using 1 thread.
Warning: This run includes BLAS/LAPACK linear algebra operations which
currently disregard the --threads limit. If this is problematic, you
may want to recompile against single-threaded BLAS/LAPACK.
--covar: 10 covariates loaded.
Before main variant filters, 2535 founders and 0 nonfounders present.
Calculating allele frequencies... done.
Total genotyping rate is 0.907692.
24885 variants and 2535 people pass filters and QC.
Phenotype data is quantitative.
Writing linear model association results to
/Users/Jon/data/plinkTest.assoc.linear ... done.

real 0m13.167s
user 0m13.071s
sys 0m0.080s

from hail.