mt-digital / badscience-solutions Goto Github PK

From Leo Tiokhin: "This strongly selects against individuals who have phenotypes that aren't in the most extreme values and thereby selects for individuals who do anything possible to get in the top percentiles of phenotype values. For instance, if I'm in the bottom 20% of false alpha levels (e.g. 0.03 or something) but 15% of the population is composed of individuals who have alpha levels of 0.02, then even though I'm pretty rigorous relative to most people, I only get a grant 0.85^9= 23% of the time."

R code snippet showing differences, with resulting plots pasted below (all three also thanks to Leo):

#wright-fisher
nums <- 0:100
fitness <- 0:100
fitness2 <- fitness/sum(fitness)
nextgen <- sample(nums, size=1e4, replace=TRUE, prob=fitness2)
hist(nextgen, main = paste("Wright-Fisher. Mean = ", mean(nextgen), "SD = ", round(sd(nextgen), 3)))

#paul version
nextgen_paul <- rep(NA, 1e4)
d <- 10
for(i in 1:1e4){
  paul_sample <- sample(nums, size = d, replace = FALSE)
  indexes <- which(nums %in% paul_sample)
  fitnesses <- fitness2[indexes]
  selected_indv <- which.max(fitnesses)
  nextgen_paul[i] <- nums[indexes[selected_indv]]
}
hist(nextgen_paul, main = paste("Paul Implementation. Mean = ", mean(nextgen_paul), "SD = ", round(sd(nextgen_paul), 3)))

It has a parameter X, such that a proportion X of the time, funds are allocated to the most rigorous lab, and otherwise (1-X of the time) are allocated at random. We can vary X systematically, for a few values of r=p and G to consider how good the funders need to be to make a difference.

Rank all proposals in terms of rigor (alpha). Eliminate any lab with an alpha less than some threshold Z. Randomly select one of the remaining proposals. We can then vary Z to see how good funding reviewers need to be for it to matter.