ropensci / drat Goto Github PK

View Code? Open in Web Editor NEW

3.0 4.0 2.0 1.5 GB

drat repository for nightly builds of ropensci packages and common dependencies

Home Page: http://packages.ropensci.org/

License: Other

R 57.71% HTML 34.74% Shell 7.55%

drat

drat's Introduction

rOpenSci drat Repository

Welcome to the rOpenSci drat repository. This repository contains the latest nightly builds from the master branch of all rOpenSci packages currently on GitHub, along with the development versions of common dependencies, which include many packages from Hadley Wickham.

This allows users to install development versions of our software without specialized functions such as install_github(), allows dependencies not hosted on CRAN to still be resolved automatically, and permits the use of update.packages().

Quick Start

To use, simply add packages.ropensci.org to your existing list of R repos, such as:

options(repos = c("http://packages.ropensci.org", getOption("repos")))

(If you don't have any default CRAN mirrors selected yet by getOption("repos"), you may want to add one now). You can also include this line in specific install.packages() requests:

install.packages("taxize", repos = c("http://packages.ropensci.org", "http://cran.rstudio.com"))

Details

This is made possible by the excellent tools provided in Dirk Eddelbuettel's drat package and Rich FitzJohn's drat.builder. Nightly builds are performed using CircleCi and packages are served through an Amazon S3 static site. Configuration details and scripts necessary for this can be found in our GitHub repo, ropensci/drat. The list of rOpenSci packages included in on the nightly builds of this repository is automatically generated using ropensci/ropkgs. Once a package has been onboarded to our domain, there is no need to manually add it here to ensure it is included and updated. The list of third-party packages provided by this repo is found in packages.txt.

This includes the following files:

circle.yml CI file telling Circle how to build packages (running build.R) and deploy them (by running deploy_S3.R).
build.R Main script file for generating the drat repo.
deploy_s3.R deploy to Amazon S3 using cloudyr/aws.s3 package (alpha).
parse_s3_logs.R a script to parse Amazon S3 download logs into the same anonomous download summary csv format provided by RStudio's own CRAN mirror.
packages.txt A plain-text list of third party dependencies provided by the package. Use this script to add additional packages to the repo that are not hosted on rOpenSci GitHub account.

Other files:

ropensci.R a script used by build.R which uses ropkgs to generate a list of all ropensci packages to be added to the drat repo (via writing an ropensci.txt file)
packages.json, ropensci.json metadata files created by drat.builder to avoid rebuilding packages with no new commits.

Please report any issues here.

License: BSD-2

drat's People

Contributors

Stargazers

Watchers

Forkers

ilarischeinin jimsforks

drat's Issues

Problem with ropensci drat url

@cboettig I keep getting

Error in install.packages : Line starting '<html> ...' is malformed!

After any use of our drat repo, e.g,

options(repos = c("http://packages.ropensci.org", "https://cran.rstudio.com/"))
install.packages("gender")

gives the above

Unable to install 'genderdata' from this website

The attempt to install the 'genderdata' package from this website failed,
following the message I received when I tried to use the function gender_df()
in the 'gender' package:

> ...
The genderdata package needs to be installed.
Install the genderdata package? 

1: Yes
2: No

Selection: 1
Installing the genderdata package.
Installing package into ‘/Users/johnm1/Library/R/3.2/library’
(as ‘lib’ is unspecified)
Error in value[[3L]](cond) : Failed to install the genderdata package.
 Please try installing the package for yourself 
 using the following command: 
 install.packages("genderdata", repo = "http://packages.ropensci.org")
> install.packages("genderdata", repo = "http://packages.ropensci.org")
Installing package into ‘/Users/johnm1/Library/R/3.2/library’
(as ‘lib’ is unspecified)
Error in install.packages : Line starting '<html> ...' is malformed!

However the following was successful

devtools::install_github("lmullen/gender-data-pkg")

I now get the message that the package needs to be updated:

> z <- gender_df(hurricNamed, name_col="Name", year_col=c("Year"))
The genderdata package needs to be updated.
Install the genderdata package? 

1: Yes
2: No

Selection: 1
Installing the genderdata package.
Installing package into ‘/Users/johnm1/Library/R/3.2/library’
(as ‘lib’ is unspecified)
Error in value[[3L]](cond) : Failed to install the genderdata package.
 Please try installing the package for yourself 
 using the following command: 
 install.packages("genderdata", repo = "http://packages.ropensci.org")

> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.4 (Yosemite)

locale:
[1] en_NZ.UTF-8/en_NZ.UTF-8/en_NZ.UTF-8/C/en_NZ.UTF-8/en_NZ.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] gender_0.5.0    DAAG_1.22       lattice_0.20-33 knitr_1.10.5   

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.0         xml2_0.1.1          magrittr_1.5        splines_3.2.2      
 [5] devtools_1.8.0      MASS_7.3-43         R6_2.1.0            stringr_1.0.0      
 [9] httr_1.0.0          dplyr_0.4.2         tools_3.2.2         parallel_3.2.2     
[13] grid_3.2.2          nlme_3.1-121        genderdata_0.1      gamlss.data_4.3-0  
[17] latticeExtra_0.6-26 DBI_0.3.1           git2r_0.10.1        rversions_1.0.2    
[21] htmltools_0.2.6     survival_2.38-3     yaml_2.1.13         lazyeval_0.1.10    
[25] digest_0.6.8        assertthat_0.1      RColorBrewer_1.1-2  gamlss_4.3-5       
[29] curl_0.9.1          gamlss.dist_4.3-5   memoise_0.2.1       rmarkdown_0.7      
[33] stringi_0.5-5

PACKAGES file gets rewritten with only changed packages

We include the .json files created by drat.builder in GitHub to provide a record of the state of each package and determine if it needs to be rebuilt.

Unfortunately, when we do this, it seems like drat.builder creates a new PACKAGES list that overwrites the old one and contains only the packages that have changed and thus been rebuilt and re-added to drat. (Curiously, drat.builder seems to report a [build] message for all packages, but a [drat] for only those that changed. This results in being unable to install any package that wasn't updated, since it has no entry in PACKAGES.

Not sure if this is just a problem of having PACKAGES and PACKAGES.gz etc live on Amazon S3 instead of in the github repo, but this problem appears to persist even when we pull those copies from Amazon. So maybe it has something to do with the drat.builder logic wrt .json files which I don't understand. Leaving the .json files out of the Git repo forces circleci to rebuild all packages every time, which works but seems inefficient.

Hoping @richfitz will have some brilliant insight to get me unstuck. cc'ing @lmullen who has also been a huge help in trying to debug this.

Lincoln: meanwhile, I've just removed the .json files so that we at least have a brute force fix. hopefully that means things won't break every other day until Rich and I can get to the bottom of this.

Broken link "http://packages.ropensci.org/downloads.csv"

http://packages.ropensci.org/downloads.csv returns:

Page Not found
Looks like you've followed a broken link or entered a URL that doesn't exist on this site.

Needs to be corrected in README.md

Automatically compute download counts in CircleCI nightly runs

Tasks for the parse_s3_logs.R script:

Download log files from AWS (needs #2 fixed)
Parse raw log files (and append to previously parsed records in log.csv file)
Tidy the resulting data.frame into the format used by RStudio (anonymized IP, etc)
Write out the tidy, anonymized data.frame to downloads.csv
Push downloads.csv to GitHub or public S3 subdirectory for sharing / other applications
Push the parsed log.csv back to S3
Delete the already parsed & captured logs from S3

Then add a call to source("parse_s3_logs.R") to build.R.

Amazon S3 upload failing now?

Add a pkg?

What's the workflow for adding a pkg? i should probably know :(

drat::addRepo() does not persist across R sessions and does not work in .Rprofile

Starting from a new R session I can add a repo using drat::add_repo() as expected.

getOption("repos")
#>                        CRAN 
#> "https://cran.rstudio.com/" 
#> attr(,"RStudio")
#> [1] TRUE
drat::addRepo("OHDSI")
getOption("repos")
#>                            CRAN                           OHDSI 
#>     "https://cran.rstudio.com/" "https://OHDSI.github.io/drat/" 
#> attr(,"RStudio")
#> [1] TRUE

However when I restart R I would expect the repo to remain but it appears to be removed.

# After restarting R
getOption("repos")
#>                        CRAN 
#> "https://cran.rstudio.com/" 
#> attr(,"RStudio")
#> [1] TRUE

I added the following code to my .Rprofile in my user home directory

message("Running .Rprofile for user")
drat::addRepo("OHDSI")
message(paste("Repos are:", paste(getOption("repos"), collapse = ";   ")))

When I restart R I see

#> Restarting R session...

#> Running .Rprofile for user
#> Repos are: @CRAN@;   https://OHDSI.github.io/drat/

But when I check the repos option I don't see the new repo.

getOption("repos")
#>                        CRAN 
#> "https://cran.rstudio.com/" 
#> attr(,"RStudio")
#> [1] TRUE

I'm I correct that this behavior is unexpected? How should I add a repo that persists across R sessions? Why does running getOption("repos") in .Rprofile show the repo when running it interactively does not?

Thanks!

Fix automatic downloads of log files from AWS

Currently the AWS getbucket methods don't seem to respect marker header, making it impossible for us to download more than the max 1000 log files because we cannot loop over repeated calls (yes, S3 likes to create loads of little files for logs, with no subdirectory structure).

cloudyr/aws.s3#20

downloads csv link broken

http://packages.ropensci.org/downloads.csv which is linked in the README "page not found"

drat repository not building new versions?

USAboundariesData is at version 0.3.1 on the master branch and has been for a week or two. https://github.com/ropensci/USAboundariesData

Running install.packages("USAboundariesData", repos = c("http://packages.ropensci.org")) installs version 0.3.0.

I looked through the Circle CI logs but couldn't find any obvious errors. Is the drat repository building new versions of packages?

Allow use of https

packages.ropensci.org should probably support https:// as well as http:// connections.

Previously I believe this would be prohibitively expensive (Cloudfront had a service for ~$600 / Month).

However AWS recently provides SSH authentication at no charge for AWS apps, which I think could be used here.

https://aws.amazon.com/blogs/aws/new-aws-certificate-manager-deploy-ssltls-based-apps-on-aws/

Better describe common dependencies?

The README states "along with the development versions of common dependencies, which include many packages from Hadley Wickham."

I have three suggestions

replacing "Hadley Wickham" with "the tidyverse" or "RStudio".
add a link to the list of these common dependencies (if this is the right link).
renaming hadleyverse.txt common_dependencies.txt (they're not all tidyverse packages)

Build failing?

@sckott circle-ci build seems to be timing out on fetching ropensci/apipkgen --- is that a private package? (Seems to be asking for github author, I just get a 404 at that pkg). Any ideas?

Create a pretty landing page

The index.html is pretty retro at the moment: http://packages.ropensci.org/
(Just a vanilla pandoc conversion of the README from this repo).

Might be nice to have something more integrated with the ropensci web design, like http://ropensci.org/packages. Not sure if that page is computed programmatically using ropkgs (@sckott?) but something like that would be cool, and could be updated automatically very easily by the nightly builds we already do on CircleCI for the drat repo.

Maybe something more basic would be to at least include a table of the current packages/versions available on drat; perhaps along with download stats and some overall summary stats (once we have enough data); e.g. OS, region, etc.