Not caching downloads between deploys about heroku-buildpack-r HOT 10 CLOSED

Hi @jefflab

I am aware of this, and yes the bits you referred to from the compile script were removed intentionally since it wasn't doing anything and I wanted to "clean up" the script.

I haven't had time to implement the caching features yet. If you are able to give it a go it would be much appreciated.

Thanks for your feedback.

from heroku-buildpack-r.

Thanks for clarifying! My intention is to "give it a go" in the next week or so.

If there are any "lessons learned" that you'd like me to keep in mind, please send them my way.

from heroku-buildpack-r.

Sorry for the delayed response. I finally got around to working on this today.

The following things informed my solution:

1. The cran output during my deploy was from libraries being installed
2. The Buildpack API documentation states that CACHE_DIR is not intended for large amounts of data (it is meant to be used for resolved dependency files)
3. Binary dependencies should be compiled with heroku run because Vulcan is deprecated
4. It seems that best practice is to store compiled binaries on S3 and then copy them over during deploy
5. This article describes using a .vendor_urls file to tell the custom buildpack which binaries need to be installed

Based on the above information, I came up with the following proposed solution:

https://github.com/jefflab/heroku-buildpack-r/commit/cdc88316f097122a0157faf0e35f335de9d08420

To add library dependencies, the steps are:

1. Install the libraries manually using heroku run
2. Use tar -czvf to capture the library binaries with the appropriate path
3. Upload the tar.gz to S3
4. Add the S3 url to a file called .vendor_urls in their project

If you think a solution like this should be added to the project let me know. I'll write up better documentation and create a pull request

from heroku-buildpack-r.

@jefflab @virtualstaticvoid What ever happened to this solution?

I agree that having some sort of caching would be really helpful to speed up Heroku deployments.

from heroku-buildpack-r.

@andrewmatthewthompson I never got around to build the generic solution. Instead I just ended up directly copying the binaries I needed in a separate build pack.

from heroku-buildpack-r.

Okay thanks for the update @jefflab.

On a complete aside do you know what the build_with_heroku command is on this page (i.e. how does that command end up installed somewhere so you can use it to build a customer version of R): https://github.com/virtualstaticvoid/heroku-buildpack-r/blob/master/support/README.md

from heroku-buildpack-r.

@andrewmatthewthompson My understanding is that build_with_heroku just refers to this script:

https://github.com/virtualstaticvoid/heroku-buildpack-r/blob/master/support/build_with_heroku

You don't have to install it or anything. I never tried to run this script directly, but I used it as an example for how to compile binaries and get them on S3. Took a bit of trial and error for me to get it working. Good luck.

from heroku-buildpack-r.

The build_with_heroku script and others are used for compiling the R binaries which are then packaged in the buildpack archive. When you push your application code to Heroku, this buildpack archive is fetched from S3 and extracted into your slug. It is possible to cache the extracted archive, so that deploys are faster, but this hasn't been implemented yet.

Also, you can include additional R CRAN packages to be installed in the init.r file, and it would be beneficial to cache these too.

For cedar-14, I've extracted these scripts into their own repository.

from heroku-buildpack-r.

Another aspect of the caching is for when additional packages are installed, as part of the deploy, that there outputs get cached. I.e. The slow part is compiling these packages to include in the slug.

Perhaps the title of this issue should rather be "Not caching additional installed packages between deploys".

from heroku-buildpack-r.

Fixed by #68 and #70

from heroku-buildpack-r.