jeroen / curl Goto Github PK

View Code? Open in Web Editor NEW

212.0 212.0 69.0 8.24 MB

A Modern and Flexible Web Client for R

Home Page: https://jeroen.r-universe.dev/curl

License: Other

R 41.22% C 58.76% Shell 0.02%

curl's Introduction

curl

A Modern and Flexible Web Client for R

The curl() and curl_download() functions provide highly configurable drop-in replacements for base url() and download.file() with better performance, support for encryption (https, ftps), gzip compression, authentication, and other libcurl goodies. The core of the package implements a framework for performing fully customized requests where data can be processed either in memory, on disk, or streaming via the callback or connection interfaces. Some knowledge of libcurl is recommended; for a more-user-friendly web client see the 'httr' package which builds on this package with http specific tools and logic.

Documentation

About the R package:

Vignette: curl: A Modern and Flexible Web Client for R

Other resources:

libcurl handle options overview (use with handle_setopt in R)

Hello World

There are three download interfaces (memory, disk and streaming). Always start by setting up a request handle:

library(curl)
h <- new_handle(copypostfields = "moo=moomooo")
handle_setheaders(h,
  "Content-Type" = "text/moo",
  "Cache-Control" = "no-cache",
  "User-Agent" = "A cow"
)

Perform request and download response in memory:

# Perform the request
req <- curl_fetch_memory("http://httpbin.org/post", handle = h)

# Show some outputs
parse_headers(req$headers)
cat(rawToChar(req$content))
str(req)

Or alternatively, write response to disk:

tmp <- tempfile()
curl_download("https://httpbin.org/post", tmp, handle = h)
readLines(tmp)

Or stream response via Connection interface:

con <- curl("https://httpbin.org/post", handle = h)
open(con)

# Get 3 lines
readLines(con, n = 3)

# Get remaining lines and close connection
readLines(con)
close(con)

Installation

Binary packages for OS-X or Windows can be installed directly from CRAN:

install.packages("curl")

Installation from source on Linux requires libcurl. On Debian or Ubuntu use libcurl4-openssl-dev:

sudo apt-get install -y libcurl-dev

On Fedora, CentOS or RHEL use libcurl-devel:

sudo yum install libcurl-devel

MacOS using curl from homebrew

On MacOS libcurl is included with the system, so usually nothing extra is needed. However if you want to build against the very most recent version of libcurl, which also has many extra features enabled (sftp, http2), install curl from homebrew and then recompile the R package.

brew install curl pkg-config

You need to set the PKG_CONFIG_PATH environment variable to help R find the non-default curl, when building from source. Run this in a clean R session which does not have the curl package loaded already:

Sys.setenv(PKG_CONFIG_PATH="/usr/local/opt/curl/lib/pkgconfig")
install.packages("curl", type = "source")

Afterwards confirm the version using curl::curl_version().

Development version

Because devtools and httr depend on curl, installing with install_github does not work well. The easiest way to install the development version of curl is a clean R session:

install.packages("https://github.com/jeroen/curl/archive/master.tar.gz", repos = NULL)

Of course windows users need Rtools to compile from source.

curl's People

Contributors

Stargazers

Watchers

curl's Issues

Setting httpheader to named list crashes curl

h <- new_handle()
handle_setopt(h, 
  httpheader = c(Accept = "application/json, text/xml, application/xml, */*")
)

x <- curl_perform("http://google.com", h)

I'm not sure if you want to make this work or not, but it should at least throw an error.

Should we use curl_easy_perform?

It either succeeds or fails - if there's a long time out for some reason (e.g. curl::curl_fetch_memory("http://httpbin.org/delay/10")) , there's no way to interrupt the transfer. Maybe we should instead use curl_multi_perform, regularly checking for interrupts?

This problem only occurs before the server responds with headers or body, since we're already checking for interrupts in those places.

Review use of error()

Check that when the error function is called, stuff is still cleaned up correctly.

Possible to implement asynchronous (i.e. non-blocking io) in curl?

From r-lib/httr#271:

This is not an issue, more of a question that I wanted to pose to the community. If there is already a method for achieving this, I apologize for the repeat and would appreciate being pointed in the right direction.

I use httr extensively and often find myself in situations where the needed aggregation of data requires hundreds of thousands of REST calls. This becomes performance limiting in R because it seems like every http call made using httr is a blocking call.

Is there any plan or path forward for enabling asynchronous io within curl similar to what exists in Python via aiohttp or Scala via Akka-http?

Support for SIGINT

Implement R_CheckUserInterrupt();.

Fails to install on RHEL 5.6

> install.packages("curl")
Installing package into ‘/usr/lib64/R/library’
(as ‘lib’ is unspecified)
trying URL 'http://cran.rstudio.com/src/contrib/curl_0.8.tar.gz'
Content type 'application/x-gzip' length 241067 bytes (235 KB)
==================================================
downloaded 235 KB

* installing *source* package ‘curl’ ...
** package ‘curl’ successfully unpacked and MD5 sums checked
Found libcurl in /usr/include/curl/curl.h
** libs
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c callbacks.c -o callbacks.o
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c curl.c -o curl.o
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c download.c -o download.o
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c escape.c -o escape.o
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c fetch.c -o fetch.o
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c form.c -o form.o
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c getdate.c -o getdate.o
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c handle.c -o handle.o
handle.c: In function ‘set_handle_defaults’:
handle.c:75: error: ‘CURLOPT_CONNECTTIMEOUT_MS’ undeclared (first use in this function)
handle.c:75: error: (Each undeclared identifier is reported only once
handle.c:75: error: for each function it appears in.)
make: *** [handle.o] Error 1
ERROR: compilation failed for package ‘curl’
* removing ‘/usr/lib64/R/library/curl’

The downloaded source packages are in
        ‘/tmp/Rtmp441QyQ/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
Warning message:
In install.packages("curl") :
  installation of package ‘curl’ had non-zero exit status
>

$ yum info curl-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.usc.edu
 * epel: fedora-epel.mirror.lstn.net
 * extras: repos.dfw.quadranet.com
 * updates: mirrors.adams.net
Installed Packages
Name       : curl-devel
Arch       : i386
Version    : 7.15.5
Release    : 17.el5_9
Size       : 643 k
Repo       : installed
Summary    : Files needed for building applications with libcurl.
URL        : http://curl.haxx.se/
License    : MIT
Description: cURL is a tool for getting files from FTP, HTTP, Gopher, Telnet, and
           : Dict servers, using any of the supported protocols. The curl-devel
           : package includes files needed for developing applications which can
           : use cURL's capabilities internally.

Name       : curl-devel
Arch       : x86_64
Version    : 7.15.5
Release    : 17.el5_9
Size       : 751 k
Repo       : installed
Summary    : Files needed for building applications with libcurl.
URL        : http://curl.haxx.se/
License    : MIT
Description: cURL is a tool for getting files from FTP, HTTP, Gopher, Telnet, and
           : Dict servers, using any of the supported protocols. The curl-devel
           : package includes files needed for developing applications which can
           : use cURL's capabilities internally.

Let me know if I can provide any more info!

Curl hangs on HEAD request

Not sure what's going wrong here:

library(curl)
h <- new_handle()
handle_setopt(h, customrequest = "HEAD")
curl_fetch_memory("http://google.com", handle = h)

If I cmd + c in a debugger, the backtrace is

(lldb) bt 10
* thread #1: tid = 0x1457395, 0x00007fff838785c2 libsystem_kernel.dylib`poll + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff838785c2 libsystem_kernel.dylib`poll + 10
    frame #1: 0x00007fff8b538378 libcurl.4.dylib`Curl_poll + 140
    frame #2: 0x00007fff8b533791 libcurl.4.dylib`curl_multi_wait + 525
    frame #3: 0x00007fff8b52ccba libcurl.4.dylib`curl_easy_perform + 245
    frame #4: 0x0000000102fe9a79 curl.so`R_curl_fetch_memory + 169
    frame #5: 0x000000010007b2c7 libR.dylib`do_dotcall(call=0x0000000106d20400, op=<unavailable>, args=<unavailable>, env=<unavailable>) + 327 at dotcode.c:1251
    frame #6: 0x00000001000a546c libR.dylib`Rf_eval(e=0x0000000106d20400, rho=0x00000001038cd4b0) + 988 at eval.c:655

It doesn't seem to time out (I waited well over 10s)

New features(?)

For the second time in a row, I've started working on a problem and Hadley or someone else has informed me you're already doing it ;p. I figure I should get involved in this (even if it means learning the C API. And, you know: if you'll have me).

Things I'd like to add, as a starter: bindings to easy_unescape and easy_escape, ideally ones that split the scheme out beforehand to avoid the :// getting encoded.

And, what are your thoughts on adding functionality that isn't found in the underlying curl library? URL composition and decomposition (as httr does it, but in C and vectorised), value extraction from parameters, that sort of thing.

Proxy settings in Windows

Read proxy settings in windows to mimic setInternet2:

http://stackoverflow.com/questions/202547/how-do-i-find-out-the-browsers-proxy-settings

Warnings on httr::progress()

Seems that httr::progress() now throws warnings when used on new httr 1.0 version - e.g., a discussion here traitecoevo/taxonlookup#11 (comment)

This is the warning message after using progress()

Warning messages:
1: In curl::curl_fetch_disk(url, x$path, handle = handle) :
  progress callback must return boolean
...

add easy_escape

While I appreciate the urge to keep this library very lightweight, curl/easy.h includes easy_escape(), which does what it says on the tin - really simple URL encoding. If the idea is for this to act as a replacement for RCurl in the context of httr and similar libraries, you might want to look at making this available (or including it automatically in the curl() function): escaping will be needed, and R's URLencode has many limitations that make it a pain.

Quick fix for homebrew (OS X) users

I thought I would just pass you a note for homebrew/OSX users. In homebrew the curl headers are not put in any default search path to avoid conflicts with any system curl (unfortunately OS X does not ship with any headers for curl 😞 ).

So you might want to consider just adding an else if to search the homebrew location,

elif [ -r /usr/local/opt/curl/include/curl/curl.h ]; then CURL_INCLUDES="/usr/local/opt/curl/include"

Anyway, thanks for the package. If you want I could always make a pull request, but I probably won't get to that until Sunday night (EST).

Equivalent to RCurl::base64

What would you recommend?

SSL validation

Why do you turn it off by default? It seems like a really really bad idea.

update to 0.93 failed

Hi, R 3.2.2 on Ubuntu 15.04 64 bit. Currently curl 0.92 is installed.

* installing *source* package ‘curl’ ...
** package ‘curl’ successfully unpacked and MD5 sums checked
Found pkg-config cflags and libs!
Using PKG_CFLAGS=
Using PKG_LIBS=-lcurl 
** libs
"/usr/lib/R/bin/Rscript" "../tools/options.R"
Error in file(filename, "r", encoding = encoding) : 
  cannot open the connection
Calls: source -> file
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
  cannot open file '../tools/option_table.txt': No such file or directory
Execution halted
Makevars:7: recipe for target 'options' failed
make: *** [options] Error 1
ERROR: compilation failed for package ‘curl’
* removing ‘/usr/lib/R/library/curl’
* restoring previous ‘/usr/lib/R/library/curl’

The downloaded source packages are in
    ‘/tmp/RtmpOpqUvf/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
Warning message:
In install.packages(update[instlib == l, "Package"], l, contriburl = contriburl,  :
  installation of package ‘curl’ had non-zero exit status

Fetch fails when debugfunction set

and it breaks the handle

library(curl)
h <- new_handle(verbose = TRUE, debugfunction = function(type, msg) NULL)
curl_fetch_memory("http://had.co.nz", handle = h)
curl_fetch_memory("http://had.co.nz", handle = h)

Weird memory bug

This is fine:

repeat print(length(readLines(curl("https://code.jquery.com/jquery-2.1.3.js"))))

But this sometimes gives a memory error:

repeat print(length(readLines(curl("https://raw.githubusercontent.com/isaacs/sax-js/master/lib/sax.js"))))

After a while:

R(2199,0x7fff73c79300) malloc: *** error for object 0x7f8fbfb097c0: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

The problem seems to happen in curl_multi_perform, but before the callback function is called. It seems like this was introduced in curl 0.3, I'm not seeing it in curl 0.2.

Error installing on RHEL

I'm having trouble installing the library on a RHEL 6.4 machine. I have libcurl-devel installed. See the error message below.

> devtools::install_github("jeroenooms/curl")
Downloading github repo jeroenooms/curl@master
Installing curl
'/usr/lib64/R/bin/R' --vanilla CMD INSTALL  \
  '/tmp/RtmpAsJe5G/devtools23fc5c74c008/jeroenooms-curl-b92b20b'  \
  --library='/opt/Rapp/jrowen/R/x86_64-redhat-linux-gnu-library/3.1'  \
  --install-tests 

* installing *source* package ‘curl’ ...
** libs
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG  -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c curl.c -o curl.o
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG  -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c download.c -o download.o
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG  -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c escape.c -o escape.o
gcc -m64 -std=gnu99 -I/usr/include/R -DNDEBUG  -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -c options.c -o options.o
In file included from options.h:15,
                 from options.c:3:
option_table.h:168: error: ‘CURLOPT_GSSAPI_DELEGATION’ undeclared here (not in a function)
make: *** [options.o] Error 1
ERROR: compilation failed for package ‘curl’
* removing ‘/opt/Rapp/jrowen/R/x86_64-redhat-linux-gnu-library/3.1/curl’
Error: Command failed (1)

$ rpm -qa | grep libcurl
libcurl-7.19.7-40.el6_6.3.x86_64
libcurl-devel-7.19.7-40.el6_6.3.x86_64

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.4 (Santiago)

Equivalent to RCurl::getCurlInfo

In the event of an HTTP error (particularly those in the 4XX series), it's useful to get error messages from the server to know what went wrong. In RCurl this is accomplished by RCurl::getCurlInfo(handle) after the error has occurred. Is there an equivalent in curl?

Error while loading curl

I have the latest version of curl(0.9.4) and each time I am trying to load the library. It throws the following:

Error in .loadNamespaceImpl(package, path, keep.source, partial): error executing useDynLib for dynamic library 'curl' from package 'curl' loaded from C:/abc/xyz/library:
Error in library.dynam(chname = chname, package = package, lib.loc = ...: Foreign binary curl could not be loaded

Any idea on what could be causing it?

Need curl_stream() or similar

Which returns same information as curl_perform(), plus a connection to listen to

curl over TOR

Any way to add easy switch for using curl over the TOR network?
in shell you can run curl over tor easily:

sudo apt-get install tor
curl icanhazip.com
usewithtor curl icanhazip # some warnings on the way but the different ip achieved

or any particular place in the code where I would need to look at?

curl_download() should return same info as curl_perform()

Since you want to be able to check status etc.

SSL connection error

On OSX:

> curl_perform("https://media.mongodb.org/zips.json", handle = new_handle(verbose=T))
* Hostname was NOT found in DNS cache
*   Trying 54.230.69.49...
* Connected to media.mongodb.org (54.230.69.49) port 443 (#0)
* SSL peer handshake failed, the server most likely requires a client certificate to connect
* Closing connection 0
Error in curl_perform("https://media.mongodb.org/zips.json", handle = new_handle(verbose = T)) : 
  SSL connect error

best practice re: closing the connection?

I am using curl() following usage I see in the official examples, like so:

x <- readLines(curl(SOME_URL)

but inside a loop over SOME_URL. I then got lots of warnings about closing unused connections. Am I supposed to be more careful and always store then close the connection? Does it even matter?

con <- curl(SOME_URL)
x <- readLines(con)
close(con)

handle_setform() docs

It would be nice to have a bit more documentation here (including example of uploading a file to a form)

How best to expose symbols

Curl has hundreds of C symbols that identify option names or possible values for options names. The tricky part is that the available symbols depend on the version of libcurl. The libcurl author has included a plain-text table that lists which symbols were introduced or deprecated in which version. This script converts this table into a dataframe and looks up the actual integer value behind each symbol so we can set it at runtime.

Currently the table is not symbol exported: curl:::curl_symbols. I am not sure if this should be exported, or if we should only test internally in the curl if a symbol is available.

Installing on shared cluster

Our shared cluster uses environment modules to install most programs and shared libraries to a non-standard location, and add them to the path as needed. I can tell the package configuration where curl.h is by setting the CURL_INCLUDES environment variable (in my case =/sw/curl/include), but the install eventually fails:

gcc -std=gnu99 -shared -L/sw/R-3.2.1/lib -L/usr/local/lib64 -o curl.so callbacks.o curl.o download.o escape.o fetch.o form.o getdate.o handle.o ieproxy.o init.o options.o utils.o version.o -lcurl -L/sw/R-3.2.1/lib -lR
/sw/binutils-2.25/bin/ld: cannot find -lcurl
collect2: ld returned 1 exit status
make: *** [curl.so] Error 1
ERROR: compilation failed for package ‘curl’

Problem with garbage collector

Looks like the garbage collector is overwriting memory. Bit hard to reproduce, but in rstudio this usually works:

library(curl)
con <- curl("http://localhost/test.txt")
gc(F)
debug(readLines)
readLines(con)

'curl' Installation failed.

HI, I was trying to install R package 'culr' on a RHEL machine. I got the following errors.

"* installing source package âcurlâ ...
** package âcurlâ successfully unpacked and MD5 sums checked
Found pkg-config cflags and libs!
Using PKG_CFLAGS=-I/nz/export/ae/appsysroot/host64/include
Using PKG_LIBS=-L/nz/export/ae/appsysroot/host64/lib64 -lcurl
** libs
"/nz/export/ae/languages/r/3.1/host64/lib64/R/bin/Rscript" "../tools/options.R"
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c callbacks.c -o callbacks.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c curl.c -o curl.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c download.c -o download.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c escape.c -o escape.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c fetch.c -o fetch.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c form.c -o form.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c getdate.c -o getdate.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c handle.c -o handle.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c ieproxy.c -o ieproxy.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c init.c -o init.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c utils.c -o utils.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -I/nz/export/ae/languages/r/3.1/host64/lib64/R/include -DNDEBUG -I/nz/export/ae/appsysroot/host64/include -m64 -I/nz/export/ae/sysroot/host64/include -fpic -m64 -O1 -c version.c -o version.o
/nz/export/ae/sysroot/host64/bin/gcc -std=gnu99 -shared -m64 -Bdirect,--hash-style=both,-Wl,-O1 -L/nz/export/ae/core/thirdParty/intel_mkl/current/lib/em64t -L/nz/export/ae/sysroot/host64/lib64 -L/nz/export/ae/sysroot/host64/usr/lib64 -L/nz/export/ae/sysroot/host64/x86_64-rhel5-linux-gnu/lib64 -Wl,-rpath=/nz/export/ae/core/thirdParty/intel_mkl/current/lib/em64t:/nz/export/ae/sysroot/host64/lib64:/nz/export/ae/sysroot/host64/usr/lib64:/nz/export/ae/sysroot/host64/x86_64-rhel5-linux-gnu/lib64 -o curl.so callbacks.o curl.o download.o escape.o fetch.o form.o getdate.o handle.o ieproxy.o init.o utils.o version.o -L/nz/export/ae/appsysroot/host64/lib64 -lcurl -L/nz/export/ae/languages/r/3.1/host64/lib64/R/lib -lR
installing to /nz/export/ae/languages/r/3.1/host64/lib64/R/library/curl/libs
** R
** data
*** moving datasets to lazyload DB
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object '/nz/export/ae/languages/r/3.1/host64/lib64/R/library/curl/libs/curl.so':
libcurl.so.4: cannot open shared object file: No such file or directory
Error: loading failed
Execution halted
ERROR: loading failed

removing â/nz/export/ae/languages/r/3.1/host64/lib64/R/library/curlâ
Warning message:
In install.packages("/tmp/renjith/curl_0.9.3.tar.gz") :
installation of package â/tmp/renjith/curl_0.9.3.tar.gzâ had non-zero exit status
"

After this, I manually compiled the curl.so and ran ldd on it. The following is the result:
[nz@umabofanzd05h1 src]$ ldd curl.so
linux-vdso.so.1 => (0x00007fff1730c000)
libcurl.so.4 => not found
libR.so => not found
libc.so.6 => /nz/export/ae/sysroot/host64/lib64/libc.so.6 (0x00002b7b6769d000)
/lib64/ld-linux-x86-64.so.2 (0x0000003d91400000)
[nz@umabofanzd05h1 src]$ locate libcurl.so.4
/nz/export/ae/appsysroot/curl-7.44.0/lib/.libs/libcurl.so.4
/nz/export/ae/appsysroot/curl-7.44.0/lib/.libs/libcurl.so.4.4.0
/nz/export/ae/appsysroot/host64/lib64/libcurl.so.4
/nz/export/ae/appsysroot/host64/lib64/libcurl.so.4.4.0
/nz/export/ae/appsysroot/spu64/lib64/libcurl.so.4
/nz/export/ae/appsysroot/spu64/lib64/libcurl.so.4.4.0
[nz@umabofanzd05h1 src]$ locate libR.so
/nz/export/ae/languages/r/3.1/host64/lib64/R/lib/libR.so
/nz/export/ae/languages/r/3.1/spu64/lib64/R/lib/libR.so

Any idea why curl.so can't be loaded? Thanks for any help!

Segmentation fault when calling handle_ functions with improper values

The following functions result in segmentation faults:

curl::handle_setform(list(dummy=1))
curl::handle_setopt(list(dummy=1))
curl::handle_cookies(list(dummy=1))

I discovered this because I was getting a new error in curl::handle_setopt: "Unknown option: http.version", and in the process I forgot to feed handle_setopt the handle. I do not expect the code to work, but do expect it to error more gracefully.

/bin/bash invalid on OpenBSD

On OpenBSD bash is installed (it is not available in the base system) at /usr/local/bin/, so https://github.com/jeroenooms/curl/blob/master/configure#L1 fails.

If you do not want to use /bin/sh, another solution would be "/usr/bin/env bash".

However, using bash at all is considered bad style: https://cran.r-project.org/doc/manuals/r-devel/R-exts.html#Writing-portable-packages states "Bash extensions also need to be avoided in shell scripts". So /bin/sh would be the best solution.

curl_fetch_stream: return value of 'fun' not evaluated - desired behaviour?

This relates to r-lib/httr#352.

If I understand correctly httr::write_stream is mostly offhanded to curl::curl_fetch_stream which does not take into account the value that is returned by the callback function in such way as defined in the httr docs.

As a reference the docs from ?httr::write_stream:

write_stream(f)

Arguments

f: Callback function. It should have a single argument, a raw vector containing the bytes recieved from the server. This will usually be 16k or less. It should return the length of bytes processed - if this is less than the input length, the function will terminate.

If the callback function's return value would actually be checked to be the length of bytes processed, which it doesn't ATM, this would allow for terminating the stream when required.

This quick hack:

curl_fetch_stream <- function(url, fun, handle = new_handle()){
  con <- .Call(R_curl_connection, url, "rb", handle, FALSE)
  on.exit(close(con))
  while(length(bin <- readBin(con, raw(), 8192L))){
    fun_eval<-fun(bin)
    if(!is.numeric(fun_eval) || fun_eval<length(bin)) break
    fun_eval
  }
  handle_response_data(handle)
}

Would allow e.g. for:

foo<-1
bar<-5

GET("https://jeroenooms.github.io/data/diamonds.json",
    write_stream(function(x) {
      print(length(x))
      foo<<-foo+1
      if(foo > bar) return(0)

      length(x)
    })
)

Or maybe I'm taking this too far and there is a much easier way to terminate the handler when wanted? 😁

handle_setopt() that takes list

Would be useful to have variant of handle_setopt() that lets you pass in a list of options

passing multiple form fields

Dear Jeroen,

I am using curl for https://github.com/soodoku/clarifai

I am trying to pass multiple urls to the API, like so:
curl -H "Authorization: Bearer <access_token>" --data-urlencode "url=http://www.clarifai.com/img/metro-north.jpg" --data-urlencode "url=http://www.clarifai.com/img/metro-north.jpg" https://api.clarifai.com/v1/tag/

I can do that with:
handle_setform(h, url = "url1", url="url2")

But the number of urls being uploaded depends on user input. I am having trouble with building that.
I know I can't do: c("url = string1, url = string2") etc. There may be a do.call version of this but haven't found a way forward.

Any suggestions? Thanks.

Install fails on CentOS 5.5

Hello,

I'm interested in using your rjade package, but had an issue installing the curl package on which it depends. Here's the output from my install attempt:

> install.packages("rjade")
also installing the dependencies 'curl', 'V8'

trying URL 'http://cran.mtu.edu/src/contrib/curl_0.9.1.tar.gz'
Content type 'application/x-gzip' length 242408 bytes (236 KB)
opened URL
==================================================
downloaded 236 KB

trying URL 'http://cran.mtu.edu/src/contrib/V8_0.6.tar.gz'
Content type 'application/x-gzip' length 110013 bytes (107 KB)
opened URL
==================================================
downloaded 107 KB

trying URL 'http://cran.mtu.edu/src/contrib/rjade_0.1.tar.gz'
Content type 'application/x-gzip' length 44597 bytes (43 KB)
opened URL
==================================================
downloaded 43 KB

* installing *source* package 'curl' ...
** package 'curl' successfully unpacked and MD5 sums checked
Found libcurl in /usr/include/curl/curl.h
** libs
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c callbacks.c -o callbacks.o
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c curl.c -o curl.o
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c download.c -o download.o
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c escape.c -o escape.o
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c fetch.c -o fetch.o
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c form.c -o form.o
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c getdate.c -o getdate.o
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c handle.c -o handle.o
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c ieproxy.c -o ieproxy.o
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c init.c -o init.o
/broad/software/free/Linux/redhat_5_x86_64/pkgs/gcc_4.8.2/bin/gcc -std=gnu99 -I/xchip/cogs/tools/opt/R-3.1.3/lib64/R/include -DNDEBUG -I/usr/include -I/usr/local/include    -fpic  -g -O2  -c options.c -o options.o
In file included from options.c:3:0:
options.h:7:27: error: 'CURLOPT_SOURCE_USERPWD' undeclared here (not in a function)
 #define CINIT(a,b,c) {#a, CURLOPT_##a}
                           ^
option_table.h:114:3: note: in expansion of macro 'CINIT'
   CINIT(SOURCE_USERPWD, OBJECTPOINT, 123),
   ^
options.h:7:27: error: 'CURLOPT_SOURCE_PREQUOTE' undeclared here (not in a function)
 #define CINIT(a,b,c) {#a, CURLOPT_##a}
                           ^
option_table.h:115:3: note: in expansion of macro 'CINIT'
   CINIT(SOURCE_PREQUOTE, OBJECTPOINT, 127),
   ^
options.h:7:27: error: 'CURLOPT_SOURCE_POSTQUOTE' undeclared here (not in a function)
 #define CINIT(a,b,c) {#a, CURLOPT_##a}
                           ^
option_table.h:116:3: note: in expansion of macro 'CINIT'
   CINIT(SOURCE_POSTQUOTE, OBJECTPOINT, 128),
   ^
options.h:7:27: error: 'CURLOPT_SOURCE_URL' undeclared here (not in a function)
 #define CINIT(a,b,c) {#a, CURLOPT_##a}
                           ^
option_table.h:120:3: note: in expansion of macro 'CINIT'
   CINIT(SOURCE_URL, OBJECTPOINT, 132),
   ^
options.h:7:27: error: 'CURLOPT_SOURCE_QUOTE' undeclared here (not in a function)
 #define CINIT(a,b,c) {#a, CURLOPT_##a}
                           ^
option_table.h:121:3: note: in expansion of macro 'CINIT'
   CINIT(SOURCE_QUOTE, OBJECTPOINT, 133),
   ^
make: *** [options.o] Error 1
ERROR: compilation failed for package 'curl'
* removing '/xchip/cogs/tools/opt/R-3.1.3/lib64/R/library/curl'
ERROR: dependency 'curl' is not available for package 'V8'
* removing '/xchip/cogs/tools/opt/R-3.1.3/lib64/R/library/V8'
ERROR: dependency 'V8' is not available for package 'rjade'
* removing '/xchip/cogs/tools/opt/R-3.1.3/lib64/R/library/rjade'

The downloaded source packages are in
    '/tmp/RtmpmVgwR6/downloaded_packages'
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
Warning messages:
1: In install.packages("rjade") :
  installation of package 'curl' had non-zero exit status
2: In install.packages("rjade") :
  installation of package 'V8' had non-zero exit status
3: In install.packages("rjade") :
  installation of package 'rjade' had non-zero exit status

Also, some info about the version of curl on the system:

$ curl --version
curl 7.23.0 (x86_64-unknown-linux-gnu) libcurl/7.23.0 OpenSSL/0.9.8b zlib/1.2.7 libidn/0.6.5 libssh2/1.3.0

Please let me know if there's any other info I can provide. Thanks a lot!

Equivalent to RCurl::getCurlOptionTypes?

e.g.

curl_option_types <- function(opts = curl::curl_options()) {
  type_name <- c("integer", "string", "function", "number")
  type <- floor(opts / 10000)

  type_name[type + 1]
}

curl package new version creating “msvcr80.dll” missing warning message

I am currently using R version 3.2.1, and when I updated the "curl" package to version 0.9.2, I started getting following message after I run script that uses curl package (there is no such message with scripts that don't use curl):

"100% downloaded: rsession.exe-System Error"
The program can't start because MSVC80.dll is missing from your computer.
Try installing the program to fix this problem.

There are OK and cross (as in multiplication sign) options. If I chose one of them, program continues without any error. It's, however, annoying that I get this message whenever I run the script that uses functions from curl package.

The description file of curl package has following:

Version: 0.9.2
Depends: R (≥ 2.10.0)

But, on the closer look, I see this: "package ‘curl’ was built under R version 3.2.2". So, is this saying that I have to install new version of R (3.2.2) to run without any warning message? What if I don't want to update R version (now), but I do want to use new version of "curl" package (0.9.2)?

Note, that I have tried both with baseR and RStudio. Both gives the warning message when running scripts that use "curl" package. I would appreciate if you could take a closer look into this issue. I am using Windows 8.

Configure script does not confirm curl was compiled with libz support

When pointed at a libcurl compiled without libz support enabled, the package will install succesfully, but segfault when collecting version info here https://github.com/jeroenooms/curl/blob/master/src/version.c#L12 because the libz_version field of the struct is NULL.

If libz support is mandatory then the package configuration script should fail if the libcurl it finds doesn't have it.

Does not build on snow leopard

The curl package for snow-leopard has not been updated since 0.9.1. Not sure why, it passes check on my snow-leopard server. @s-u

List curl options using preprocessor

This runs the preprocessor -E over curl.h to resolve symbols. We extract only option symbols:

CFLAGS=$(pkg-config --cflags libcurl)
echo '#include <curl/curl.h>' | cc $CFLAGS -std=gnu99 -E -xc - | grep "^[ \t]*CURLOPT_*"

Thereby we do not have to search for curl.h but instead use cflags from pkg-config.

Inconsistent close(con)

See this file where I just switched to curl::curl() because the site switched from http to https and url() is no longer of use.

I can close(con) in the first routine following readLines(); in all the other ones following read.table() I get an error. That seems odd. Is this a bug?

Support function callbacks

I use them in three places:

To upload a file as request body (readfunction)
To display a nicer progress bar (progressfunction)
To display a nicer verbose output (debugfunction)

Issue installing on OS X El Capitan

I've got R and curl installed via MacPorts on OS X, and when I try to install curl (even after checking out the latest from github), it keeps failing at the very end of the install.

$ R CMD INSTALL .
WARNING: ignoring environment value of R_HOME
* installing to library ‘/Users/steevmi1/Library/R/3.2/library’
* installing *source* package ‘curl’ ...
Found pkg-config cflags and libs!
Using PKG_CFLAGS=-I/opt/local/include
Using PKG_LIBS=-L/opt/local/lib -lcurl
** libs
make: Nothing to be done for `all'.
installing to /Users/steevmi1/Library/R/3.2/library/curl/libs
** R
** data
*** moving datasets to lazyload DB
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error in dyn.load(file, DLLpath = DLLpath, ...) :
  unable to load shared object '/Users/steevmi1/Library/R/3.2/library/curl/libs/curl.so':
  dlopen(/Users/steevmi1/Library/R/3.2/library/curl/libs/curl.so, 6): Symbol not found: _SSLv2_client_method
  Referenced from: /opt/local/lib/libcurl.4.dylib
  Expected in: /opt/local/lib/libssl.1.0.0.dylib
 in /opt/local/lib/libcurl.4.dylib
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/Users/steevmi1/Library/R/3.2/library/curl’
* restoring previous ‘/Users/steevmi1/Library/R/3.2/library/curl’

Curl not working with IPv6 on Windows

I have a web service that has some clients with IPv6-only connectivity that need to use httr (and hence curl). On Mac we have had this work well, but we're were surprised to find on Windows it fails.

The error message is "unable to resolve host". While we ultimately expect DNS with AAAA record to be used, testing with IP is a good step:

con= curl(url="http://[2001:468:c80:a202:0:b074:0:c082]/~cgaylord/x.txt")
readLines(con)

gives "cannot resolve host [2001"

(I will leave that file there for a while so others can test.)

Results otherwise are variable, but we have had success with Mac at least.

My thought is: is this a deficiency of a statically linked libcurl that does not have proper IPv6 functionality? I'm happy to drop another library in, if there's a suggestion of how best to do this.

Thanks!

anyNA requires R >= 3.1.0

@jeroenooms you added the anyNA() function in commit ff76a61 but you don't require R >= 3.1.0 in the DESCRIPTION file. I get an error that anyNA() is not found when using R < 3.1.0. Either you do the sensible thing and just use any(is.na(keys)) or you bump the minimum required version of R to 3.1.0.

Working with very strange redirects

I have a very strange use case that I would like to submit for consideration because I get different results with curl_download than I do with curl from the shell. Please feel free to close the issue straight away if the problem is not related enough to curl.

I'm starting with redirect URLs like this one:

http://www.senato.it/loc/link.asp?tipodoc=CAM.DEP&leg=17&id=30350

By using httr, I get the URL it redirects to:

http://www.camera.it/leg17/29?tipoAttivita=&tipoVisAtt=&tipoPersona=&shadow_deputato=30350&idLegislatura=17

Now, if I navigate to this page, I sometimes get a Web page for a male Italian MP, but I also sometimes get an empty page with content type application/json that simply says "format json not implemented yet". Here's the output with httr:

> GET("http://www.camera.it/leg17/29?tipoAttivita=&tipoVisAtt=&tipoPersona=&shadow_deputato=30350&idLegislatura=17")
Response [http://www.camera.it/leg17/29?tipoAttivita=&tipoVisAtt=&tipoPersona=&shadow_deputato=30350&idLegislatura=17]
  Date: 2015-08-04 12:22
  Status: 200
  Content-Type: application/json; charset=utf-8
  Size: 31 B
> GET("http://www.camera.it/leg17/29?tipoAttivita=&tipoVisAtt=&tipoPersona=&shadow_deputato=30350&idLegislatura=17") %>% content
Error in parse_string(txt, bigint_as_char) : 
  lexical error: invalid string in json text.
                                       format json not implemented yet
                     (right here) ------^

So I turn to curl, thinking that it might save me from that very strange issue. Why? Because every time that I try curl -v on that last URL in the shell, I get HTML instead of the "JSON" thing:

curl -v http://www.camera.it/leg17/29?tipoAttivita=&tipoVisAtt=&tipoPersona=&shadow_deputato=30350&idLegislatura=17

My issue is, when I use curl_download, it does not get the HTML like curl in-the-shell does: it gets the wrong "JSON" thing.

Question, therefore, is: how do I emulate curl in-the-shell's settings with curl in R, in order to get HTML instead of "JSON" through these awful links?

The output of curl -v for the above example is copied in this Gist. My curl in-the-shell version is copied below.

curl 7.30.0 (x86_64-apple-darwin13.0) libcurl/7.30.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz

Error in curl::curl_fetch_memory(url, handle = handle)

i get this error when i try to run gs_key function from googlesheets packages
temp= gs_key(ggdoc_2)
Authentication will be used.
Error in curl::curl_fetch_memory(url, handle = handle) :
Timeout was reached
How to slove it?

Problem loading shared library, causing failure to install package (0.9.3/master@a50e34643cf93ce578eb90a29f765c25760a12ae)

I had trouble tonight installing the curl package under R 3.2.1 with a manually built and installed curl library at 7.44.0, installed under a prefix $HOME/prefix. Many other source packages could be successfully built with the same configuration, and in the end some unclear combination of changes made the error go away. I'm still puzzled what the problem was, and I'm wondering whether R and/or the Makefile dealing correctly with my Makevars file. I'm not very familiar with the Makevars file in R, it could well be incorrect.

Can you look into this issue and tell me what has caused the problem and whether something should be fixed?

Context

~/.R/Makevars:

PKG_LIBS = -L${HOME}/prefix/lib        
PKG_CPPFLAGS = -I${HOME}/prefix/include

Environment variables

PKG_CONFIG_PATH=$HOME/prefix/lib/pkgconfig
Linux *.nl 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux

Sys.getenv('LD_LIBRARY_PATH')
# [1] "/usr/lib/R/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/default-java/jre/lib/amd64/server:/home/myname/prefix/lib/"

GCC

gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.9.2-10' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --with-arch-32=i586 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Debian 4.9.2-10)

Failure

install.packages('curl') resulted in this failure:

* installing *source* package ‘curl’ ...
** package ‘curl’ successfully unpacked and MD5 sums checked
Found pkg-config cflags and libs!
Using PKG_CFLAGS=-I/home/myname/prefix/include 
Using PKG_LIBS=-L/home/myname/prefix/lib -lcurl 
** libs
"/usr/lib/R/bin/Rscript" "../tools/options.R"
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c callbacks.c -o callbacks.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c curl.c -o curl.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c download.c -o download.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c escape.c -o escape.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c fetch.c -o fetch.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c form.c -o form.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c getdate.c -o getdate.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c handle.c -o handle.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c ieproxy.c -o ieproxy.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c init.c -o init.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c utils.c -o utils.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG "-I/home/myname/prefix/include"    "-I/home/myname/prefix/include" -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c version.c -o version.o
gcc -std=gnu99 -shared -L/usr/lib/R/lib -Wl,-z,relro -o curl.so callbacks.o curl.o download.o escape.o fetch.o form.o getdate.o handle.o ieproxy.o init.o utils.o version.o -L/home/myname/prefix/lib -L/usr/lib/R/lib -lR
installing to /home/myname/R/x86_64-pc-linux-gnu-library/3.2/curl/libs
** R
** data
*** moving datasets to lazyload DB
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/home/myname/R/x86_64-pc-linux-gnu-library/3.2/curl/libs/curl.so':
  /home/myname/R/x86_64-pc-linux-gnu-library/3.2/curl/libs/curl.so: undefined symbol: curl_easy_escape
Error: loading failed

Success

Commenting out the Makevars, and building from the current master revision with
R CMD INSTALL --configure-vars="INCLUDE_DIR=$HOME/prefix/include LIB_DIR=$HOME/prefix/lib" "$HOME/installers/curl"
is successful:

* installing *source* package ‘curl’ ...
** package ‘curl’ successfully unpacked and MD5 sums checked
Found pkg-config cflags and libs!
Using PKG_CFLAGS=-I/home/myname/prefix/include 
Using PKG_LIBS=-L/home/myname/prefix/lib -lcurl 
** libs
"/usr/lib/R/bin/Rscript" "../tools/options.R"
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c callbacks.c -o callbacks.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c curl.c -o curl.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c download.c -o download.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c escape.c -o escape.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c fetch.c -o fetch.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c form.c -o form.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c getdate.c -o getdate.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c handle.c -o handle.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c ieproxy.c -o ieproxy.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c init.c -o init.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c utils.c -o utils.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/home/myname/prefix/include      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g  -c version.c -o version.o
gcc -std=gnu99 -shared -L/usr/lib/R/lib -Wl,-z,relro -o curl.so callbacks.o curl.o download.o escape.o fetch.o form.o getdate.o handle.o ieproxy.o init.o utils.o version.o -L/home/myname/prefix/lib -lcurl -L/usr/lib/R/lib -lR
installing to /home/myname/R/x86_64-pc-linux-gnu-library/3.2/curl/libs
** R
** data
*** moving datasets to lazyload DB
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (curl)

Installation error: Error in source("tools/option_table.txt", env) unexpected input

I am trying to install the curl package but keep running into the following error. I've run install.packages("curl") in the R console and in RStudio and received this error both times. Running R CMD INSTALL curl_0.9.6.tar.gz also returns the error.

I'm on a Mac with El Capitan and using a Homebrew-installed version of R. Both Homebrew and the R package are up to date. I've installed R on Homebrew with the default settings and with the --with-openblas flag, both resulting in the same installation error when running install.packages("curl").

Some more information, with the error at the bottom:

brew info r

homebrew/science/r: stable 3.2.4 (bottled), HEAD
Software environment for statistical computing
http://www.r-project.org/
/usr/local/Cellar/r/3.2.4_1 (2,177 files, 54M) *
  Built from source with: --with-openblas
From: https://github.com/Homebrew/homebrew-science/blob/master/r.rb

r --version

R version 3.2.4 Revised (2016-03-16 r70336) -- "Very Secure Dishes"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin15.4.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
http://www.gnu.org/licenses/.

install.packages("curl")

Installing package into ‘/Users/tom/Library/R/3.2/library’
(as ‘lib’ is unspecified)
--- Please select a CRAN mirror for use in this session ---
HTTPS CRAN mirror

 1: 0-Cloud [https]                2: Austria [https]
 3: Belgium (Ghent) [https]        4: Chile [https]
 5: China (Beijing 4) [https]      6: Colombia (Cali) [https]
 7: France (Lyon 1) [https]        8: France (Lyon 2) [https]
 9: France (Paris 2) [https]      10: Germany (Münster) [https]
11: Iceland [https]               12: Italy (Padua) [https]
13: Japan (Tokyo) [https]         14: Mexico (Mexico City) [https]
15: New Zealand [https]           16: Russia (Moscow) [https]
17: Spain (A Coruña) [https]      18: Spain (Madrid) [https]
19: Switzerland [https]           20: UK (Bristol) [https]
21: UK (Cambridge) [https]        22: USA (CA 1) [https]
23: USA (KS) [https]              24: USA (MI 1) [https]
25: USA (TN) [https]              26: USA (TX) [https]
27: USA (WA) [https]              28: (HTTP mirrors)


Selection: 26
trying URL 'https://cran.revolutionanalytics.com/src/contrib/curl_0.9.6.tar.gz'
Content type 'application/octet-stream' length 262545 bytes (256 KB)
==================================================
downloaded 256 KB

* installing *source* package ‘curl’ ...
** package ‘curl’ successfully unpacked and MD5 sums checked
Found pkg-config cflags and libs!
Using PKG_CFLAGS=
Using PKG_LIBS=-lcurl
** libs
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c callbacks.c -o callbacks.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c curl.c -o curl.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c download.c -o download.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c escape.c -o escape.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c fetch.c -o fetch.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c form.c -o form.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c getdate.c -o getdate.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c handle.c -o handle.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c ieproxy.c -o ieproxy.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c init.c -o init.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c interrupt.c -o interrupt.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c nslookup.c -o nslookup.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c utils.c -o utils.o
clang -I/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/include -DNDEBUG  -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include    -fPIC  -g -O2  -c version.c -o version.o
clang -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/usr/local/Cellar/r/3.2.4_1/R.framework/Resources/lib -L/usr/local/opt/openblas/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/openssl/lib -L/usr/local/lib -o curl.so callbacks.o curl.o download.o escape.o fetch.o form.o getdate.o handle.o ieproxy.o init.o interrupt.o nslookup.o utils.o version.o -lcurl -F/usr/local/Cellar/r/3.2.4_1/R.framework/.. -framework R -lintl -Wl,-framework -Wl,CoreFoundation
installing to /Users/tom/Library/R/3.2/library/curl/libs
** R
** data
*** moving datasets to lazyload DB
** inst
** preparing package for lazy loading
Error in source("tools/option_table.txt", env) :
  tools/option_table.txt:1:1: unexpected input
1:
       ^
Error : unable to load R code in package ‘curl’
ERROR: lazy loading failed for package ‘curl’
* removing ‘/Users/tom/Library/R/3.2/library/curl’

The downloaded source packages are in
    ‘/private/var/folders/v1/6x2gz2jj1257syfr2_p05nkw0000gn/T/RtmpOf07ZT/downloaded_packages’
Warning message:
In install.packages("curl") :
  installation of package ‘curl’ had non-zero exit status

Any ideas why this might be happening and how I can fix it? Thanks.