r-lib / httr Goto Github PK
View Code? Open in Web Editor NEWhttr: a friendly http package for R
Home Page: https://httr.r-lib.org
License: Other
httr: a friendly http package for R
Home Page: https://httr.r-lib.org
License: Other
I was trying to extend the github demo to create a repo on github
. I did the oauth
dance successfully and was able to execute the GET requests correctly. But when I tried to create a repo on github
, I failed.
Github documentation says that I need to pass the name of the repo as the scope, but I was unable to figure out where I should pass this. The POST request I sent was
POST('https://api.github.com/user/repos', github_sig)
PS. For some reason oauth2.0_token
is not exported. Is that intentional?
Provide implementation for google as reference for implementors.
Possible in principle, but currently make it very easy to crash R.
library(httr)
down_progress <- function() {
bar <- NULL
function(down, up) {
total <- down[1]
if (total == 0) {
cat(".")
return()
}
current <- down[2]
if (is.null(bar)) {
bar <<- txtProgressBar(max = total)
}
setTxtProgressBar(bar, current)
}
}
prg <- config(noprogress = 0L, progressfunction = down_progress())
x <- GET("http://courses.had.co.nz/12-oscon/slides.zip", prg)
x <- GET("https://dl.dropbox.com/u/41902/ggplot2-course.zip", prg)
I'm probably doing something wrong here, but it's not clear how I specify content requests in PUT.
I'm trying to access the Figshare API with httr.
Authentication with httr and oauth 1 seems to work fine:
figshare_auth <- function(cKey = getOption("FigshareKey", stop("Missing Figshare consumer key")),
cSecret = getOption("FigsharePrivateKey", stop("Missing Figshare app secret")),
token = getOption("FigshareToken", stop("Missing Figshare token")),
token_secret = getOption("FigsharePrivateToken", stop("Missing Figshare Secret Token"))){
require(httr)
myapp <- oauth_app("rfigshare", key = cKey, secret=cSecret)
sig <- httr:::sign_ouath1.0(myapp, token = token, token_secret = token_secret)
}
session <- figshare_auth()
And most of the basic GET commands work. But I try to a post method and get:
figshare_create <-
function(title="test", description="description of test", type="dataset", session){
require(RJSONIO)
base <- "http://api.figshare.com/v1"
method <- paste("my_data/articles")
request <- paste(base, method, sep="/")
meta <- toJSON(list("title"=title, "description"=description,
"type"=type, "Content-Type" = "application/json"))
POST(request, config=session, body=meta)
}
and get the error
figshare_create(session=session)
Response [http://api.figshare.com/v1/my_data/articles]
Status: 401
{"error": "Incorect content_type. Expected application/json, received:"}
The GET -based methods work fine (those calls also require the 'session' variable, but I guess this could still be an authentication issue since they shouldn't actually need to authenticate since those should be public methods)?
Any ideas? (I'm building out API in rfigshare package here: https://github.com/ropensci/rfigshare)
Probably using the ROauth package
Currently, oauth_listener
uses the built in httpd server which has the limitation that if help has already been started you can not change the port. Switching to httpuv would remedy this (and the blocking nature of httpuv might actually be an advantage). It should be a simple switch given that httpuv supports Rook.
I haven't had this error prior to this with other APIs, but the Springer Journals API for some reason gives something in the header that makes parsed_content think that it is html or perhaps xml that is returned when json is actually returned. I want JSON output, but I get error when wrapping the below call in parsed_content()
For example
tt <- GET(http://api.springer.com/metadata/json?api_key=2pn639bshyrju39zfewygkgd&q=dna)
Then element 4 in the list gives
tt[4]
content-type in tt[4] is given as "text/html; charset=utf-8", which is perhaps where the problem is?
Thanks! Scott
See email to Duncan
I'm getting this curl
call to work just fine
curl --form "[email protected]" http://taxosaurus.org/submit
Where myfile.txt is
"Dichorisandra diederichsanae"
"Echeveria longiflora"
"Gottschea cuspidiloba"
"Virola loretensis"
"Pterobryella praenitens"
"Ochthocosmus floribundus"
"Geissois superba"
"Ginkgo biloba"
"Escallonia angustifolia"
"Pterigynandrum julaceum"
But I can't quite get this to work using POST
or postForm
. I think I'm missing something obvious, but I don't know what it might be.
Hi,
Thanks for this great package, incredibly useful and easy to use :-) One issue I have encountered is with setting a "Referer" header if the Referer URL is https...
> test <- GET('https://google.ca', c(add_headers('Referer'='https://www.google.com'), verbose()))
* Adding handle: conn: 0x4f02c30
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 1 (0x4f02c30) send_pipe: 1, recv_pipe: 0
* About to connect() to google.ca port 443 (#1)
* Trying 74.125.228.120...
* Connected to google.ca (74.125.228.120) port 443 (#1)
* successfully set certificate verify locations:
* CAfile: /bla/R/x86_64-pc-linux-gnu-library/3.0/RCurl/CurlSSL/cacert.pem
CApath: /etc/ssl/certs
* SSL connection using ECDHE-RSA-RC4-SHA
* Server certificate:
* subject: C=US; ST=California; L=Mountain View; O=Google Inc; CN=*.google.ca
* start date: 2013-11-20 14:48:09 GMT
* expire date: 2014-03-20 00:00:00 GMT
* subjectAltName: google.ca matched
* issuer: C=US; O=Google Inc; CN=Google Internet Authority G2
* SSL certificate verify ok.
> GET / HTTP/1.1
Host: google.ca
Accept: */*
Accept-Encoding: gzip
https://www.google.com
< HTTP/1.1 301 Moved Permanently
< Location: https://www.google.ca/
< Content-Type: text/html; charset=UTF-8
[...]
Note that the Referer header value is present in the request but the key is MIA...
Your assistance would be greatly appreciated.
Thanks!
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
Reset config options after each call. May be possible to do in make_request
by setting all defined config options to have an empty string.
HTTR_SECRET=$(cat /dev/urandom | head -c 10000 | openssl sha1)
travis encrypt HTTR_SECRET=$HTTR_SECRET
openssl aes-256-cbc -k "$HTTR_SECRET" -in keys.rds.enc -out keys.rds
before_script
, decrypt file with openssl aes-256-cbc -k "$HTTR_SECRET" -in keys.rds.enc -d -a -out keys.rds
This is probably my failure to implement correctly rather than a bug, but I cannot get the file upload step to work on the FigShare API.
Perhaps I am not handling the content-type correctly, or perhaps I am using the call to "upload_file" in the wrong place. Here's the error from the verbose debug message (secure keys wiped):
> fs_upload(myid, "rfigshare.png", session=session)
* Connection #0 seems to be dead!
* Closing connection #0
* About to connect() to api.figshare.com port 80 (#0)
* Trying 107.20.170.85... * connected
> PUT /v1//my_data/articles/95796/files HTTP/1.1
Host: api.figshare.com
Accept: */*
Accept-Encoding: gzip
Cookie: figsh.09=339b6670bd7b6947765a00000000000000063580
Content-Type: multipart/form-data
Authorization: OAuth oauth_consumer_key="qMXXXXXXXXXXXXXXX", oauth_nonce="AD2U40JuVE", oauth_signature="ZXrwXXXXXXXXXXVHwQ%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1347300651", oauth_token="SrXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", oauth_version="1.0"
Content-Length: 72
Expect: 100-continue
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 401 Unauthorized
< Server: nginx/0.8.54
< Date: Mon, 10 Sep 2012 18:10:53 GMT
< Content-Type: application/json; charset=UTF-8
< Connection: keep-alive
< Content-Length: 383
<
* Connection #0 to host api.figshare.com left intact
>
e.g. don't add to parsers list. Instead use type = "text"
and parse yourself.
How can the authentication token be refreshed using the refresh_token instead of the user needing to grant access again? Is it possible for httr to automatically refresh using the refresh_token when the access_token ised but has expired?
Problem described as follows:
With the OAuth 2.0 client secret for my app held in an environment variable named "GANALYTICS_CONSUMER_SECRET"
Note: The client_id
below has been replaced with a dummy value.
Using Windows 7 64-bit and R version 3.0.0, the below code in the R GUI will successfully request access from the user to Google Analytics, then pass the resulting authentication token back to R via a redirect to the localhost. (Note this does not currently work in RStudio.)
library(httr)
authorize.urlPath <- "auth"
access.urlPath <- "token"
base_url <- "https://accounts.google.com/o/oauth2"
appname <- "GANALYTICS"
client_id <- "123456789012.apps.googleusercontent.com"
scope_url <- "https://www.googleapis.com/auth/analytics.readonly"
endpoint <- oauth_endpoint(
request = NULL,
authorize = authorize.urlPath,
access = access.urlPath,
base_url = base_url
)
app <- oauth_app(
appname = appname,
key = client_id
)
access_token <- oauth2.0_token(
endpoint = endpoint,
app = app,
scope = scope_url
)
Passing access_token
to sign_oauth2.0
results in the a malformed configuration query if passed to GET
, i.e. the list object is flattened to a string starting as follows "access_token=list%28access%5Ftoken%20%3D%20%22
and ending %22%29"
However, passing access_token$access_token
to sign_oauth2.0
works.
Therefore the following is executed to create the configure and send the request to Google Analytics (note the Google Analytics profile ID has been replaced with a dummy value):
request.config <- sign_oauth2.0(access_token = access_token$access_token)
query.url <- "https://www.googleapis.com/analytics/v3/data/ga?ids=ga%3A12345678&dimensions=ga%3Adate&metrics=ga%3Avisits&start-date=2013-04-21&end-date=2013-05-05&max-results=50"`
response <- GET(
url = query.url,
request.config
)
The above successfully retrieves the requested data from Google Analytics.
However, the token expires after 3600 seconds and needs to be refreshed using the refresh_token
item from the access_token
list object:
> print(access_token)
$access_token
[1] "########"
$token_type
[1] "Bearer"
$expires_in
[1] 3600
$refresh_token
[1] "########"
Note: the $access_token
and $refresh_token
have been masked from view.
Once the token has expired, if used as above, it results in Google Analytics returning a 401 HTTP status code with the following authentication error message "Login Required"
How can the authentication token be refreshed using the refresh_token instead of the user needing to grant access again? Is it possible for httr to automatically refresh using the refresh_token when the access_token ised but has expired?
Thank you.
e.g.
library(httr)
search <- "httr package daterange:2456294-2456659"
r <- GET("https://www.google.com/search", query = list(q = search))
stop_for_status(r)
c <- content(r)
xpathSApply(c, "//a/@href")
from http://stackoverflow.com/questions/20888625/scrape-url-from-google-search-using-r-httr
Probably same basic model as OAuth, but need to check.
it looks like there is a problem connecting to the fitbit api using the oauth1.0 method: after tokens are retrieved from the api, all calls are rejected with an invalid signature error.
It seems that the issue is with the oauth_signature function using curlEscape, while the OAuth 1.0 specification requires curlPercenEncode (see http://oauth.net/core/1.0/#encoding_parameters for details).
Changing the code solves the issue for the fitbit site, although I could not test whether this breaks anything for other api .
A script I've written using httr works fine interactively, but dies with the following error when run via Rscript:
Error in getCurlHandle(cookiefile = cookie_path, .defaults = list()) :
could not find function "getClass"
Calls: POST ... handle_url -> find_handle -> handle -> getCurlHandle -> .Call
The workaround is to add library(methods)
to the user code. Presumably httr should do that...
I am not sure if this is an issue with httr
or RStudio
, but I was not able to complete the browser authentication if I run the github demo from within RStudio
. I suspect it is because RStudio opens a different port by default. Here is part of a message which might help.
In server$start(port = 1410, quiet = TRUE) :
RStudio has already started the web server on port 32387
I receive a 404 error when trying to authenticate on a client machine using RStudio Server. I believe the 404 derives from the fact that the endpoint is localhost
rather than the IP address of the machine hosting the Rstudio server session.
Wondering if there's a workaround for this.
regarding this SO post on httr and this SO post on RCurl - it's currently impossible to load a large file if that file is bigger than available RAM.
i have 64GB of RAM, so these two lines work for me in R x64 and do not work in R 32-bit (both on windows) --
require(httr)
x <- GET( "http://www2.census.gov/acs2011_5yr/pums/csv_pus.zip" )
> parse_url("http://[email protected]:80/?a=1&b=2")$password
[1] NA
Should be NULL
.
try demo,get this:
Error: could not find function "oauth2.0_token"
Error: could not find function "oauth2.0"
google, facebook, github, ...
Simple acceptance tests that omit/include those components which are optional should suffice. Perhaps in conjunction with build_url
.
There may be useful logic in RCurl::isBinaryContent
and RCurl::dynCurlReader
> library(httr)
> url <- "ftp://ftp.ensembl.org/pub/release-69/gtf/ailuropoda_melanoleuca/Ailuropoda_melanoleuca.ailMel1.69.gtf.gz"
> HEAD(url)
Error in from:to : result would be too long a vector
In addition: Warning messages:
1: In max(i) : no non-missing arguments to max; returning -Inf
2: In max(i) : no non-missing arguments to max; returning -Inf
Although
curl -I ftp://ftp.ensembl.org/pub/release-69/gtf/ailuropoda_melanoleuca/Ailuropoda_melanoleuca.ailMel1.69.gtf.gz
Works fine.
Running the Google Demo with a custom app from the API Console and a devtools github httr install,
i'm getting an error for step3 "Get OAuth credentials":
cred <- oauth2.0(google, myapp,
scope = "https://www.googleapis.com/auth/userinfo.profile")
Fehler: konnte Funktion "oauth2.0" nicht finden
de-en:
Error: could not find function "oauth2.0"
Trying to manipulate your Google API demo for use with the Google Docs API. I can't seem to get past the point of defining cred
.
Everything "seems" to work fine:
library(httr)
google <- oauth_endpoint(
NULL,
"auth",
"token",
base_url = "https://accounts.google.com/o/oauth2"
)
myapp <- oauth_app(
"google",
"100...apps.googleusercontent.com",
secret="MY_SECRET"
)
cred <- oauth2.0_token(
google,
myapp,
scope = "https://www.googleapis.com/auth/drive.readonly"
)
A browser window asking for authorization pops up, and then an otherwise blank page loads saying "Authentication complete - you can now close this page and return to R.".
However, looks at cred
, I see:
error
"redirect_uri_mismatch"
I'm not at all familiar with OAuth, but would be happy to debug upon instruction.
Here's my RScript:
library('httr')
myapi <- oauth_endpoint(NULL, "authorize", "security/authorize", base_url= "https://myapi.com/v2")
myapp <- oauth_app("myapi", "my_key", "my_secret")
cred <- oauth2.0_token(myapi, myapp)
Here's the result:
Loading required package: Rook
Loading required package: tools
Loading required package: brew
Waiting for authentication in browser...
Authentication complete.
Error in file(con, "r") : cannot open the connection
Here's what the browser says after I login:
HTTP GET: http://localhost:43580/custom/OAuth/cred?code=TUZ2SlJkZDBDUzFReFNFalFaUU9keHk1bllCSTU0cC9TaXZTY1ZITXptSmV4K2NCS3h5bytEckVFbU1ESXFSUnZVZ3NoRWZUcHNoN1FtcTU3cVNPUkFUdk5mNHB4QkpKVjFmTENwc0N4ZkRqYWRyZW9weXUrMGtjZGpvak5lTGNPYUlsWk1PUHJMS2JzOHlST0VEZjB0YVFWTXhIZTZueTRoNHJYLzI3SEhDR0pOT3NTTlJHcG9QNW1vRGdteDM4dHF4Z1E5L0RFRTZuayt4UFYxMmRFWEZMYStsZFdDZmc1bmw5Q2NWdTI1ekdxQUtaYnAvbmQxMFlhUWxvUTIyTFdXeG90UHpJbGJjUUVvLzRPeGdGTUQzRmlDWms4eWJ6Q0VMR3NtbGV5RDlLNEFDNUNadm5wN2w1UnhxdVlNWm9WQi9Sck1HWHlHaFIvMTZMRFZ1M2pnPT0=
Authentication complete - you can now close this page and return to R.
If I execute the script again, it hangs at "Waiting for authentication in browser..." and the browser says this after I login:
HTTP GET: http://localhost:1410/custom/OAuth/cred?code=UTJCSzZRbGk0TnUwU2EwQ1hFUFc0ODZnQ25RWUlOdUxzMVcwSmRTS25maEw4dnVsYjI2dGxOemFnMTErU09EYlVpYk5FaG85MmNjV0taTHFsV2MzdlE5Z244bWN4NExLbVVCSWpGMExVcjZZU0pkSEtFeG94MmVaQkdCR2JuZ01oM0hYZDIvVUt3cnZlZFJxU3ppYWFTM3R0SDIyUFRSamVZdjNMYThRbjhkRkZKYitZY3N6MlRJamtrMkl2eHJEQk44Y1JyZVlBOTV5QlJacVVzQXlkd0E2c2N0SFNSN05LTjhxSzA4YTdCUkVSK08vYkJvSUxqVTFNbytJcStVSjJmdHdtNldZTFFMVncrek1hODdoaXd4ZFVGR1JMajlvYURoMVRvZUdkcFZrQzhYaHpvWHhHZWwzK25jZFlvbk5ObXlqNDdSUUdvVlpkKzlkbFRYbnNnPT0=
Oops! Google Chrome could not connect to localhost:1410
At least for google:
Tests
E.g. http://httpbin.org/gzip
.
Need to retrieve binary data by default, and then convert to text based on content type.
Download from http://curl.haxx.se/ca/cacert.pem
and set default config to use it.
e.g.
parsers$`text/csv` <- function(x, ...) read.csv(text = x, stringsAsFactors = FALSE, ...)
library(httr)
url <- "http://www.kegg.jp/kegg-bin/show_pathway?eco00260/eco%3ab0002%09%23ff0000,%23ffff00/eco%3ac00263%09%2300ff00,yellow"
GET(url)
results in:
Response [http://www.kegg.jp/kegg-bin/show_pathway?eco00260/eco%3ab0002%09%23ff0000,%23ffff00/eco%3ac00263%09%2300ff00,yellow=NA]
Status: 200
Content-type: text/html
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Usage</title>
<link rel="stylesheet" href="/css/kegg2.css" type="text/css">
</head>
<body>
<h3>cannot create 0 (no link ko_kid)type </h3>
</body>
</html>
Whereas if I paste that URL into a browser, I get different (much longer) results.
Am I doing something wrong or is it a bug?
Dan
sessionInfo()
R Under development (unstable) (2012-12-15 r61341)
Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] httr_0.2
loaded via a namespace (and not attached):
[1] digest_0.6.0 RCurl_1.95-3 stringr_0.6.2 tools_3.0.0
Can you bump the version number on the github copy to indicate that this version isn't identical to the 0.1.1 version on CRAN? (README also says the package isn't on CRAN at all yet). Thanks!
i.e. if output is json (or xml?) should be able to get back as an R list.
parsed_content
for XML content returns in class XMLDocument. It would be much more useful to have an object of class XMLInternalDocument, as returned by XML::xmlParse
, which can be used with getNodeSet
, xpathSApply
etc.
Maybe I'm just a dense user in this case though. How do I go from XMLDocument to an XMLInternalDocument object anyhow? (Normally I've used getURLContent
followed by xmlParse
, which gives this format).
Thanks!
oauth_callback with no args, how to change a callback url like http://127.0.0.1:1410/
Set cookielist
or cookiejar
to tempfile, and then parse info$cookielist
, splitting into tabs and selecting elements 6 and 7.
For http status messages, to make it easier for package users to react to specific types of error. stop_for_status
should generate error of correct type
to e.g. retry after pause, or after some other action (like updating auths), or returning a default value, or retry with verbose, or browser, ...
Maybe call it something like on_failure
?
There seems to be a problem with the current version on CRAN: after loading the library, calling the sign_ouath1.0 function gives the following error message
> sign_ouath1.0
Error: object 'sign_ouath1.0' not found
although the corresponding help vignette is there.
Is there (or could you consider adding) a method to escape HTML characters for URLs, e.g. replacing :
with %3A
, etc?
I'm trying to make a GET request to the greycite API like so:
http://greycite.knowledgeblog.org/json?uri=http%3A%2F%2Fwww.carlboettiger.info
But would rather have the user pass the query url as such "http://carlboettiger.info". (I thought I recalled you had a method for this but couldn't find it just now).
Thanks!
The set of mime types used by parse_auto is a fixed property of any given version of httr. It is generated manually by the package using the function cache_media, which pulls the set of mime types from a hopefully authoritative source.
However, it's missing such gems as gz: application/x-gzip.
It can't be replaced (as far as I know) during use of the package because I can't replace data in a namespace (only functions) using techniques such as unlockBinding.
request: allow package user to specify a mime type source to replace or extend mime types during a session, or add a simple mime type source.
addMimeTypes(nameValueList)
replaceMimeTypes(nameValueList)
addMimeType("gz","application/x-gzip")
Related to: simpler mechanism to replace mime type handlers.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.