Given locate_credentials() might return region, or use the default, that value should be returned.

From @chrowe via cloudyr/aws.s3#240

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

Is it possible to retrieve S3 credentials using the config package instead of environment variables without have to specify them every time?

Put your code here:
I know that I can set them inline like this...
config.yml

default:
  s3:
    bucket: 'my_bucket'
    key: YOUR_AWS_ACCESS_KEY
    secret: YOUR_AWS_SECRET_ACCESS_KEY

R code

s3conf <- config::get("s3")
get_bucket(
  bucket = s3conf$bucket,
  key =  s3conf$key,
  secret =  s3conf$secret
) 

But it would be nice to be able to just set that once like you can with Sys.setenv

Hi, I been having a problem with use_credentials().

My variables in /.aws/credentialsare fine:
"AWS_ACCESS_KEY_ID"
"AWS_SECRET_ACCESS_KEY"
"AWS_DEFAULT_REGION"

I also set the Region to us-west-2, changing the default.

If I make use of aws s3 ls in the terminal it works fine, but R its not working,

Any ideas of what could be the problem?

This is the code I'm running:

use_credentials(profile = Sys.getenv("AWS_PROFILE", "default"),
                file = default_credentials_file())
bucketlist(verbose=TRUE)

And I get this:

Checking for credentials in user-supplied values
Checking for credentials in Environment Variables
Using Environment Variable 'AWS_ACCESS_KEY_ID' for AWS Access Key ID
Using Environment Variable 'AWS_SECRET_ACCESS_KEY' for AWS Secret Access Key
Using Environment Variable 'AWS_DEFAULT_REGION' for AWS Region ('us-west-2')
S3 Request URL: https://s3-us-west-2.amazonaws.com/
Executing request with AWS credentials
Parsing AWS API response
Client error: (403) Forbidden

Thanks for your help!

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

Our organization has recently switched to IAM Identity Center (formerly known as AWS SSO) for for authentication. Users obtain temporary credentials for a configured profile with the aws sso login command, as shown in this example from the AWS docs.

Configuring a named profile to use IAM Identity Center creates a JSON file in the ~/.aws/sso/cache directory (see here). I believe all AWS SDKs can use this this for authentication now, as an alternative to the traditional key / secret key combination.

Because credentials are not automatically added to the ~/.aws/.credentials file, the use_credentials() function is currently not aware of them, I think. As a workaround, I currently use the aws-export-credentials tool to update the .credentials file.

It would be great to support authentication with temporary credentials in aws.signature!

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

I'm using the aws.lambda package INSIDE an aws lambda (custom R runtime) and I'm seeming to have difficulties invoking a lambda getting:
UnrecognizedClientException The security token included in the request is invalid

I believe it has something to do with aws.signature rather than aws.lambda since locate_credentials is happening here.

I can manually provide:

• Sys.setenv("AWS_ACCESS_KEY_ID", "")
• Sys.setenv("AWS_SECRET_ACCESS_KEY", "")
• Sys.setenv("AWS_DEFAULT_REGION", "")

But I am getting the UnrecognizedClientException even though my role is allowed to Invoke my target lambda.

## load package
library("aws.lambda")

Sys.setenv("AWS_DEFAULT_REGION" = "us-west-2") # Needed for me to do manually for some reason (default is us-east-1 otherwise)
Sys.setenv(TZ = "GMT")

handler <- function(...) {
  payload = list(user_id = "1", event = "events.invite.email")
  result <- invoke_function(
    name = "MyLambdaName",
    payload = toJSON(payload, auto_unbox = TRUE),
    type = "RequestResponse",
    log = "None"
  )
  return(fromJSON(toJSON(list(code = 0))))
}

Session Info:

R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Amazon Linux AMI 2018.03

Matrix products: default
BLAS:   /opt/R/lib/libRblas.so
LAPACK: /opt/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] lubridate_1.7.4   feather_0.3.5     aws.lambda_0.1.6  aws.s3_0.3.12    
[5] data.table_1.12.2 jsonlite_1.6      httr_1.4.1        logging_0.10-108 

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.2          magrittr_1.5        xml2_1.2.2         
 [4] hms_0.5.1           aws.signature_0.5.2 R6_2.4.0           
 [7] rlang_0.4.0         stringr_1.4.0       tools_3.6.1        
[10] digest_0.6.20       tibble_2.1.3        crayon_1.3.4       
[13] base64enc_0.1-3     vctrs_0.2.0         curl_4.1           
[16] zeallot_0.1.0       stringi_1.4.3       compiler_3.6.1     
[19] pillar_1.4.2        backports_1.1.4     pkgconfig_2.0.2 

Note that communication to aws.s3 works great and so I know the correct roles are being assumed.

This works fine when running on my local machine which makes me think there is something wrong with how this package performs specifically when running on the aws lambda runtime environment

Simplest thing is to replicate the Python examples from AWS documentation.

And the individual steps described here.

Can you please release a new version (also to cran), with the accumulated small fixes / enhancements.

This makes the life a lot easier for those who do not directly install from github.

Also contributing the current version of the package to conda-forge is then easier.

Many thanks for your work!

Hi Thomas,

Great work on the 'awspack' packages. We're using them quite a bit!
Just a thought for future development - keeping AWS creds in dotfiles has drawbacks when working with a team. The new Parameter Store in AWS Systems Manager looks like a cool alternative.

Cheers
Kyle

https://aws.amazon.com/blogs/mt/interesting-articles-on-ec2-systems-manager-parameter-store/
https://stp5.net/blog/post/secrets-in-aws/

Hi Thomas,

I just downloaded your cool aws.signature package, and quickly gave it a try. As the R console output below demonstrates, there was no joy using either the read_credentials or use_credentials functions:

> library(aws.signature)
> packageVersion("aws.signature")
[1] ‘0.2.5’
> aws.signature::read_credentials(file = "~/.aws/credentials")
Error: 'read_credentials' is not an exported object from 'namespace:aws.signature'

The canonical_request, signature_v2_auth, signature_v4, signature_v4_auth, and string_to_sign functions are all available. Also, I've replicated the issue on both Ubuntu 14.04 and OS X.

I don't know anything about package building, so I can't send you a pull request. However, is it possible these functions weren't made available during your build process?

Thanks again for all of your hard work on the cloudyr packages!
Kyle

The current order of checking for EC2 then ECS credentials means that ECS credentials are never* used, as an ECS container will almost always be running on an EC2 host. The container credentials can give a job more, or different permissions compared to the EC2 instance.

I assume this change in ordering was made as the previous ECS detection was broken (See cloudyr/aws.ec2metadata#3). Once that is fixed, it should be possible to check for ECS credentials first, without that breaking credential discovery. Sorry for that error.

This would make it a little easier friendlier

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

When calling locate_credentials(profile='myprofile') I was surprised to see that it kept returning the credentials associated with a different profile. This appears to happen because locate_credentials gives precedence to credentials that are present in environment variables -- it'd be great to log or warn the user somehow that locate_credentials is disregarding the profile argument in favor of using the credentials found in the environment.

Not sure to what extent this behavior is intended -- if the maintainer has ideas about what you want to do in this situation I can take a stab at implementing.

A task in ECS can use a role that is different from the VMs that are part of the ECS cluster. If a task is configured with a task role then the environment variable will be populated with a AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable that which can be used like this to retrieve a set of temporary credentials:

curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

Reference: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

Possible bug in installation from Git master - I think it's due to the list in credentials_to_list

return(list(key = key, secret = secret, session_token = session_token, region = region))

As adding in else statements with NULLs for key, secret, session_token, etc, fixes the installation. Added example on PR.

> remotes::install_github("cloudyr/aws.signature")
Downloading GitHub repo cloudyr/aws.signature@master
✔  checking for file ‘/private/var/folders/np/fy2_8btj5334yh48qbl9m1rm0000gq/T/Rtmpd3rB2U/remotes156fd7b6f2753/cloudyr-aws.signature-df1ddb4/DESCRIPTION’ ...
─  preparing ‘aws.signature’:
✔  checking DESCRIPTION meta-information ...
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  building ‘aws.signature_0.4.5.tar.gz’
   
Installing package into ‘/Users/lissa.hyacinth/Rlibs’
(as ‘lib’ is unspecified)
* installing *source* package ‘aws.signature’ ...
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error: package or namespace load failed for ‘aws.signature’:
 .onLoad failed in loadNamespace() for 'aws.signature', details:
  call: credentials_to_list(cred, region = region, default_region = default_region, 
  error: object 'key' not found
Error: loading failed
Execution halted
``

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

My .aws/config file has entries like this

[profile power]
role_arn = arn:aws:iam::1234567891011:role/PowerUserRole
source_profile = default
region = eu-central-1

Which specifies both region and role. This works with aws cli and, surely, there is a common expectation that it should work here as well. Currently .aws/config seems to be ignored (related #18, cloudyr/aws.s3#168).

Would be great if,

1. profiles where searched in .aws./config first for region and role
2. Any roles be assumed automatically like in aws cli

Package functions allow us to override the default credentials found in ~/.aws/credentials with alternative values, but we have no comparable way to merely "blank" or unset those credentials. This creates unexpected behavior and errors that can be difficult to debug. For example, if you have no aws_secret_access_key defined in ~/.aws files, then the following code successfully uploads to this public upload-only AWS bucket as expected:

library("aws.signature")
library("aws.s3")
write.csv(iris, 'iris.csv')
aws.s3::put_object('iris.csv',
                   bucket = "submissions",
                   region = "data",
                   base_url = "ecoforecast.org")

If, however, you have your own secret access key in an ~/.aws directory, the same code will throw a 403 Access Forbidden error, even though the bucket does not require access, because it tries to authenticate anyway with your secret key. Note that attempting to set "AWS_SECRET_ACCESS_KEY" = "" env var to an empty string or passing secret="" to put_object does not help the situation. Here's a fully reproducible minimal example of the error; but note it will overwrite any existing ~/.aws/credentials

library("aws.signature")
library("aws.s3")
write.csv(iris, 'iris.csv')
dir.create("~/.aws")
writeLines(
"[default]
aws_access_key_id = ACCESS_KEY
aws_secret_access_key = SECRET_KEY
aws_session_token = TOKEN",
"~/.aws/credentials")

aws.s3::put_object('iris.csv',
                   bucket = "submissions",
                   region = "data",
                   secret = "", 
                   base_url = "ecoforecast.org")

Would you consider a way to temporarily unset default credentials, such as allowing a blank string or NULL value?

This error is also difficult for users to diagnose and debug. Being able to 'blank' credentials out would probably be useful in other contexts as well. (e.g. I'm not a security expert but in general I could imagine a user wanting finer-grained control of whether or not package functions were touching secure credentials when they weren't needed).

Example from cloudyr/aws.s3#166

Report from Vitalina Komashko:

system.time(aws.s3::copy_object(from_object = from_object[1], to_object = to_object[1],
from_bucket = from_bucket, to_bucket = to_bucket, verbose = TRUE))
Using Environment Variable 'AWS_ACCESS_KEY_ID' for AWS Access Key ID
Using Environment Variable 'AWS_SECRET_ACCESS_KEY' for AWS Secret Access Key

Checking bucket region using get_location('<REDACTED>')
Executing request using bucket region us-west-1
S3 Request URL: https://s3-us-west-1.amazonaws.com/<REDACTED>
Executing request with AWS credentials
Parsing AWS API response
Success: (200) OK
    user   system  elapsed
   0.836    0.674 2437.777

This runs about 10x slower than the AWS CLI. Issue may be with locate_credentials():

/usr/bin/time -l aws s3 cp s3://<REDACTED> --profile corporate
copy: s3://<REDACTED> to s3://<REDACTED>

      263.67 real         2.24 user         0.65 sys

I have an EC2 instance with an attached IAM role that should allow me to download a file from a bucket. In the bare instance, I can run

aws s3 cp s3://mybucket/file .

and it works fine. It also works from inside the docker container, running a bash terminal through Rstudio. I can even locate appropriate credentials through aws.signature and aws.ec2metadata !

tryCatch({
    aws.signature::locate_credentials()
}, error = print)
## $key
## [1] "<my key>"
## 
## $secret
## [1] "<my secret>"
##
## $session_token
## [1] "<my token>"
##
## $region
## [1] "us-east-1"

But when I call aws.signature::use_credentials(),

aws.signature::use_credentials()
## Error in read_credentials(file) : 
##   File '/home/rstudio/.aws/credentials' does not exist.

This means that all of the aws.s3::get_object(), etc. functions fail.

Is there something manual I need to do to link the IAM role to the use_credentials() function?

I have a section in my ~/.aws/credentials file that looks like this:

[devrole]
role_arn = arn:aws:iam::912345678902:role/my-role-dev
source_profile = default
s3 =
    signature_version = s3v4

That fails to parse properly:

> aws.signature::read_credentials()$devrole
$ROLE_ARN
[1] "arn:aws:iam::912345678902:role/my-role-dev"

$SOURCE_PROFILE
[1] "default"

$S3
[1] NA

$`    SIGNATURE_VERSION`
[1] "s3v4"

http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuration-file says that this is a "nested configuration value", but doesn't give much more guidance - I couldn't find the "Nested Configuration section" it mentions. However, this format does get parsed properly by the aws command-line tool itself.

I'm using:

> packageVersion('aws.signature')
[1] ‘0.3.5’

See below. Does aws.signature pkg support cross account profiles? I have situation where there is a special login profile (abcdef) which has a temporary token set(this token is periodically refreshed a utility). I need to access resources (like S3, SQS) in account with Profile marketingadmin. Both these profiles are separate accounts. When I try say list_queues etc, I get invalid token.

Any help appreciated.
PS - If you need, I can send precise configuration and error messages

Dinesh

[profile marketingadmin]
role_arn = arn:aws:iam::123456789012:role/marketingadmin
source_profile = abcdef

[abcdef]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
token= AQoDYXdzEPT//////////wEXAMPLEtc764bNrC9SAPBSM22wDOk4x4HIZ8j4FZTwdQWLWsKWHGBuFqwAeMicRXmxfpSPfIeoIYRqT
flfKD8YUuwthAx7mSEI/qkPpKPi/kMcGdQrmGdeehM4IC1NtBmUpp2wUE8phUZampKsburEDy0KPkyQDYwT7WZ0wq5VSXDvp75YU9
HFvlRd8Tx6q6fE8YQcHNVXAkiY9q6d+xo0rKwT38xVqr7ZD0u0iPPkUL64lIZbqBAz+scqKmlzm8FDrypNC9Yjc8fPOLn9FX9KSYv
KTr4rvx3iSIlTJabIQwj2ICCR/oLxBA==

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

~/.aws/credentials:
[default]
aws_access_key_id = ABC
aws_secret_access_key = DEF
# foo = bar

Expected: key=ABC, secret=DEF
Actual:

> str(aws.signature::locate_credentials())
List of 4
 $ key          : chr "ABC"
 $ secret       : chr "DEF\n# foo = bar"
 $ session_token: NULL
 $ region       : chr "us-east-1"

Note the embedded newline in the secret and the trailing content causing the authentication to fail.

> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.6

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_NZ.UTF-8/en_NZ.UTF-8/en_NZ.UTF-8/C/en_NZ.UTF-8/en_NZ.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] aws.s3_0.3.21

loaded via a namespace (and not attached):
[1] httr_1.4.1          compiler_3.6.3      R6_2.4.1           
[4] tools_3.6.3         base64enc_0.1-3     curl_4.3           
[7] aws.signature_0.5.2 xml2_1.3.0          digest_0.6.25      

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

Description

It'd be great if this package supported assuming roles from web identity tokens:

Calling AssumeRoleWithWebIdentity does not require the use of AWS security credentials. Therefore, you can distribute an application (for example, on mobile devices) that requests temporary security credentials without including long-term AWS credentials in the application. You also don't need to deploy server-based proxy services that use long-term AWS credentials. Instead, the identity of the caller is validated by using a token from the web identity provider.

From SO:

require("httr")
require("RCurl")
require("stringr")

#0: get key and secret from envvars, and set up request parameters

aws.key <- Sys.getenv("AWS_KEY")
aws.secret <- Sys.getenv("AWS_SECRET_KEY")

verb <- "GET"
zone <- "ec2.amazonaws.com"

ami.number <- "ami-xxxxxxxxx"

params <- list(
  Action = "DescribeImages",
  ImageId.1 = ami.number,
  Version = "2012-10-01",
  Expires = "2012-11-20T12:00:00Z",
  SignatureVersion = 2,
  SignatureMethod = "HmacSHA1",
  AWSAccessKeyId = aws.key)

# 1a: Sort the UTF-8 query string components by parameter name
params <- params[order(names(params))]

# 1b: URL encode the parameter name and values
params_e <- lapply(params, curlEscape)
names(params_e) <- curlEscape(names(params_e))
params_str <- str_c(names(params_e), "=", unlist(params_e), collapse = "&")
params_str <- gsub("%2E",".",gsub("%2D","-",params_str))

#2: Create the string to sign
string_to_sign <- str_c(
  toupper(verb), "\n",
  tolower(zone), "\n",
  "/", "\n",
  params_str
)

#3: Calculate an RFC 2104-compliant HMAC
#4: Convert the resulting value to base64.
hmac <- hmac_sha1(aws.secret, string_to_sign)

params$Signature <- hmac

GET(paste0("https://",zone),query=params)

Hey there. I may have encountered a bug.

While troubleshooting some access issues, I discovered that aws.signature::locate_credentials() is reporting my region as us-east-1, while I can verify that us-east-1 appears no where in my .aws profile, any stray environment variables, Renviorn, etc.

edit: I should stress that my profile config explicitly sets my default region to us-west-2, which is the region I'd expect to see reported.

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

Put your code here:

## load package
library("aws.s3")

## code goes here
aws.signature::locate_credentials()

$key
[1] "xxx"

$secret
[1] "xxx"

$session_token
NULL

$region
[1] "us-east-1"

## session info for your system
sessionInfo()

R version 3.6.3 (2020-02-29)
Platform: x86_64-apple-darwin17.7.0 (64-bit)
Running under: macOS High Sierra 10.13.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /usr/local/Cellar/openblas/0.3.9/lib/libopenblasp-r0.3.9.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.6.3        aws.ec2metadata_0.2.0 tools_3.6.3           base64enc_0.1-3       curl_4.3              aws.signature_0.5.2   jsonlite_1.6.1       
[8] digest_0.6.25         packrat_0.5.0    

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

I am building a shiny app that uses images and pdf documents from a private s3 bucket. Can I use the aws.signature package to get a presigned URL to plug into my shiny code (see below)? I think to do this I would need to use the signature_v4_auth function but I am confused about how to specify the arguments (action, query_args, canonical_headers, request_body, etc.).

If this is possible, is it secure or does it expose my key and secret?

ui  <- fluidPage(
  htmlOutput("image")
)

server <- function(input, output) {

  output$image <- renderText({

    presigned_url <- signature_v4_auth(service = "s3",
                                verb = "GET", 
                                request_body = "https://s3.amazonaws.com/mybucket/myimage.png",
                                key = Sys.getenv("AWS_ACCESS_KEY_ID"),
                                secret = Sys.getenv("AWS_SECRET_ACCESS_KEY")
                              )
    paste("<img src = ",'"',presigned_url'", height = "30px">', sep = "")
   })
}

Base R on Windows:

> path.expand("~")
[1] "C:/Users/kadrach/Documents"

This causes problems here, as the file is generally located in %USERPROFILE%\.aws\credentials, e.g. C:\Users\kadrach\.aws\credentials.

Please specify whether your issue is about:

• a possible bug
• [x ] a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request
  Error in rbind(info, getNamespaceInfo(env, "S3methods")) :
  number of columns of matrices must match (see arg 2)
  ERROR: lazy loading failed for package ‘aws.signature’

• removing ‘/Users/nannewton/Library/R/3.5/library/aws.signature’

If you are reporting (1) a bug or (2) a question about code, please supply:

• a fully reproducible example using a publicly available dataset (or provide your data)
• if an error is occurring, include the output of traceback() run immediately after the error occurs
• the output of sessionInfo()

Put your code here:

## load package
library("aws.signature")

## code goes here
install.packages("aws.signature", repos = c(cloudyr = "http://cloudyr.github.io/drat", getOption("repos")))

## session info for your system
sessionInfo()

Add check_for_keypair() function that does the following:

1. Checks whether key, secret key, and session token have been specified. If so, uses them.
2. If missing, check environment variables. If so, uses them.
3. If missing, check a central credentials file using use_credentials(). If so, uses them.
4. If missing, try to assume role from EC2 metadata via:

get_ec2_role <- function(role) {
  if (missing(role)) {
    role <- aws.ec2metadata::metadata$iam_role_names()
    if (!length(role)) {
      stop("No IAM role profile available in instance metadata")
    }
  }
  # return role credentials as list
  aws.ec2metadata::metadata$iam_role(role[1L])
}

This shouldn't set environment variables because credentials are temporary, so it should simply pass them through the signing functions.

All tests are passing on CRAN but they are really slow. Something is taking much longer than it should.

* using log directory 'd:/RCompile/CRANguest/R-devel/aws.signature.Rcheck'
* using R version 3.5.0 alpha (2018-03-30 r74499)
* using platform: x86_64-w64-mingw32 (64-bit)
* using session charset: ISO8859-1
* checking for file 'aws.signature/DESCRIPTION' ... OK
* checking extension type ... Package
* this is package 'aws.signature' version '0.4.0'
* checking CRAN incoming feasibility ... Note_to_CRAN_maintainers
Maintainer: 'Thomas J. Leeper <[email protected]>'
* checking package namespace information ... OK
* checking package dependencies ... OK
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for hidden files and directories ... OK
* checking for portable file names ... OK
* checking serialization versions ... OK
* checking whether package 'aws.signature' can be installed ... OK
* checking installed package size ... OK
* checking package directory ... OK
* checking DESCRIPTION meta-information ... OK
* checking top-level files ... OK
* checking for left-over files ... OK
* checking index information ... OK
* checking package subdirectories ... OK
* checking R files for non-ASCII characters ... OK
* checking R files for syntax errors ... OK
* loading checks for arch 'i386'
** checking whether the package can be loaded ... OK
** checking whether the package can be loaded with stated dependencies ... OK
** checking whether the package can be unloaded cleanly ... OK
** checking whether the namespace can be loaded with stated dependencies ... OK
** checking whether the namespace can be unloaded cleanly ... OK
** checking loading without being on the library search path ... OK
** checking use of S3 registration ... OK
* loading checks for arch 'x64'
** checking whether the package can be loaded ... OK
** checking whether the package can be loaded with stated dependencies ... OK
** checking whether the package can be unloaded cleanly ... OK
** checking whether the namespace can be loaded with stated dependencies ... OK
** checking whether the namespace can be unloaded cleanly ... OK
** checking loading without being on the library search path ... OK
** checking use of S3 registration ... OK
* checking dependencies in R code ... OK
* checking S3 generic/method consistency ... OK
* checking replacement functions ... OK
* checking foreign function calls ... OK
* checking R code for possible problems ... [46s] OK
* checking Rd files ... OK
* checking Rd metadata ... OK
* checking Rd line widths ... OK
* checking Rd cross-references ... OK
* checking for missing documentation entries ... OK
* checking for code/documentation mismatches ... OK
* checking Rd \usage sections ... OK
* checking Rd contents ... OK
* checking for unstated dependencies in examples ... OK
* checking examples ...
** running examples for arch 'i386' ... [61s] NOTE
Examples with CPU or elapsed time > 10s
                  user system elapsed
signature_v2_auth    0      0   20.00
signature_v4         0      0   20.02
** running examples for arch 'x64' ... [61s] NOTE
Examples with CPU or elapsed time > 10s
                  user system elapsed
signature_v2_auth 0.02      0   20.03
signature_v4      0.00      0   20.03
* checking for unstated dependencies in 'tests' ... OK
* checking tests ...
** running tests for arch 'i386' ... [152s] OK
  Running 'testthat.R' [151s]
** running tests for arch 'x64' ... [152s] OK
  Running 'testthat.R' [151s]
* checking PDF version of manual ... OK
* DONE
Status: 2 NOTEs

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

If you are reporting (1) a bug or (2) a question about code, please supply:

• a fully reproducible example using a publicly available dataset (or provide your data)
• if an error is occurring, include the output of traceback() run immediately after the error occurs
• the output of sessionInfo()

Put your code here:
I think this has to do with https://github.com/cloudyr/aws.signature/blob/master/R/read_credentials.R#L75 and https://github.com/cloudyr/aws.signature/blob/master/R/read_credentials.R#L86, where you get

#> $default$`[`
#> [1] NA

when you have multiple accounts.

Reproducible Example

library(aws.signature)
x = '[default]
aws_access_key_id = blashasdf
aws_secret_access_key = blashasdf
[new]
aws_access_key_id = blashasdf
aws_secret_access_key = blashasdf'
tfile = tempfile()
writeLines(x, tfile)
aws.signature::read_credentials(tfile)
#> $default
#> $default$AWS_ACCESS_KEY_ID
#> [1] "blashasdf"
#> 
#> $default$AWS_SECRET_ACCESS_KEY
#> [1] "blashasdf"
#> 
#> $default$`[`
#> [1] NA
#> 
#> 
#> $new
#> $new$AWS_ACCESS_KEY_ID
#> [1] "blashasdf"
#> 
#> $new$AWS_SECRET_ACCESS_KEY
#> [1] "blashasdf"
#> 
#> 
#> attr(,"class")
#> [1] "aws_credentials"

^{Created on 2019-05-09 by the reprex package (v0.2.1)}

Since credentials can be obtained from the ~/.aws/credentials file, I was wondering if the default-region parameter can also be obtained from the AWS CLI config file ~/.aws/config?

Please specify whether your issue is about:

• a possible bug
• [O] a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

Hello! I am new to R and I am trying to write some functionality for others to use AWS API Gateway. I was wondering if you would be able to provide an example or some direction for getting authorization needed for a GET Request using httr. What functions would I use to get the correct signature for the request?

R
key <- "FAKEKJKJBDKEY"
secret <- "FAKELKJLKSJSLKJSKEY"
region <- "us-east-2"
t <- format(Sys.time(),"%Y%m%dT%H%M%SZ", tz = "UTC")
signed <- signature_v4_auth(
  datetime = t,
  service = "execute-api",
  verb = "GET",
  action = "/test/demo/nameOfFile.json",
  canonical_headers = list(
    "Host"= 'fakehost-api.us-east-1.amazonaws.com',
    "X-AMZ-Date"=t
  ),
  request_body = "",
  signed_body=TRUE,
  key = key,
  secret= secret,
  region = region
)

response <- GET(url = "https://fakehost-api.us-east-1.amazonaws.com/test/demo/nameOfFile", add_headers(signed))

Thank you!

Use travisci to trigger dependent builds of other cloudyr AWS packages

For the .travis.yml here:

r_github_packages:
- cloudyr/travisci
after_success:
- Rscript -e "travisci::restart_last_build('cloudyr/aws.s3')"

Plus, need to encrypt a valid Travis-CI token.

For the .travis.yml at cloudyr/aws.s3, etc., use the GitHub version of aws.signature:

r_github_packages:
- cloudyr/aws.signature

Please specify whether your issue is about:

• [* ] a possible bug
• [* ] a suggested code or documentation change, improvement to the code, or feature request

aws.signature always inserts us-east-1 if no region is specified. However there is no way to specify 'no region' in url requests.

Amazon assumes us-east-1 is your region if you don't include it in the path based or vhost based access to the bucket. Some third party object stores depend on this functionality to operate properly. There is no need to add us-east-1 to the vhost or path if no region is specified. Doing so breaks functionality in this library.

Would love to get signature_v2_auth working for Amazon Product API (MWS, not product advertising).

The query is just a random UPC/EAN I picked to try and get my signatures working. I have set my secret key and access key etc using sys.setenv. I got that datetime straight out of the StringToSign produced by the MWS scratchpad.

I am doing this:
signature_v2_auth(datetime = '2017-06-19T22%3A19%3A12Z', verb = 'POST', service = 'mws.amazonservices.com', path = '/Products/2011-10-1', query_args = '5051892198158')

I am not getting the correct signature as a response. One lead as to what might be wrong, when using the Amazon Product API scratchpad, their example always wants SellerId and MarketplaceId and signature_v2_auth does not seem have any method of passing that information. They also want Action (in my case "ListMatchingProducts" however, there is also no place to pass this argument into signature_v2_auth.

Also, I am using signature_v2_auth because the scratchpad seems to default to version2 when I navigate to the Product API section.

Any help is appreciated. I have also tried to create signatures on my own using RCurl and RDigest, but failed. It would be useful if the documentation had better examples for version 4 as well, I would like to try version 4 but I'm not sure exactly what to put in for each argument such as canonical_headers and request_body.

Per https://github.com/cloudyr/aws.signature/blob/master/R/read_credentials.R#L67, parse_credentials is expecting key-value pairs separated by a =. This causes keys without the space (i.e. KEY=VALUE) to fail to parse.

AWS documentation is not fully consistent on the file format with the Boto docs specifying no spaces (http://boto3.readthedocs.io/en/latest/guide/configuration.html) and some docs using the space-equals-space format.

Changing to use a regex with optional whitespace would allow both formats to function.

There is no license provided for this excellent work (or any of the Cloudyr projects I looked at). Please consider a GPL or BSD/MIT license. Thank you.

Hi there! I recently took over maintenance of aws.lambda and aws.ec2. A few of us are using the cloudyr slack to coordinate ideas for the aws.* family of packages, and aws.signature is fairly central to that idea.

Are you in that slack? I see another Jon there, maybe that's you? If so, can you join us in the #aws channel? If not, can you join via https://forms.gle/GhZoRqLQLcSjcmYh9 ?

I look forward to working with you!

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

signature_v4_auth() produces unsigned payload requests, because it doesn't add x-amz-content-sha256 to the canonical headers (see https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html). It does compute the body hash which is needed, so it would be really easy for signature_v4_auth() to add an option to include x-amz-content-sha256 in the computation (e.g. signed_payload=TRUE). It is much harder for the caller to do it, because it would have to either call signature_v4_auth() twice (once to get the body hash and second time to actually sign it) or compute the body hash manually and add it to the explicit canonical headers.

Apparently some back-ends require signed payload (see cloudyr/aws.s3#362 ) and Amazon may, too, at some point so it would be good to add this.

An acceptable query list to httr can include integers, e.g. query = list("max-keys" = 2). This causes signature_v4_auth throws the relatively obscure error Error in strsplit(URL, "") : non-character argument even though httr is capable of handling query lists of mixed type and handles the conversions correctly.

This package is not being actively maintained. If you're interested in contributing or taking over, please express your interest here.

Changes introduced in the commit ad8e9ba break the signature generation if the request_body is not a character vector, e.g. a raw vector.

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

Cloudyr does not support container_providers (boto3 class) in its credential chain (boto3 credential providers chain).

When using a service like AWS Batch, boto3 will consume and use the permissions tied to the AWS Batch Job Role (step 9 here) before using the ECS Compute Cluster's Instance Role (step 4d here). As far as I can tell, this isn't included at all in cloudyR's locate_credentials

Can we add this provider to the cloudyR credential chain? It looks like you just need to hit 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI with that environment variable expanded and get the creds: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

Hello. I'm trying to generate a resigned URL using signature_v4_auth. This was inspired by this. I add to make some changes according to the documentation, and use URLencode a few times to make sure it matches what AWS is providing me with when I manually generate a presigned URL for the same test object.

Regardless of the change I do, I always get a signature mismatch. I've tried signing the body, or not.
I'm a little confused when I look at this documentation page. I'm not seeing the UNSIGNED-PAYLOAD anywhere in the code, and so as a result using their parameter I do not end up with the same signature. In my signature$CanonicalRequest I have the body hash instead.

My code so far:

get_aws_signed_url <- function(file,
                               bucket = "my_bucket_name",
                               timeout_seconds = 30,
                               key = "my_key",
                               secret = "my_secret",
                               token = "my_token",
                               region = "us-east-1") {
  # API Implmented according to https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html#query-string-auth-v4-signing-example
  algorithm <- "AWS4-HMAC-SHA256"
  time <- Sys.time()
  date_time <- format(time, "%Y%m%dT%H%M%SZ", tz = "UTC")

  # Build query parameters
  date <- glue::glue("/{format(time,'%Y%m%d', tz = 'UTC')}/") %>%
    URLencode(reserved = TRUE)
  region_encoded <- glue::glue("{region}/") %>% URLencode(reserved = TRUE)
  amzn <- "s3/aws4_request" %>% URLencode(reserved = TRUE)

  # Query parameters, this portion is implemented with the help of https://github.com/cloudyr/aws.s3/blob/master/R/s3HTTP.R
  request_body <- ""
  body_hash <- tolower(digest::digest(request_body,
                                      file = is.character(request_body) &&
                                        file.exists(request_body),
                                      algo = "sha256", serialize = FALSE))

  signature <- aws.signature::signature_v4_auth(datetime = date_time,
                                                region = region,
                                                service = "s3",
                                                verb = "GET",
                                                action = glue::glue("/{file}"),
                                                key = key,
                                                secret = secret,
                                                session_token = URLencode(token, reserved = TRUE),
                                                request_body = "",
                                                signed_body = TRUE,
                                                query_args = list(`X-Amz-Algorithm` = algorithm,
                                                                  `X-Amz-Credential` = glue::glue("{key}{date}{region_encoded}{amzn}"),
                                                                  `X-Amz-Date` = date_time,
                                                                  `X-Amz-Expires` = timeout_seconds,
                                                                  `X-Amz-SignedHeaders` = "host",
                                                                  `x-amz-content-sha256` = body_hash
                                                                  ),
                                                algorithm = algorithm,
                                                canonical_headers = list(host = glue("{bucket}.s3.amazonaws.com")),
                                                force_credentials = TRUE)
  
  return(glue::glue("https://{bucket}.s3.{region}.amazonaws.com/{file}?X-Amz-Algorithm={signature$Query$`X-Amz-Algorithm`}&X-Amz-Credential={signature$Query$`X-Amz-Credential`}&X-Amz-Date={signature$Query$`X-Amz-Date`}&X-Amz-Expires={signature$Query$`X-Amz-Expires`}&X-Amz-SignedHeaders={signature$Query$`X-Amz-SignedHeaders`}&X-Amz-Signature={signature$Signature}&X-Amz-Security-Token={signature$SessionToken}&x-amz-content-sha256={signature$Query$`x-amz-content-sha256`}"))
}


## session info
> sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.1

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] tictoc_1.0.1             glue_1.6.2               aws.signature_0.6.0      aws.s3_0.3.21            viridis_0.6.2            viridisLite_0.4.0       
 [7] DataExplorer_0.8.2       gt_0.6.0                 DT_0.23                  reactlog_1.1.0           shinydashboardPlus_2.0.3 shinycssloaders_1.0.0   
[13] vroom_1.6.0              patchwork_1.1.1          rebus_0.1-3              forcats_0.5.1            stringr_1.4.0            dplyr_1.0.9             
[19] purrr_0.3.4              readr_2.1.2              tidyr_1.2.0              tibble_3.1.8             tidyverse_1.3.1          shinyWidgets_0.7.5      
[25] plotly_4.10.0            ggplot2_3.4.0            shinyjs_2.1.0            shinydashboard_0.7.2     shiny_1.7.1             

loaded via a namespace (and not attached):
 [1] colorspace_2.0-3      ellipsis_0.3.2        rprojroot_2.0.3       base64enc_0.1-3       fs_1.5.2              rstudioapi_0.14       farver_2.1.0         
 [8] bit64_4.0.5           fansi_1.0.3           lubridate_1.8.0       xml2_1.3.3            cachem_1.0.6          knitr_1.39            jsonlite_1.8.2       
[15] broom_1.0.1           dbplyr_2.1.1          png_0.1-8             RAthena_2.6.0         compiler_4.2.1        httr_1.4.2            backports_1.4.1      
[22] assertthat_0.2.1      Matrix_1.4-1          fastmap_1.1.0         lazyeval_0.2.2        cli_3.4.1             later_1.3.0           htmltools_0.5.3      
[29] tools_4.2.1           igraph_1.3.2          gtable_0.3.0          rebus.base_0.0-3      Rcpp_1.0.9            cellranger_1.1.0      jquerylib_0.1.3      
[36] vctrs_0.5.1           crosstalk_1.1.1       xfun_0.31             networkD3_0.4         rebus.datetimes_0.0-1 rvest_1.0.1           mime_0.12            
[43] lifecycle_1.0.3       rebus.numbers_0.0-1   scales_1.2.1          hms_1.0.0             promises_1.2.0.1      parallel_4.2.1        yaml_2.3.5           
[50] curl_4.3.2            reticulate_1.26       gridExtra_2.3         sass_0.4.1            stringi_1.7.6         checkmate_2.1.0       rlang_1.0.6          
[57] pkgconfig_2.0.3       evaluate_0.15         lattice_0.20-45       fontawesome_0.4.0     labeling_0.4.2        htmlwidgets_1.5.4     bit_4.0.4            
[64] tidyselect_1.1.2      here_1.0.1            magrittr_2.0.3        R6_2.5.1              generics_0.1.0        DBI_1.1.3             pillar_1.8.1         
[71] haven_2.5.0           withr_2.5.0           rebus.unicode_0.0-2   modelr_0.1.8          crayon_1.4.1          utf8_1.2.2            tzdb_0.3.0           
[78] rmarkdown_2.14        grid_4.2.1            readxl_1.4.0          data.table_1.14.2     reprex_2.0.1          digest_0.6.29         xtable_1.8-4         
[85] httpuv_1.6.5          arrow_10.0.0          munsell_0.5.0         bslib_0.3.1  

I somehow feel that some things in there could/should be simplified - like I have to manually generate the credentials, because signature$Credential is not encoded properly, or I'm getting this completely backward. Apologies if that's the case.

Thanks ahead of time for the clarifications.

Test for aws.s3 fail when trying to pass headers. The API responds with a "SignatureDoesNotMatch" error.

Add functionality to sign based upon .aws/credentials files of the form:

[default]
aws_access_key_id = ACCESS_KEY
aws_secret_access_key = SECRET_KEY
aws_session_token = TOKEN

[Alice]
aws_access_key_id = Alice_access_key_ID
aws_secret_access_key = Alice_secret_access_key

[Bob]
aws_access_key_id = Bob_access_key_ID
aws_secret_access_key = Bob_secret_access_key

The current test handling could be improved by using mocks to stub out the necessary file system code. At the moment, tests can't be run on my machine as /tmp is on another file system.

after I run "aws.signature::use_credentials()", it does read the information from ~/.aws/credentials. However, the secrete key string misses the last character. I don't know why. Everything else is correct, it just clip out the last character. Because of that, function like s3read_using reports an error.

I manually corrected the secrete key by using: Sys.setenv("AWS_SECRET_ACCESS_KEY"=Correct Key), everything works fine.

I think there is a bug in the "parse_credentials" function in the file "read_credentials.R"