Comments (10)
Hi @davidski,
From my testing I get the following error:
Error in f() : No credentials provided
From paws
and noctua
> library(DBI)
>
> con <- dbConnect(noctua::athena(), profile_name ="child-profile", region = "eu-west-1")
Error in f() : No credentials provided
>
> athena <- paws::athena(config = list(credentials=list(profile="child-profile"), region = "eu-west-1"))
> athena$list_work_groups()
Error in f() : No credentials provided
Ignore above i had the wrong profile linked
from noctua.
Do you perhaps have an ~/.aws/credentials
file setup, with creds for the demo_profle
profile? In my configuration, I do not use a credentials
file at all. https://github.com/99designs/aws-vault stores credentials in my keychain and call STS for ephemeral credentials, which is what paws.common now supports.
EDIT: corrected the profile name to match yours
from noctua.
Oh my giddy.... Somewhere in my testing I fat fingered a profile name in my live configuration. With that fixed, everything works seamlessly. While I'm delighted that works, I'm very sorry to take up so much of your time for user error. I'll blame the US Thanksgiving holiday. đ Thanks again for the package and all the support. Really looking forward to putting this through it's paces!
from noctua.
Hi @davidski,
The current implementation of noctua
should be able to take full advantage of the recent updates that paws.common
has to offer.
When building the config list for paws
, noctua
does the following rough steps:
- Check for Hardcoded credentials
- Check for environment variables
- if all fails leaves the rest to paws.common
The function nocuta:::cred_set
simply builds the config list and passes it to paws.common, if that list is empty (user using .aws/config files) then paws.common
should be able to do the rest :)
noctua:::cred_set(aws_access_key_id = NULL,
aws_secret_access_key = NULL,
aws_session_token = NULL,
profile_name = NULL,
region_name = NULL)
# returns
list()
nocuta:::cred_set
also lets you pass in profile_name (which are set in .aws/config
files). For example:
noctua:::cred_set(aws_access_key_id = NULL,
aws_secret_access_key = NULL,
aws_session_token = NULL,
profile_name = "made-up",
region_name = NULL)
# returns
$credentials
$credentials$profile
[1] "made-up"
Changing the .aws/config
example:
# .aws/config
[profile made_up]
region = eu-west-1
output = json
# R: noctua
library(DBI)
con <- dbConnect(noctua::athena(), profile_name = "made_up")
dbGetInfo(con)
$profile_name
[1] "made_up"
$s3_staging
[1] ########
$dbms.name
[1] "default"
$work_group
[1] "primary"
$poll_interval
NULL
$encryption_option
NULL
$kms_key
NULL
$expiration
NULL
$keyboard_interrupt
[1] TRUE
$region_name
[1] "eu-west-1"
$paws
[1] "0.1.9"
$noctua
[1] "1.9.1"
# .aws/config change:
[profile made_up]
region = eu-west-2
output = json
# R: noctua
library(DBI)
con <- dbConnect(noctua::athena(), profile_name = "made_up")
dbGetInfo(con)
$profile_name
[1] "made_up"
$s3_staging
[1] ######
$dbms.name
[1] "default"
$work_group
[1] "primary"
$poll_interval
NULL
$encryption_option
NULL
$kms_key
NULL
$expiration
NULL
$keyboard_interrupt
[1] TRUE
$region_name
[1] "eu-west-2"
$paws
[1] "0.1.9"
$noctua
[1] "1.9.1"
Please check #90 for any extra information.
Please let me know if I haven't been able to answer your question đ
from noctua.
Thanks for the quick response! This is a complicated workflow to explain and I appreciate the patience.
I do get different behavior with noctua
(CRAN v1.9.1 - MacOS) as compared with paws
. Here's a redacted configuration to demonstrate what works with the boto-powered CLI and, most recently, with paws.common
but not with noctua
.
~/.aws/config
[profile base]
region = us-east-2
credential_process = aws-vault exec --json base --prompt=osascript
[profile child-profile]
source_profile = base
include_profile = base
role_arn = arn:aws:iam::XXXX/XXXX
The following works under paws
. By working, I mean there is a prompt (via aws-vault) for a MFA token if required, and the call "just works" đ This was implemented in paws
via paws-r/paws#328
athena <- paws::athena(config = list(credentials=list(profile="child-profile"), region = "us-east-2"))
athena$list_work_groups()
The following fails under noctua
with Error in f() : No credentials provided
library(DBI)
con <- dbConnect(noctua::athena(),
profile_name = "child-profile",
region = "us-east-2",
s3_staging_dir = 's3://XXX')
from noctua.
Update:
This is very confusing. When I updated my .aws/config
to point to the correct profile noctua
and paws
worked perfectly.
# .aws/config
[default]
region = eu-west-1
output = json
[profile child-profile]
source_profile = default
include_profile = default
role_arn = arn:aws:iam::XXX/XXX
> library(DBI)
>
> con <- dbConnect(noctua::athena(), profile_name ="child-profile", region = "eu-west-1")
>
> athena <- paws::athena(config = list(credentials=list(profile="child-profile"), region = "eu-west-1"))
> athena$list_work_groups()
$WorkGroups
$WorkGroups[[1]]
$WorkGroups[[1]]$Name
[1] "primary"
$WorkGroups[[1]]$State
[1] "ENABLED"
$WorkGroups[[1]]$Description
[1] ""
$WorkGroups[[1]]$CreationTime
[1] "2019-08-22 15:14:47 GMT"
$NextToken
character(0)
> dbGetQuery(con, "select * from iris")
Info: (Data scanned: 3.63 KB)
sepal_length sepal_width petal_length petal_width species
1: 5.1 3.5 1.4 0.2 setosa
2: 4.9 3.0 1.4 0.2 setosa
3: 4.7 3.2 1.3 0.2 setosa
4: 4.6 3.1 1.5 0.2 setosa
5: 5.0 3.6 1.4 0.2 setosa
---
146: 6.7 3.0 5.2 2.3 virginica
147: 6.3 2.5 5.0 1.9 virginica
148: 6.5 3.0 5.2 2.0 virginica
149: 6.2 3.4 5.4 2.3 virginica
150: 5.9 3.0 5.1 1.8 virginica
I will try to see if it works when I point it to a staging profile
from noctua.
When I created a staging profile (demo_profile) noctua
and paws
seem to be working fine:
# .aws/config
[default]
region = eu-west-1
output = json
[profile demo_profile]
region = eu-west-1
output = json
[profile child-profile]
source_profile = demo_profile
include_profile = demo_profile
role_arn = arn:aws:iam::XXX/XXX
> library(DBI)
>
> con <- dbConnect(noctua::athena(), profile_name ="child-profile", region = "eu-west-1")
> athena <- paws::athena(config = list(credentials=list(profile="child-profile"), region = "eu-west-1"))
> athena$list_work_groups()
$WorkGroups
$WorkGroups[[1]]
$WorkGroups[[1]]$Name
[1] "primary"
$WorkGroups[[1]]$State
[1] "ENABLED"
$WorkGroups[[1]]$Description
[1] ""
$WorkGroups[[1]]$CreationTime
[1] "2019-08-22 15:14:47 GMT"
$NextToken
character(0)
> dbGetQuery(con, "select * from iris")
Info: (Data scanned: 3.63 KB)
sepal_length sepal_width petal_length petal_width species
1: 5.1 3.5 1.4 0.2 setosa
2: 4.9 3.0 1.4 0.2 setosa
3: 4.7 3.2 1.3 0.2 setosa
4: 4.6 3.1 1.5 0.2 setosa
5: 5.0 3.6 1.4 0.2 setosa
---
146: 6.7 3.0 5.2 2.3 virginica
147: 6.3 2.5 5.0 1.9 virginica
148: 6.5 3.0 5.2 2.0 virginica
149: 6.2 3.4 5.4 2.3 virginica
150: 5.9 3.0 5.1 1.8 virginica
from noctua.
Thanks @davidski for bring aws-vault to my attention it looks pretty awesome. From my testing, noctua
should be able to support paws
using aws-vault. Just created a video hope it helps to demonstrate noctua
using paws
new features :)
# .aws/config file
[default]
region = eu-west-1
output = json
[profile base]
region = eu-west-1
credential_process = aws-vault exec --json base --prompt=osascript
[profile child-profile]
source_profile = base
include_profile = base
role_arn = arn:aws:iam::XXX/XXX
# aws-vault listed sessions
> aws-vault list
Profile Credentials Sessions
======= =========== ========
default - -
base base sts.GetSessionToken:51m33s
child-profile - -
from noctua.
Awesome that it's working for you! Now I have to figure out what's different in my environment.
from noctua.
@davidski no worries I am glad it's working for you. Plus I am happy you pointed me in the direction of aws-vault
. Feel free to raise any future tickets.
from noctua.
Related Issues (20)
- Caching issue HOT 2
- Method to set unload at a package level HOT 1
- Release noctua 2.4.0 on to cran
- Prevent Noctua from printing Data Scanned -information HOT 7
- Release noctua 2.5.0 HOT 3
- Release noctua 2.6.0 HOT 1
- Sub-query fails with dplyr interface indicating "Only one sql statement is allowed" HOT 6
- cran-2.6.1 release
- Can I set various parameters in `.aws/config` file and have `DBI::dbConnect()` read those directly from that file? HOT 5
- Add catalog support HOT 26
- Column Bucketing
- Allow for Partition columns to change data types
- Can't write/append an empty data frame
- Connecting using long-term-creds returns Error 400 HOT 5
- dbFetch(..., n=small number) is quite slow when run on a large result set HOT 4
- `dbExistsTable()` doesn't work anymore HOT 3
- fix: for dbplyr 2.3.3.9000 +
- dbExistsTable() returns an incorrect result when the table name is defined by Id() or SQL() HOT 1
- InvalidRequestException with dbGetQuery HOT 4
- [Question]: Requesting guidance and best practices - Athena shinyApp with noctua HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
đ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. đđđ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google â¤ī¸ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from noctua.