Comments (7)
It looks like it is more complicated than I initially thought. It looks like dbplyr has renamed alot of backend functions from db_... to dbplyr_... .I will have to spend sometime investigating what has changed.
from noctua.
Thanks for highlighting this to me. I wasn't aware of the change to in_schema
. To my knowledge in_schema
is just a function and I won't be able to create a AthenaConnection method for it. However something like this should work:
#' @rdname backend_dbplyr
db_query_fields.AthenaConnection <- function(con, sql, ...) {
# check if sql is dbplyr schema
in_schema <- inherits(sql, c("dbplyr_schema", "ident"))
if(in_schema) {
if(inherits(sql, "ident")) {
if (grepl("\\.", sql)) {
dbms.name <- gsub("\\..*", "" , sql)
Table <- gsub(".*\\.", "" , sql)
} else {
dbms.name <- con@info$dbms.name
Table <- sql}
}
if(inherits(sql, "dbplyr_schema")){
dbms.name <- sql$schema
Table <- sql$table
}
# If ident, get the fields from Glue
tryCatch(
output <- con@ptr$glue$get_table(DatabaseName = dbms.name,
Name = Table)$Table)
col_names = vapply(output$StorageDescriptor$Columns, function(y) y$Name, FUN.VALUE = character(1))
partitions = vapply(output$PartitionKeys,function(y) y$Name, FUN.VALUE = character(1))
c(col_names, partitions)
} else { # If a subquery, query Athena for the fields
# return dplyr methods
sql_select <- pkg_method("sql_select", "dplyr")
sql_subquery <- pkg_method("sql_subquery", "dplyr")
dplyr_sql <- pkg_method("sql", "dplyr")
sql <- sql_select(con, dplyr_sql("*"), sql_subquery(con, sql), where = dplyr_sql("0 = 1"))
qry <- dbSendQuery(con, sql)
on.exit(dbClearResult(qry))
res <- dbFetch(qry, 0)
names(res)
}
}
from noctua.
Ah, that's right, I didn't realize the dbplyr_schema object offers those parts separately.
Great and fast work as always!
from noctua.
What do you think, will these renamings make it difficult to make things simultaneously compatible with the old and new dbplyr?
from noctua.
@OssiLehtinen from looking into the dbplyr
version 2.0.0 package it actually looks like it calls the same db_query_fields
function. This means noctua
won't need to worry about changing back end functions from 1 version to the other. I have also managed to identify what is causing the issue.
When the parameter sql
comes into db_query_fields.AthenaConnection
it has already be parsed. Meaning it already has quotes around it for example: '"default"."iris"'
This is causing the issue with the current implementation of db_query_fields.AthenaConnection
. To resolve we can just remove the quotation and it should work :)
#' @rdname backend_dbplyr
db_query_fields.AthenaConnection <- function(con, sql, ...) {
# check if sql is dbplyr schema
in_schema <- inherits(sql, "ident")
if(in_schema) {
if (grepl("\\.", sql)) {
schema_parts <- gsub('"', "", strsplit(sql, "\\.")[[1]])
} else {
schema_parts <- c(con@info$dbms.name, gsub('"', "", sql))}
# If dbplyr schema, get the fields from Glue
tryCatch(
output <- con@ptr$glue$get_table(DatabaseName = schema_parts[1],
Name = schema_parts[2])$Table)
col_names = vapply(output$StorageDescriptor$Columns, function(y) y$Name, FUN.VALUE = character(1))
partitions = vapply(output$PartitionKeys,function(y) y$Name, FUN.VALUE = character(1))
c(col_names, partitions)
} else {
# If a subquery, query Athena for the fields
# return dplyr methods
sql_select <- pkg_method("sql_select", "dplyr")
sql_subquery <- pkg_method("sql_subquery", "dplyr")
dplyr_sql <- pkg_method("sql", "dplyr")
sql <- sql_select(con, dplyr_sql("*"), sql_subquery(con, sql), where = dplyr_sql("0 = 1"))
qry <- dbSendQuery(con, sql)
on.exit(dbClearResult(qry))
res <- dbFetch(qry, 0)
names(res)
}
}
from noctua.
@OssiLehtinen merged PR #118. Please update your noctua
using:
remotes::install_github("dyfanjones/noctua")
I am planning to release these updates to the cran soon. However I would like include some improvements when uploading data to aws athena: #112 (if you could provide any benchmarking that would be really appreciated).
from noctua.
Hi Dyfan, I'm sorry for leaving you hanging. It's just been impossible to find the time to help with the benchmarking.
from noctua.
Related Issues (20)
- Prevent Noctua from printing Data Scanned -information HOT 7
- Release noctua 2.5.0 HOT 3
- Release noctua 2.6.0 HOT 1
- Sub-query fails with dplyr interface indicating "Only one sql statement is allowed" HOT 6
- cran-2.6.1 release
- Can I set various parameters in `.aws/config` file and have `DBI::dbConnect()` read those directly from that file? HOT 5
- Add catalog support HOT 26
- Column Bucketing
- Allow for Partition columns to change data types
- Can't write/append an empty data frame
- Connecting using long-term-creds returns Error 400 HOT 5
- dbFetch(..., n=small number) is quite slow when run on a large result set HOT 4
- `dbExistsTable()` doesn't work anymore HOT 3
- fix: for dbplyr 2.3.3.9000 +
- dbExistsTable() returns an incorrect result when the table name is defined by Id() or SQL() HOT 1
- InvalidRequestException with dbGetQuery HOT 4
- [Question]: Requesting guidance and best practices - Athena shinyApp with noctua HOT 2
- Unload option returns `null` results when `s3_staging_dir` is a bucket only HOT 1
- Speeding up `noctua` installation by cutting down unnecessary pieces HOT 3
- Add support for describe? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from noctua.