Giter Club home page Giter Club logo

Comments (7)

DyfanJones avatar DyfanJones commented on September 25, 2024 1

It looks like it is more complicated than I initially thought. It looks like dbplyr has renamed alot of backend functions from db_... to dbplyr_... .I will have to spend sometime investigating what has changed.

from noctua.

DyfanJones avatar DyfanJones commented on September 25, 2024

Thanks for highlighting this to me. I wasn't aware of the change to in_schema. To my knowledge in_schema is just a function and I won't be able to create a AthenaConnection method for it. However something like this should work:

#' @rdname backend_dbplyr
db_query_fields.AthenaConnection <- function(con, sql, ...) {
  
  # check if sql is dbplyr schema
  in_schema <- inherits(sql, c("dbplyr_schema", "ident"))
  
  if(in_schema) {
    
    if(inherits(sql, "ident")) {
      if (grepl("\\.", sql)) {
        dbms.name <- gsub("\\..*", "" , sql)
        Table <- gsub(".*\\.", "" , sql)
      } else {
        dbms.name <- con@info$dbms.name
        Table <- sql}
    }
    
    if(inherits(sql, "dbplyr_schema")){
      dbms.name <- sql$schema
      Table <- sql$table
    }
    
    # If ident, get the fields from Glue
    
    tryCatch(
      output <- con@ptr$glue$get_table(DatabaseName = dbms.name,
                                       Name = Table)$Table)
    
    col_names = vapply(output$StorageDescriptor$Columns, function(y) y$Name, FUN.VALUE = character(1))
    partitions = vapply(output$PartitionKeys,function(y) y$Name, FUN.VALUE = character(1))
    
    c(col_names, partitions)
    
  } else { # If a subquery, query Athena for the fields
    # return dplyr methods
    sql_select <- pkg_method("sql_select", "dplyr")
    sql_subquery <- pkg_method("sql_subquery", "dplyr")
    dplyr_sql <- pkg_method("sql", "dplyr")
    
    sql <- sql_select(con, dplyr_sql("*"), sql_subquery(con, sql), where = dplyr_sql("0 = 1"))
    qry <- dbSendQuery(con, sql)
    on.exit(dbClearResult(qry))
    
    res <- dbFetch(qry, 0)
    names(res)
  }
}

from noctua.

OssiLehtinen avatar OssiLehtinen commented on September 25, 2024

Ah, that's right, I didn't realize the dbplyr_schema object offers those parts separately.

Great and fast work as always!

from noctua.

OssiLehtinen avatar OssiLehtinen commented on September 25, 2024

What do you think, will these renamings make it difficult to make things simultaneously compatible with the old and new dbplyr?

from noctua.

DyfanJones avatar DyfanJones commented on September 25, 2024

@OssiLehtinen from looking into the dbplyr version 2.0.0 package it actually looks like it calls the same db_query_fields function. This means noctua won't need to worry about changing back end functions from 1 version to the other. I have also managed to identify what is causing the issue.

When the parameter sql comes into db_query_fields.AthenaConnection it has already be parsed. Meaning it already has quotes around it for example: '"default"."iris"'

This is causing the issue with the current implementation of db_query_fields.AthenaConnection. To resolve we can just remove the quotation and it should work :)

#' @rdname backend_dbplyr
db_query_fields.AthenaConnection <- function(con, sql, ...) {
  
  # check if sql is dbplyr schema
  in_schema <- inherits(sql, "ident")
  
  if(in_schema) {
    if (grepl("\\.", sql)) {
      schema_parts <- gsub('"', "", strsplit(sql, "\\.")[[1]])
    } else {
      schema_parts <- c(con@info$dbms.name, gsub('"', "", sql))}
    
    # If dbplyr schema, get the fields from Glue
    tryCatch(
      output <- con@ptr$glue$get_table(DatabaseName = schema_parts[1],
                                       Name = schema_parts[2])$Table)
    
    col_names = vapply(output$StorageDescriptor$Columns, function(y) y$Name, FUN.VALUE = character(1))
    partitions = vapply(output$PartitionKeys,function(y) y$Name, FUN.VALUE = character(1))
    
    c(col_names, partitions)
    
  } else { 
    # If a subquery, query Athena for the fields
    # return dplyr methods
    sql_select <- pkg_method("sql_select", "dplyr")
    sql_subquery <- pkg_method("sql_subquery", "dplyr")
    dplyr_sql <- pkg_method("sql", "dplyr")
    
    sql <- sql_select(con, dplyr_sql("*"), sql_subquery(con, sql), where = dplyr_sql("0 = 1"))
    qry <- dbSendQuery(con, sql)
    on.exit(dbClearResult(qry))
    
    res <- dbFetch(qry, 0)
    names(res)
  }
}

from noctua.

DyfanJones avatar DyfanJones commented on September 25, 2024

@OssiLehtinen merged PR #118. Please update your noctua using:

remotes::install_github("dyfanjones/noctua")

I am planning to release these updates to the cran soon. However I would like include some improvements when uploading data to aws athena: #112 (if you could provide any benchmarking that would be really appreciated).

from noctua.

OssiLehtinen avatar OssiLehtinen commented on September 25, 2024

Hi Dyfan, I'm sorry for leaving you hanging. It's just been impossible to find the time to help with the benchmarking.

from noctua.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.