Make the createDF method even more concise about spark-daria HOT 6 CLOSED

@MrPowers this comes to mind immediately.

def createDF(rows: List[Row], fields: List[(String, DataType, Boolean)]) = {
  val structFields = fields.map(field => {
    StructField(field._1, field._2, field._3)
  })
  spark.createDF(rows, structFields)
}

val df = createDF(
  List(
    Row("Alice", 12),
    Row("Bob", 45)
  ), List(
    ("Name", StringType, true),
    ("Age", IntegerType, true)
  )
)

Having to do it without using List[Row] needs more thinking. I will let you know when I think of something.

from spark-daria.

@MrPowers I don't think it would be possible to eliminate Row() because, signature of apply for Row is def apply(values: Any*) it takes in variable number of arguments, limiting factor is Tuples in scala can not have variable length.

from spark-daria.

@MrPowers @lizparody we could do something like this,

implicit def value2tuple[T](x: T): Tuple1[T] = Tuple1(x)

def createDF(values: List[Product], fields: List[(String, DataType, Boolean)]) = {
  val rows = values.map(value => {
    Row(value.productIterator.toList: _*)
  })
  val structFields = fields.map(field => {
    StructField(field._1, field._2, field._3)
  })
  spark.createDF(rows, structFields)
}

val namesDF = createDF(
  List(
    ("Alice"),
    ("Bob")),
  List(
    ("Name", StringType, true)
  ))

val personDF = createDF(
  List(
    ("Alice", 12),
    ("Bob", 45)),
  List(
    ("Name", StringType, true),
    ("Age", IntegerType, true)
  ))

Implicit is needed to handle only for cases like namesDF where only column is present. Approach is not straightforward.

Another approach would be to construct,

implicit def toRow(value: (Any, Any)): Row = {
 Row(value._1, value._2)
}

Obvious issue with this would be the need to create implicit defs equals to all possible Tuple type class, which is 22. Thoughts?

from spark-daria.

@snithish - Awesome work!

I started coding this up in #7.

I couldn't get the implicit def value2tuple stuff to work... can you please take a look at the code / spec and let me know what I'm doing wrong?

We can try the implicit def toRow if we can't get the implicit def value2tuple approach to work.

Thanks again for the great work. Your Scala coding skills are impressive!

from spark-daria.

@MrPowers Had an icky feeling about how we were doing overloading. Cleaned the logic up using Generics and Pattern matching. Uses the same overloading logic in a concise manner in #8

from spark-daria.

Awesome work @snithish!!!! You're Scala skills are amazing and I'm going to start a study plan to get up to your level!

I merged in #7 / #8 and am excited about the new functionality. I'm going to package a new JAR file, upload it to Spark Packages, and start using the new syntax in spark-spec.

Thanks again for the help!

from spark-daria.