nevillelyh / shapeless-datatype Goto Github PK

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/example.proto

Maps which need to be mapped via an implicit conversion (e.g. String => Array[Byte]), or contain different case classes (even if they share the same structure: e.g. mapping from Map[String, X] to Map[String, Y] where X(a: Int) and Y(a: Int)), do not compile.

Hi,

I don't think this library currently supports Maps for Avro. Will this be supported in the (near) future or is it possible to add it as custom type?
Thanks

I have a 4 converters where I'm calling toEntityBuilder and if I call sbt clean compile I consistently get the following StackOverflowError - https://gist.github.com/grahamar/f9100091d61f0313327a07e6aba06435#file-stack-txt

But If I don't clean it will work, which makes me think it's a race condition... All my code looks like - https://gist.github.com/grahamar/f9100091d61f0313327a07e6aba06435#file-authorconverter-scala, nothing complex.

My types are all very straight forward too:

case class Author(
  id: Long = 0L,
  source_id: Option[Long] = None,
  name: Option[String] = None,
  enabled: Option[Boolean] = None,
  avatar_url: Option[String] = None,
  extracted_name: Option[String] = None,
  item_name: Option[String] = None,
  unique_id: Option[String] = None,
  deleted_at: Option[String] = None,
  created_at: Option[String] = None,
  updated_at: Option[String] = None
)

Any help is appreciated, I don't have enough knowledge of shapeless to know where too look either...

In case if case class contains case class, inner case class correctly interpreted as RECORD however, if custom mapper provided (as in example in documentation), schema generator treats it as field of custom type, but ToMappable derived in still treats it as RECORD.

This test (as per documentation) succeeds:

import shapeless.datatype.bigquery._
import java.net.URI

implicit val uriBigQueryType = BigQueryType.at[URI]("STRING")(v => URI.create(v.toString), _.toString)

case class Page(uri: URI, rank: Int)

val t = BigQueryType[Page]
val r = t.toTableRow(Page(URI.create("www.google.com"), 42))

// this is as expected
r.get("uri") ==== "www.google.com"

while this fails:

import shapeless.datatype.bigquery._

case class Yuri(ss: String)
implicit val uriBigQueryType = BigQueryType.at[Yuri]("STRING")(v => new Yuri(v.toString), _.ss)

case class Page(uri: Yuri, rank: Int)

val t = BigQueryType[Page]
val r = t.toTableRow(Page(new Yuri("www.google.com"), 42))

// this FAILS, instead of string, it contains linked hash map.
r.get("uri") ==== "www.google.com"

Note that the only difference here is that instead of system non-case URI class, custom case class Yuri is used (and if I switch it from being case class to normal class, the problem is gone).

I think that reason is that ToMappable.hconsToMappable0 (which constructs basing on internal structure of case class) implicit has bigger priority than ToMappable.hconsToMappable1 (which uses externally provided mapper).

This can be achieved the same way as done in #11 which would add support for maps too, however I have found that to be able to use a Set I also needed to define a CanBuild[T, Set[T]] so I guess the same would be true for Map

So that users don't have to import package._ to get all the implicits.

This serializes, but BigQuerySchema does not know about the custom types. Note: releases is a SCollection[Release].

      val bqType: BigQueryType[Release] = BigQueryType[Release]
      releases
        .map { a =>
          implicit val enumType
            : BaseBigQueryMappableType[MyEnum] =
            BigQueryType.at[MyEnum]("STRING")(
              a => MyEnum.fromName(a.toString).orNull,
              _.name)
          bqType.toTableRow(a)
        }
        .saveAsBigQuery(
          table,
          schema = BigQuerySchema[Release]
        )

This does not serialize:

      implicit val enumType
        : BaseBigQueryMappableType[MyEnum] =
        BigQueryType.at[MyEnum]("STRING")(
          a => MyEnum.fromName(a.toString).orNull,
          _.name)
      val bqType: BigQueryType[Release] = BigQueryType[Release]
      val schema = BigQuerySchema[Release]

      releases
        .map { a =>
          bqType.toTableRow(a)
        }
        .saveAsBigQuery(
          table,
          schema = schema
        )

Exception in thread "main" java.lang.IllegalArgumentException: unable to serialize anonymous function map ...
Caused by: java.io.NotSerializableException ...

Core is quite universal, I think you can safely crossbuild it for scala JVM/JS/Native

Right now, a RecordMapper only supports Option[X] => Option[Y] mappings. It'd be nice to enhance that by:

1. Supporting X => Option[Y]
2. Supporting Option[Y] => X.

I know this is controversial, as it could lead to runtime exceptions when trying to extract None. I'd suggest to provide this as an optional feature, only enabled manually by the user when deemed appropriate.

A bit of background: we need this in order to map from our Domain Model to the classes generated by ScalaPB and viceversa. Since we are using Protobuf 3 syntax, it automatically generates Options for all message references inside messages. This Option enhancement would help us to solve the impedance mismatch between the Domain Model and the ScalaPB model.

I'll provide you with a PR later.

https://github.com/propensive/magnolia

This would allow us to map Sets and Maps, instead of being restricted to collections that inherit from Seq.

Right now TensorFlowType uses the same traits as BigQueryType and DatastoreType but doesn't implement methods that handle nested types. We should either duplicate the implicit chain logic or figure out a way to fail it at compile time.

Related to #21. Also check compatibility among recent versions.

Is it possible to have a generic AvroType? So for instance like the following class:

class AvroDeserializationSchema[IN: ClassTag : TypeTag] {
   val t = AvroType[IN]
}

I'm struggling with this since I want to make a generic avro (de)serializer.
Thanks in advance!