zalando / grafter Goto Github PK

1. create the configuration
2. create the application
3. make singletons and tune components
4. start the application
5. stop the application

Intellij does not understand the macros used by Grafter, because it does not compile Scala code to perform its syntax checks. This is an issue because Intellij will fail to detect required imports correctly, and may clean up imports it believes to be unused which will break an application.

This is probably a band-aid rather than a properly viable solution, but I've created a basic Intellij plugin to recreate the work that the macros are doing within the Intellij AST. I'm finding it useful, but feedback would be appreciated.

https://plugins.jetbrains.com/idea/plugin/9387-grafter-macro-support
https://github.com/shanethehat/grafter-intellij-plugin

Do a pass on the documentation to link pages with previous/next page so that it can be understood as a long tutorial.

From the documentation / @etorreborre's talk, I was under the impression that when using a singleton/rewriter, it makes sure the specific module only gets initialized once. However, when running the example below (dependencies akka-cluster & grafter), the ActorSystemProvider is being initialized twice.

I'm fairly sure the mistake is probably somewhere on my end, but at the same time am unable to track it down, or deduce from the documentation what I am doing wrong. Any ideas here?

package demo

import akka.actor._
import cats.Eval
import cats.data.Reader
import org.zalando.grafter._
import org.zalando.grafter.GenericReader._
import org.zalando.grafter.macros._
import org.zalando.grafter.syntax.rewriter._

@readers
case class ApplicationConfig(
    akka: AkkaConfig
)

case class AkkaConfig(
    clusterName: String
)

trait DB

object DB {
  implicit def reader: Reader[ApplicationConfig, DB] =
    AkkaDB.reader
}

case class AkkaDB(actorSystemProvider: ActorSystemProvider) extends DB with Start {
  override def start: Eval[StartResult] = StartResult.eval("DB") {}
}

object AkkaDB {
  implicit def reader: Reader[ApplicationConfig, AkkaDB] =
    genericReader
}

trait ActorSystemProvider {
  def system: ActorSystem
}

object ActorSystemProvider {
  implicit def reader[A](implicit akkaConfigReader: Reader[A, AkkaConfig]): Reader[A, ActorSystemProvider] =
    DefaultActorSystemProvider.reader[A]
}

@dependentReader
final case class DefaultActorSystemProvider private (config: AkkaConfig)
    extends ActorSystemProvider
    with Start
    with Stop {

  lazy val systemEval: Eval[ActorSystem] =
    Eval.later(ActorSystem(config.clusterName))

  lazy val system: ActorSystem =
    systemEval.value

  override def start: Eval[StartResult] = {
    StartResult.eval("akka")(system)
  }

  override def stop: Eval[StopResult] = {
    StopResult.eval("akka")(system.terminate())
  }
}

object DefaultActorSystemProvider {
  implicit def reader[A](implicit akkaConfigReader: Reader[A, AkkaConfig]): Reader[A, ActorSystemProvider] =
    akkaConfigReader.map(DefaultActorSystemProvider.apply)
}

@reader[ApplicationConfig]
case class Application(actorSystemProvider: ActorSystemProvider, db: DB)

object Main extends App {

  pureconfig.loadConfig[ApplicationConfig]("app") match {
    case Left(error) ⇒ println(error)
    case Right(applicationConfig) ⇒
      val application = GenericReader[ApplicationConfig, Application]
        .run(applicationConfig)
        .singleton[ActorSystemProvider]

      Rewriter
        .start(application)
        .flatMap {
          case results if results.exists(!_.success) ⇒
            println(results)
            Eval.now(())
          case _ ⇒
            Eval.now(())
        }
        .value

  }

}

The current "Visualize" allows to create a graph from an application and then filter nodes from it. A consequence is that the children nodes of the filtered out nodes are simply removed from the graph. It would be nicer to just exclude the filtered out nodes and keep their children (unless filtered out themselves).

Do not advise to use a package object as it leads to increased compilation times.

1. the possibility to create singletons with a predicate function
2. collect (breadth-first) the first component of a given type

At the moment the replace method returns the application graph with no indication of the success of the replacement. We could add a method replaceAtLeastOnce which would return Option[Application] with None as a result if nothing was replaced.

or something like that. See https://github.com/tpolecat/tut

This could help with compilation times but we need to validate this hypothesis.

If we apply the singleton rewriting over the following graph, we don't get the same results depending on the order we do the rewriting:

case class U(name: String) {
  override def toString = s"${getClass.getSimpleName}($name: ${System.identityHashCode(this)})"
  override def equals(a: Any) = System.identityHashCode(this) == System.identityHashCode(a)
}
case class V(name: String, u: U) {
  override def toString = s"${getClass.getSimpleName}($name: ${System.identityHashCode(this)}, $u)"
  override def equals(a: Any) = System.identityHashCode(this) == System.identityHashCode(a)
}
case class T(u: U, v1: V, v2: V){
  override def toString = s"${getClass.getSimpleName}(${System.identityHashCode(this)}, $u, $v1, $v2)"
}

val v = V("v1", U("u2"))
val graph = T(U("u1"), v, v)

   T
 / |\
U1 | \
   | /
   V
   |
   U2

graph.singleton[U].singleton[V] !== graph.singleton[V].singleton[U]

When an implicit Reader definition is missing somewhere in the graph, Shapeless will not produce a very clear message

[error]       could not find implicit value for parameter gen: shapeless.LabelledGeneric.Aux[A,Repr]
[error]       genericReader
[error]       ^
[error] one error found

It would be a lot better to read

[error]  missing reader: Reader[AppConfig, HttpConfig]

I have unfortunately no clue how to do this :-)

It would be very useful to be able to display a graph of components as a dot file, showing dependencies and where singletons have been made, with various options to remove unnecessary information (display dependencies only for example).

The following bit of code (excluded all imports and ceremony for brevity) fails to compile. (The usecase might be specific, but it would be great if it would also just work)

@readers
case class ApplicationConfig(
  dbConfig: DbConfig
)

@reader[ApplicationConfig]
case class Foo(config: ApplicationConfig, db: DB)

It fails because there's no implicit reader instance (Reader[ApplicationConfig, ApplicationConfig]) to be found.

It would be resolved if the @readers annotation would also generate the following

object ApplicationConfig {
  implicit val reader: Reader[ApplicationConfig, ApplicationConfig] =
    Reader.apply(identity)
}

Creating the implicit reader definitions for a given ApplicationConfig class is repetitive and can probably provided by creating a @Reader macro annotation.

Instead of writing

case class ApplicationConfig(
  threadPoolConfig: ThreadPoolConfig,
  serverConfig: ServerConfig
)

object ApplicationConfig {
   implicit def threadPoolConfigReader: Reader[ApplicationConfig, ThreadPoolConfig] = 
      Reader(_.threadPoolConfig)

   implicit def serverConfigReader: Reader[ApplicationConfig, ServerConfig] = 
      Reader(_.serverConfig)
}

There could be an annotation generating the implicit definitions automatically:

@Reader
case class ApplicationConfig(
  threadPoolConfig: ThreadPoolConfig,
  serverConfig: ServerConfig
)

Note that all the attributes of a config must have a different type. Otherwise we will generate conflicting Reader instances. There should be a compiler error in that case.

I have some external components, that I can't really create through grafter's functionality (they are created by an fs2.Stream).

I had the idea to add some kind of context case class to my configuration, with the "external" dependencies. However, doing it this way requires a type parameter in my config class (case class Config[F[_]](...)), which gives this error:

[error] ...: A1 takes no type parameters, expected: one
[error] @readers

Should I just not do it this way? If not, is there a way to do something like this?

@jcranky I asked the codecov support for help on that but I still don't have an answer.

Those did not exist in Kiama 2.1.0-RC1.

For interfaces we need to manually specify which instance need to be selected:

trait Database

object Database {
  implicit def reader: Reader[ApplicationConfig, Database] =
    PostgresDatabase.reader
}

It would be nice to have an annotation for this:

@default[PostgresDatabase]
trait Database

Has anyone tried compiling grafter with scala.js? Does grafter use anything JVM-specific?

This is the first step towards a more full-featured query API.

It seems like the project is building for 2.12, so can we have a release as well? :)

Currently 1.6.0

When I do:

val application: Application =
  GenericReader[ApplicationConfig, Application].
    run(testConfiguration).
    replace[Database](mockDb)

This actually instantiate the default Database and then replace it by the mock.

This behaviour is exactly what we don't want when mocking because in my case it throw an exception as it checks the validity of some credentials.

At the moment we can memoize lists and maps but it's not likely to be a good idea.

Like:

case class ApplicationConfig[E <: Env]

I was thinking that Scala 2.12 support would be nice. I see that all the dependencies (after upgrading some) have 2.12-compatible releases.

We don't need to use labelled.Generic. Generic is enough

In a similar vein to the @readers annotation, it would be nice to provide a way to remove the fairly common genericReader boilerplate:

object Foo {
  implicit def reader: Reader[ApplicationConfig, Foo] = genericReader
}

Would another macro annotation like @genericReader be a suitable solution?

In order to avoid the confusion when starting a component implementing the Start interface:

• is start the extension method provided by org.zalando.grafter.syntax.rewriter._ to start all nested components?
• or is it the start method implemented by the component?

It would be nice to be able to display attributes when producing an application graph.

Typically we can show as attributes any case class member which is not a dependency itself. For examples "configuration" case classes members fall in that category:

@reader
case class HttpServer(config: HttpServerConfig)

case class HttpServerConfig(host: String, port: Int)

Then we should see host and port printed in the .dot graph for the HttpServer.

@shanethehat I just did a test on our project and replaced genericReader definitions for 8 components with @reader annotations. Result: +30s for our build time (126s vs 96s).

Do you have any idea on how to improve that?

Hi,
I'm trying to write a reader for ElasticClient (Elasticsearch) that we usually put in its companion object.
Unfortunately, it's a library and I can't add it in the companion object and then I get this could not find implicit value for parameter gen: shapeless.LabelledGeneric.Aux[A,Repr] for genericReader.

Do you know how can I make the right import of the implicit?
importing the implicit reader Reader[A, ElasticClient] in scope doesn't seems to work.

Here is a snipet example:

import cats.data.Reader
import org.zalando.grafter.GenericReader._

case class ElasticsearchConfig(host: String)

case class ApplicationConfig(elastic: ElasticsearchConfig)

object ApplicationConfig {

  implicit def elasticReader: Reader[ApplicationConfig, ElasticsearchConfig] =
    Reader(_.elastic)
}

case class ElasticClient(host: String)

object DefaultElasticClient {

  implicit def reader[A](implicit elasticConfigReader: Reader[A, ElasticsearchConfig]): Reader[A, ElasticClient] =
    elasticConfigReader.map(conf => ElasticClient(conf.host))
}

case class Application(elastic: ElasticClient)

object Application {
  import DefaultElasticClient._
  implicit def reader: Reader[ApplicationConfig, Application] = genericReader
}

The example doesn't compile because Reader[A, B] takes two type parameters.

The current grafter sources jar is empty in Maven central.

And just sometimes, with an error like this:

[info] VisualizeSpec
[info] 
[error]  x The example graph must be correctly serialized into .dot format
[error]  'strict digraph {
[error]    "A" [shape=box];
[error]    "B # 1/2" [shape=box];
[error]    "B # 2/2" [shape=box];
[error]    "C # 1/2" [shape=box];
[error]    "C # 2/2" [shape=box];
[error]    "D" [shape=box];
[error]    "B # 1/2" -> "A"
[error]    "B # 2/2" -> "A"
[error]    "C # 1/2" -> "A"
[error]    "C # 1/2" -> "B # 1/2"
[error]    "C # 1/2" -> "B # 2/2"
[error]    "C # 2/2" -> "A"
[error]    "C # 2/2" -> "B # 2/2"
[error]    "D" -> "C # 1/2"
[error]    "D" -> "C # 2/2"
[error]  }' != 'strict digraph {
[error]    "A" [shape=box];
[error]    "B # 1/2" [shape=box];
[error]    "B # 2/2" [shape=box];
[error]    "C # 1/2" [shape=box];
[error]    "C # 2/2" [shape=box];
[error]    "D" [shape=box];
[error]    "B # 1/2" -> "A"
[error]    "B # 2/2" -> "A"
[error]    "C # 1/2" -> "A"
[error]    "C # 1/2" -> "B # 1/2"
[error]    "C # 1/2" -> "B # 2/2"
[error]    "C # 2/2" -> "A"
[error]    "C # 2/2" -> "B # 1/2"
[error]    "C # 2/2" -> "B # 2/2"
[error]    "D" -> "C # 1/2"
[error]    "D" -> "C # 2/2"
[error]  }' (VisualizeSpec.scala:24)
[error] Actual:   ...aph {
[error] ... "A" [shape=box];
[error] ...1/2" [shape=box];
[error] ...2/2" [shape=box];
[error] ...1/2" [shape=box];
[error] ...2/2" [shape=box];
[error] ... "D" [shape=box];
[error] ...> "A"
[error] ...> "A"
[error] ...> "A"
[error] ... 1/2"
[error] ... 2/2"
[error] ...> "A"
[error] ..."B # [2]/2"
[error]   "[D]" ... "[C] # [1]/2"
[error] ..."C # [2]/2"
[error] [}]
[error] Expected: ...aph {
[error] ... "A" [shape=box];
[error] ...1/2" [shape=box];
[error] ...2/2" [shape=box];
[error] ...1/2" [shape=box];
[error] ...2/2" [shape=box];
[error] ... "D" [shape=box];
[error] ...> "A"
[error] ...> "A"
[error] ...> "A"
[error] ... 1/2"
[error] ... 2/2"
[error] ...> "A"
[error] ..."B # [1]/2"
[error]   "[C # 2/2]" ... "[B] # [2]/2"
[error] ..."C # [1]/2"
[error] [  "D" -> "C # 2/2"]

The error message:

[error] (run-main-0) java.lang.ClassCastException: scala.collection.immutable.Vector cannot be cast to scala.collection.immutable.Queue
java.lang.ClassCastException: scala.collection.immutable.Vector cannot be cast to scala.collection.immutable.Queue
	at scala.collection.immutable.Queue.$plus$plus(Queue.scala:111)
	at org.bitbucket.inkytonik.kiama.relation.Relation$.loop$1(Relation.scala:61)
	at org.bitbucket.inkytonik.kiama.relation.Relation$.fromOneStepGraph(Relation.scala:64)
	at org.bitbucket.inkytonik.kiama.relation.Relation$.fromOneStep(Relation.scala:73)
	at org.zalando.grafter.Visualize.asDotString(Visualize.scala:20)
	at org.zalando.grafter.Visualize.asDotString$(Visualize.scala:16)
	at org.zalando.grafter.Visualize$.asDotString(Visualize.scala:82)
	at org.zalando.grafter.VisualizeSyntax$VisualizeSyntaxOps.asDotString(Visualize.scala:111)
	at org.zalando.grafter.VisualizeSyntax$VisualizeSyntaxOps.asDotString(Visualize.scala:107)
	at com.example.grafterhttp4sexample.main.Visualize$.delayedEndpoint$com$example$grafterhttp4sexample$main$Visualize$1(Visualize.scala:13)
	at com.example.grafterhttp4sexample.main.Visualize$delayedInit$body.apply(Visualize.scala:5)
	at scala.Function0.apply$mcV$sp(Function0.scala:34)
	at scala.Function0.apply$mcV$sp$(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at com.example.grafterhttp4sexample.main.Visualize$.main(Visualize.scala:5)
	at com.example.grafterhttp4sexample.main.Visualize.main(Visualize.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)

To reproduce:

1. create a new application with my template, and be sure to set the scala version to 2.12.2 when prompted
2. run sbt run
3. choose Visualize

The current macro annotations are using Scalameta but they are showing an issue which might not be solved so soon.

The new scalamacros could solve this problem. We need to test on one annotation if:

• scalamacros has enough features already to support the grafter use case
• the issue in Scalameta is fixed

This should not fail

final case class T()
final case class Test(t1: T, t2: T)
List(Test(T(), T())).singletons(_ => true) must not(throwAn[Exception])

in the list of cats ecosystem projects.