finagle / finagle-postgres Goto Github PK
View Code? Open in Web Editor NEWPostgreSQL protocol support for Finagle
License: Apache License 2.0
PostgreSQL protocol support for Finagle
License: Apache License 2.0
Hello, I currently get the following error against PostgreSQL 9.4 while using finangle and finatra
java.lang.AbstractMethodError: com.twitter.finagle.postgres.codec.PgCodec$$anon$1.newClientDispatcher(Lcom/twitter/finagle/transport/Transport;Lcom/twitter/finagle/Stack$Params;)Lcom/twitter/finagle/Service;
at com.twitter.finagle.builder.CodecClient$Client$3.newDispatcher(ClientBuilder.scala:1121)
at com.twitter.finagle.client.StdStackClient$$anon$1$$anonfun$make$1$$anonfun$apply$1.apply(StackClient.scala:343)
at com.twitter.finagle.client.StdStackClient$$anon$1$$anonfun$make$1$$anonfun$apply$1.apply(StackClient.scala:343)
at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:947)
at com.twitter.util.Try$.apply(Try.scala:13)
at com.twitter.util.Future$.apply(Future.scala:102)
at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:947)
at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:947)
at com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:893)
at com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:892)
at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:100)
at com.twitter.util.Promise$Transformer.k(Promise.scala:100)
at com.twitter.util.Promise$Transformer.apply(Promise.scala:110)
at com.twitter.util.Promise$Transformer.apply(Promise.scala:91)
at com.twitter.util.Promise$$anon$3.run(Promise.scala:691)
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:187)
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158)
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:216)
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:84)
at com.twitter.util.Promise.continue(Promise.scala:689)
at com.twitter.util.Promise$Chained.continue(Promise.scala:206)
at com.twitter.util.Promise$Responder$class.transform(Promise.scala:175)
at com.twitter.util.Promise$Chained.transform(Promise.scala:182)
at com.twitter.util.Future.flatMap(Future.scala:892)
at com.twitter.util.Future.map(Future.scala:947)
at com.twitter.finagle.client.StdStackClient$$anon$1$$anonfun$make$1.apply(StackClient.scala:343)
at com.twitter.finagle.client.StdStackClient$$anon$1$$anonfun$make$1.apply(StackClient.scala:343)
at com.twitter.finagle.ServiceFactory$$anon$9.apply(Service.scala:206)
at com.twitter.finagle.postgres.codec.HandleErrorsProxy.apply(PgCodec.scala:80)
at com.twitter.finagle.postgres.codec.AuthenticationProxy.apply(PgCodec.scala:107)
at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
at com.twitter.finagle.builder.CodecClient$$anon$4$$anon$2.apply(ClientBuilder.scala:1071)
at com.twitter.finagle.service.FailFastFactory.apply(FailFastFactory.scala:183)
at com.twitter.finagle.pool.WatermarkPool.apply(WatermarkPool.scala:136)
at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
at com.twitter.finagle.service.FailureAccrualFactory.apply(FailureAccrualFactory.scala:140)
at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:239)
at com.twitter.finagle.loadbalancer.HeapBalancer.apply(HeapBalancer.scala:252)
at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:239)
at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
at com.twitter.finagle.factory.StatsFactoryWrapper.apply(StatsFactoryWrapper.scala:43)
at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
at com.twitter.finagle.factory.RefcountedFactory.apply(RefcountedFactory.scala:19)
at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
at com.twitter.finagle.factory.TimeoutFactory.apply(TimeoutFactory.scala:58)
at com.twitter.finagle.service.Requeues$$anon$2$$anon$1.com$twitter$finagle$service$Requeues$$anon$$anon$$applySelf(Requeues.scala:50)
at com.twitter.finagle.service.Requeues$$anon$2$$anon$1.apply(Requeues.scala:74)
at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:239)
at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
at com.twitter.finagle.factory.TimeoutFactory.apply(TimeoutFactory.scala:58)
at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:239)
at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:239)
at com.twitter.finagle.ServiceFactory.apply(Service.scala:149)
at com.twitter.finagle.FactoryToService.apply(Service.scala:313)
at com.twitter.finagle.postgres.Client.fire(Client.scala:127)
at com.twitter.finagle.postgres.Client.send(Client.scala:135)
at com.twitter.finagle.postgres.Client.sendQuery(Client.scala:75)
at com.twitter.finagle.postgres.Client.executeUpdate(Client.scala:43)
at de.envisia.hello.HelloWorldServer.warmup(HelloWorldServer.scala:22)
at com.twitter.inject.app.App$class.main(App.scala:51)
at de.envisia.hello.HelloWorldServer.com$twitter$inject$server$TwitterServer$$super$main(HelloWorldServer.scala:13)
at com.twitter.inject.server.TwitterServer$class.main(TwitterServer.scala:33)
at de.envisia.hello.HelloWorldServer.main(HelloWorldServer.scala:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:167)
at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:166)
at scala.Option.foreach(Option.scala:257)
at com.twitter.app.App$class.nonExitingMain(App.scala:166)
at de.envisia.hello.HelloWorldServer.nonExitingMain(HelloWorldServer.scala:13)
at com.twitter.app.App$class.main(App.scala:132)
at de.envisia.hello.HelloWorldServer.main(HelloWorldServer.scala:13)
at de.envisia.hello.HelloWorldServerMain.main(HelloWorldServer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sbt.Run.invokeMain(Run.scala:67)
at sbt.Run.run0(Run.scala:61)
at sbt.Run.sbt$Run$$execute$1(Run.scala:51)
at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:55)
at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
at sbt.Logger$$anon$4.apply(Logger.scala:85)
at sbt.TrapExit$App.run(TrapExit.scala:248)
at java.lang.Thread.run(Thread.java:745)
Exception thrown in main on startup
Exception: sbt.TrapExitSecurityException thrown from the UncaughtExceptionHandler in thread "run-main-0"
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 2 s, completed 24.05.2015 20:48:30
Process finished with exit code 1
The error appears to happen ON ALL query types, correct or incorrect ones.
Oh and Integration Tests are running fine, looks like some dependency problems.
But the only different thing is finangle 6.24 and 6.25, however that shouldn't be too incompatible.
Looks like the following pull request will fix the issue: #6
This happens with the following client config:
val dbClient = Postgres
.Client()
.withCredentials(user = postgresUser, password = postgresPassword)
.database(postgresDb)
.withSessionPool
.maxSize(1)
.withBinaryParams(true)
.withBinaryResults(true)
.newRichClient(postgresHost)
(If .withBinaryParams/.withBinaryResults is set to false the inserts work)
Database setup:
create table foo (n numeric(6,2));
Scala:
def insertBigDecimal(bd: BigDecimal) =
sql"""insert into foo (n) values ($bd)"""
.exec(dbClient)
.map(rc => s"inserted $rc rows")
.handle{ case _ => "failed to insert" }
val a = BigDecimal(10)
val b = BigDecimal(10.0)
val c = BigDecimal("1E+1")
val d = BigDecimal("10")
val fa = insertBigDecimal(a)
val fb = insertBigDecimal(b)
val fc = insertBigDecimal(c)
val fd = insertBigDecimal(d)
println(a, Await.result(fa)) // (10,inserted 1 rows)
println(b, Await.result(fb)) // (10.0,inserted 1 rows)
println(c, Await.result(fc)) // (1E+1,failed to insert)
println(d, Await.result(fd)) // (10,inserted 1 rows)
Are there any plans to create version for scala 2.10? I did not find any available jar in maven repositories.
in sum, max etc same behavior (((
log.txt
The ValuesSpec json parsing test fails intermittently on travis. Some class of values that can be generated by circe-json breaks things somewhere along the line. Example log attached.
Hey! I know that the administrative tasks of being an open-source maintainer can be a severe drain on one's personal energy level and that this ticket is probably going to induce at least a slight internal groan but... It would be great to get a new release. The issue in #81 is severe enough for my use-case that I've been running a build from the source of master in order to get the fix but it would be great to be able to use a proper upstream binary release. I do also think that pretty much anyone using finagle-postgres in production ought to get that fix because it can cause very difficult-to-diagnose query failures.
There is no state transition from AggregateRowsWithoutFields when a Delete() tag is received (though it exists for Update()). This makes DELETE...RETURNING queries impossible.
I meet a problem with finagle-postgres lib
finatra-http:18.10.0
finagle-core: 18.10.0
finagle-postgres:18.10.0
class P2PAnalysisServiceTest extends FeatureTest {
override val server = new EmbeddedHttpServer(new P2PService)
val domainKey = "1"
val appKey = "1"
val date = "20180103"
val start = "20180810"
val end = "20180817"
val hour = 14
test("P2PAnalysisService#perform feature") {
server.httpGet(
path = s"service/play/count/day/${domainKey}/${appKey}?date=${date}",
andExpect = Ok
)
}
}
system stack
An exception or error caused a run to abort: java.lang.NoSuchMethodError: com.twitter.finagle.client.Transporter$SocksProxy$.param()Lcom/twitter/finagle/Stack$Param;
java.lang.Exception: java.lang.NoSuchMethodError: com.twitter.finagle.client.Transporter$SocksProxy$.param()Lcom/twitter/finagle/Stack$Param;
at com.twitter.inject.server.EmbeddedTwitterServer.$anonfun$runNonExitingMain$1(EmbeddedTwitterServer.scala:434)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at com.twitter.util.Try$.apply(Try.scala:26)
at com.twitter.util.ExecutorServiceFuturePool$$anon$4.run(FuturePool.scala:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError: com.twitter.finagle.client.Transporter$SocksProxy$.param()Lcom/twitter/finagle/Stack$Param;
and this code is throw the stack
object PostgresModule extends TwitterModule with Logging {
val postgresConn: Flag[String] = flag("postgres",
"jdbc:postgresql://xxx:xxx/xxxx?stringtype=unspecified",
"the connection url string for postgres")
@Singleton
@Provides
def postgresClient: PostgresClient = {
val url = postgresConn().substring(5)
val uri = URI.create(url)
val host = uri.getHost
val port = uri.getPort
val client = Postgres.Client().withCredentials("postgres", Some("postgres"))
.database("xxx").withSessionPool.maxSize(5)
.withBinaryResults(true).withBinaryParams(true).newRichClient(s"$host:$port")
client
}
override def singletonShutdown(injector: Injector): Unit = {
super.singletonShutdown(injector)
debug("====singletonShutdown====")
injector.instance[PostgresClient].close()
}
override def singletonStartup(injector: Injector): Unit = {
debug("=====singletonStartup=====")
super.singletonStartup(injector)
}
}
@leonmaia @jeremyrsmith I had a great success with setting up autorelease for Finch-related projects (also finagle-oauth2). I'm happy to volunteer and do the same for finagle-postgress. The end goal is to allow anyone with push access to the repo perform releases with just changing (and committing) a new version in version.sbt
.
What do you think about this?
Hello, currently it is impossible to prepare a update query, this could lead to insecure code.
I think somehow it would be great to have a similar API than Anorm where you could "prepare" a query via SQL""
or SQL""""""
and use string interpolation directly.
Also it doesn't look to hard to have a feature like that.
However the main purpose should be an alternative to executeUpdate(str: String)
so that we can INSERT or UPDATE without strings, which are easily attackable to SQL injections.
Hello,
Testing a simple connection to a local postgresql instance with Scala 2.11, I got this exception:
Exception in thread "main" java.lang.AbstractMethodError: com.twitter.finagle.stats.MetricsStatsReceiver.com
stacktrace
[error] at com.twitter.finagle.stats.StatsReceiverWithCumulativeGauges$class.$init$(CumulativeGauge.scala:133)
[error] at com.twitter.finagle.stats.MetricsStatsReceiver.(MetricsStatsReceiver.scala:175)
[error] at com.twitter.finagle.stats.MetricsStatsReceiver.(MetricsStatsReceiver.scala:179)
[error] at com.twitter.finagle.stats.MetricsStatsReceiver.(MetricsStatsReceiver.scala:180)
[error] at com.twitter.finagle.stats.MetricsStatsReceiver.(MetricsStatsReceiver.scala:181)
[error] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[error] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error] at java.lang.Class.newInstance(Class.java:442)
[error] at com.twitter.app.LoadService$$anonfun$5.apply(LoadService.scala:74)
[error] at com.twitter.app.LoadService$$anonfun$5.apply(LoadService.scala:63)
[error] at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
[error] at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
[error] at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
[error] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
[error] at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
[error] at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
[error] at com.twitter.app.LoadService$.apply(LoadService.scala:63)
[error] at com.twitter.finagle.util.LoadService$.apply(LoadService.scala:14)
[error] at com.twitter.finagle.stats.LoadedStatsReceiver$.(LoadedStatsReceiver.scala:18)
[error] at com.twitter.finagle.stats.LoadedStatsReceiver$.(LoadedStatsReceiver.scala)
[error] at com.twitter.finagle.stats.ClientStatsReceiver$.(LoadedStatsReceiver.scala:47)
[error] at com.twitter.finagle.stats.ClientStatsReceiver$.(LoadedStatsReceiver.scala)
[error] at com.twitter.finagle.client.StackClient$.(StackClient.scala:389)
[error] at com.twitter.finagle.client.StackClient$.(StackClient.scala)
[error] at com.twitter.finagle.Postgres$.com$twitter$finagle$Postgres$$defaultStack(Postgres.scala:69)
[error] at com.twitter.finagle.Postgres$Client$.apply$default$1(Postgres.scala:96)
[error] at com.example.finatrabasics.controllers.testSuite$.main(DbController.scala:63)
[error] at com.example.finatrabasics.controllers.testSuite.main(DbController.scala)
It happens trying the example client connection you have on your quickstart page
val client = Postgres.Client()
.withCredentials("user", Some("pass"))
.database("mydb")
.newRichClient("localhost:5432")
Just creating this issue in case your team would like someone succesfully using this library with newer versions of Scala.
After TooManyWaitersException
FailureAccrual marked connection as dead for 5 seconds. I guess after 5 seconds client tries to retry and again gets error (ChannelException: Remote Info: Not Available
) and mark connection as dead for 6..13.. seconds. How could I catch whats happening and restore work? Config of client is default except noFailFast
With JDBC
I can pass the ApplicationName parameter as part of the connection string - jdbc:postgresql://localhost:5435/DBNAME?ApplicationName=MyApp
- and have "MyApp" show up in various statistics on the server. It would be useful to be able to do this on finagle-postgres
as well.
Finagle clients export their configurations into a global registry if they are running from within a TwitterServer. This means finagle-postgres' password will be exported as is and anyone can query it through a JSON/HTTP endpoint.
To fix this, we'd need to override the show
function for a Password
stack param.
Something along the lines:
case class Password(password: Option[String]) extends AnyVal
implicit object Password extends Stack.Param[Password] {
def default: Password = Password(None)
override def show(p: Password): Seq[(String, () => String)] = Nil
}
See this Finagle's commit for an inspiration.
select sum(active) as active from mobile_user where date = $date and domain_key = $domainKey and app_key = $appKey
if $domainKey or $appKey is null should I create another query like this ?
select sum(active) as active from mobile_user where date = $date
I think this idea is bed?can someboy help me ?
I am trying to retrieve a date field from a postgresql table but I couldn't get the value.
My code:
val ts = row.get[LocalDateTime]("ts")
If I change to:
val ts = row.get[String]("ts")
It had strange characters.
I see the date value in postgres table and the record was inserted with finagle-postgres library.
Would this project be interested in a newer PostgresClient
that's implemented in terms of the StackClient
, al lรก finagle-mysql? I ask first instead of a PR because implementing the new PostgresClient
would require either breaking changes on com.twitter.finagle.postgres.Client
, or duplicating behavior into a newer client and sunsetting / leaving the original com.twitter.finagle.postgres.Client
.
Thoughts?
Right now finagle-postgres
caches the DNS resolution forever when retrieving custom types from Postgres. That causes to the client to be forever dead even if the naming resolution succedes afterward, causing to Finagle stop complaining about Name resolution failed
but still causing failures when trying to query anything.
[info] 18:38:46.899 DEBUG [finagle/netty4-4] connection-1 - Sent frontend message of type: com.twitter.finagle.postgres.messages.Sync$
[info] 18:38:46.899 DEBUG [finagle/netty4-4] state machine-1 - Received event com.twitter.finagle.postgres.messages.Sync$ in state com.twitter.finagle.postgres.connection.Connected$
[info] 18:38:46.899 DEBUG [finagle/netty4-4] state machine-1 - Transitioning to state com.twitter.finagle.postgres.connection.Syncing$ and emiting result
[info] 18:38:46.900 DEBUG [finagle/netty3-2] connection-1 - Received backend message of type: com.twitter.finagle.postgres.messages.ReadyForQuery
[info] 18:38:46.900 DEBUG [finagle/netty3-2] state machine-1 - Received event com.twitter.finagle.postgres.messages.ReadyForQuery in state com.twitter.finagle.postgres.connection.Syncing$
[info] 18:38:46.900 DEBUG [finagle/netty3-2] state machine-1 - Transitioning to state com.twitter.finagle.postgres.connection.Connected$ and emiting result
[info] 18:38:46.900 DEBUG [finagle/netty3-2] connection-1 - Emitting result com.twitter.finagle.postgres.messages.ReadyForQueryResponse$
[info] 18:38:46.901 WARN [finagle/netty3-2] c.t.o.s.m.NoBrokersAvailableExceptionMapper - Name resolution failed with No hosts are available for db, Dtab.base=[], Dtab.local=[]. Aborting.
For now as a workaround we can create the client adding withDefaultTypes()
and that will make it work. But only if all the types needed by the application are described in: https://github.com/finagle/finagle-postgres/blob/master/src/main/scala/com/twitter/finagle/postgres/PostgresClient.scala#L84
You must call client.select() in order to get results out of an UPDATE...RETURNING query. But, a CommandComplete tag for Update(), in the absence of a row buffer, leads to an OK() response. Since the select() method treats this as an error condition, you must be very sure that there will be at least one row updated, or the UPDATE...RETURNING query will lead to a semantically meaningless exception being thrown.
select()'s behavior could be changed to silently drop the OK() message, since it is not typically indicative of an error. No information would be lost, because in the case where an OK() message was received by select()'s callback, no rows were updated so the parameter from the Update() tag is always zero. Instead, the OK(_) case could just map an empty row set:
case OK(_) => Seq.empty[Row].map(f)
I understand why the correlating error is thrown from exec(), because in that case rows were received and there is no mapper supplied to map them. That is more easily considered an error condition.
Reproduce:
for {
_ <- client.exec("CREATE TABLE update_returning_test (foo integer)")
r <- client.query("UPDATE update_returning_test SET foo = $1 returning foo", 5)(row => row.get[Int]("foo"))
} yield r
Seems like there was a breaking change from MD5 to SHA-256 in openssl. The most appropriate solution would be to recreate the encrypted secrets with this newer version.
I'm currently using finagle-postgres in a project.
I tried running my service, with PG running, do a request and get a response, turn off PG and do the same thing, then it fails fast with a PG Error. If I turn PG back on my service will not process the request, and the connection just hangs infinitely.
Here is a code example from my service
In app.scala
val pgHost = sys.env.get("PG_HOST").get
val pgUsername = sys.env.get("PG_USERNAME").get
val pgPassword = sys.env.get("PG_PASSWORD").getOrElse("")
val pgDatabase = sys.env.get("PG_DATABASE").get
val pgClient = Client(s"$pgHost", pgUsername, Some(pgPassword), pgDatabase)
val puppyService = new PuppyServiceImpl(pgClient)
val api = makeService(puppyService)
In one of my endpoints
def getHomeForPuppy(puppyService: PuppyService): Endpoint[PuppyWithHome] =
get("topics" ? param("country") ? header("Puppy-Id")) { (country: String, puppyId: String) =>
for {
puppyHome <- puppyService.getHome(puppyId)
} yield Ok(PuppyWithHome(puppyHome))
}
And finally this is my query
def getHome(puppyId: String): Future[Set[PuppyWithHome]] =
pgClient.prepareAndQuery("""
|select number, street, state from puppy_homes where
|puppy_id = $1
""".stripMargin, playerId) { row =>
PuppyWithHome(
row.get[Integer]("number"),
row.get[String]("street"),
row.get[String]("state")
)
} flatMap { rows =>
Future.value(rows.toSet)
} rescue {
case err => {
println(err.toString)
Future.exception(PGError("puppy_homes"))
}
}
I'm trying to get the simplest query - inserting a row and getting auto-generated primary - to work with no success. Not only it doesn't work, it marks connection as dead. Here's a code to reproduce (Scala 2.11, finagle-postgres 0.4.0)
object Hello {
def main(args: Array[String]): Unit = {
val client = Postgres.Client()
.withCredentials("pgsql", Some("pgsql"))
.database("app_stats_development")
.withSessionPool.maxSize(10)
.newRichClient("localhost:5432")
val create = client.execute("CREATE TABLE IF NOT EXISTS items (id SERIAL NOT NULL PRIMARY KEY, name VARCHAR(255))")
Await.result(create)
println("Table created")
val insert = client.select("INSERT INTO items (name) VALUES ('item_name') RETURNING id") { row =>
println(row.get[Int]("id"))
}
Await.result(insert)
println("After insert")
}
}
And here's log
Nov 26, 2016 4:00:04 AM com.twitter.finagle.Init$$anonfun$4 apply$mcV$sp
INFO: Finagle version 6.39.0 (rev=f49b26aa1a89cbaec5aa7ac956ae7f379b4a5d97) built at 20161011-165646
Table created
Nov 26, 2016 4:00:05 AM com.twitter.finagle.service.FailureAccrualFactory$$anon$2$$anon$1 didMarkDead
INFO: FailureAccrualFactory marking connection to "postgres" as dead. Remote Address: Inet(localhost/127.0.0.1:5432,Map())
When issuing a TRUNCATE TABLE command the response from the server is not properly parsed and an exception is thrown:
java.lang.IllegalStateException: Unknown command complete response tag TRUNCATE TABLE
at com.twitter.finagle.postgres.messages.BackendMessageParser.parseTag(BackendMessageParser.scala:245)
at com.twitter.finagle.postgres.messages.BackendMessageParser.parseC(BackendMessageParser.scala:218)
at com.twitter.finagle.postgres.messages.BackendMessageParser.parse(BackendMessageParser.scala:30)
Observed with version 0.10.0
It would be nice if your great library would support PostgreSQL arrays
Would be great to see this on Maven Central :)
When using row.get[Long]("id")
it returns null
. row.get[Int]("id")
works as expected though.
Note: I just noticed this behavior, I'll do some tests and update this Issue with my findings later.
Currently this client have limited Transaction support. It would be great if some could start a transaction and then commit it or roll it back, correctly.
I know there were a PR at #7 but somehow it got cancled.
Hi:
I meet a problem with use queryDSL spec
val resultFuture = sql"select count(*) as toal from tableA".as[Count].run(postgresClient)
the Count be defined like this below
case class Count(total:Int =0)
and then
val resultSeq = Await.result(resultFuture, Duration(10, TimeUnit.SECONDS))
val total = if (resultSeq.isEmpty) 0 else resultSeq.head.total
but not found in case class Count ,the total is always 0
We are running a PostgreSQL database with Infobright and very large data sets. We are running some rather involved queries against the data, which can return millions of rows, which we need to process as quickly as possible. Using the normal PostgreSQL JDBC driver is way too slow. I tried running our query through this Finagle 'driver' and the performance was even worse.
We get the best performance using the native libpg C library and cursors. It seems to me that we should be able to get pretty close to that in terms of performance from finagle-postgres... but I don't see the path towards that.
Is there "missing implementation" for cursors or streaming results from large queries? I would be willing to help in the implementation here if someone can point me in the right direction.
Hi, we're using finagle-postgres for our project, it's great, but we've faced an unpleasant bug.
For example we have table:
CREATE TABLE dummy (
id INT NOT NULL,
PRIMARY KEY (id)
);
And we want to insert several values in transactional manner:
it should "work" in {
val ids = Seq(1, 2)
val transaction = dbClient.inTransaction { client =>
Future.collect {
ids.map { id =>
client.prepareAndExecute("INSERT INTO dummy (id) VALUES ($1)", id)
}
}
}
Await.result(transaction)
}
But instead we're getting an error:
com.twitter.finagle.postgres.codec.ServerError: SQLSTATE 23505: duplicate key value (id)=(2) violates unique constraint "primary"
Enabling LoggingFilter on the client gives such logs:
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Query(SELECT * FROM pg_catalog.pg_tables WHERE tableowner = 'root')
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Query(BEGIN)
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Parse(,INSERT INTO dummy (id) VALUES ($1),ArraySeq(23))
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Parse(,INSERT INTO dummy (id) VALUES ($1),ArraySeq(23))
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: BIND[0000000131]
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: BIND[0000000132]
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Describe(true,)
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Describe(true,)
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Execute(,0)
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter logException
INFO: VERY BAD
com.twitter.finagle.postgres.codec.ServerError: SQLSTATE 23505: duplicate key value (id)=(2) violates unique constraint "primary"
So we can see that sending parameters is different, but somehow it tries to insert the same value twice.
It works fine if we use Future.traverseSequentially
instead of Future.collect
, but it's not what we want.
Same error pops when we use transactional parallel SELECT
requests with client.queryAs
.
We use CockroachDB but it behave the same on PostgreSQL. For testing we use "com.dimafeng:testcontainers-scala"
but same thing reproduces with instance in docker-compose.
Client config:
val client: PostgresClient = Postgres.Client()
.withCredentials(config.user, config.password)
.database(config.database)
.filtered(loggingFilter)
.newRichClient(config.host)
.withBinaryParams
doesn't help.
Hi, I am testing finagle-postgres with CockroachDb, and noticed all negative decimal values inserted into database are strange. After some test I narrowed it down to the binaryParameter.
Here's how I reproduced it :
root@:26257/it_test_db> show create table test;
+-------+-----------------------------------------------+
| Table | CreateTable |
+-------+-----------------------------------------------+
| test | CREATE TABLE test ( |
| | |
| | a INT NOT NULL, |
| | |
| | b DECIMAL NULL, |
| | |
| | CONSTRAINT "primary" PRIMARY KEY (a ASC), |
| | |
| | FAMILY "primary" (a, b) |
| | |
| | ) |
+-------+-----------------------------------------------+
then create a client withBinaryParams(true) and tried to insert some data:
val c =Postgres
.Client()
.withCredentials("root", None)
.database("it_test_db")
.withBinaryParams(true)
.newRichClient("localhost:26257")
cala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 1, 1.1))
res0: Int = 1
scala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 2, -1.1))
res1: Int = 1
scala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 3, BigDecimal(1.1)))
res2: Int = 1
scala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 4, BigDecimal(-1.1)))
res3: Int = 1
scala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 5, BigDecimal(1)))
res4: Int = 1
scala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 6, BigDecimal(-1)))
res5: Int = 1
but in the database it became :
root@:26257/it_test_db> select * from test;
+---+------------------------+
| a | b |
+---+------------------------+
| 1 | 1.1 |
| 2 | -1.1 |
| 3 | 1.1 |
| 4 | -1844674407370955161.5 |
| 5 | 100000 |
| 6 | -0 |
+---+------------------------+
however if client is created without withBinaryParams(true)
scala> val c2 =Postgres.Client().withCredentials("root", None).database("it_test_db").newRichClient("localhost:26257")
c2: com.twitter.finagle.postgres.PostgresClientImpl = com.twitter.finagle.postgres.PostgresClientImpl@45682e3e
scala> com.twitter.util.Await.result(c2.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 7, BigDecimal(1.1)))
res6: Int = 1
scala> com.twitter.util.Await.result(c2.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 8, BigDecimal(-1.1)))
res7: Int = 1
scala> com.twitter.util.Await.result(c2.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 9, BigDecimal(1)))
res8: Int = 1
scala> com.twitter.util.Await.result(c2.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 10, BigDecimal(-1)))
res9: Int = 1
in the database it seems ok:
root@:26257/it_test_db> select * from test;
+----+------------------------+
| a | b |
+----+------------------------+
| 1 | 1.1 |
| 2 | -1.1 |
| 3 | 1.1 |
| 4 | -1844674407370955161.5 |
| 5 | 100000 |
| 6 | -0 |
| 7 | 1.1 |
| 8 | -1.1 |
| 9 | 1 |
| 10 | -1 |
+----+------------------------+
(10 rows)
Hi
I am trying to connect to a Postgres hosted by Heroku.
https://devcenter.heroku.com/articles/heroku-postgresql#heroku-postgres-ssl
import com.twitter.finagle.Postgres
val client = Postgres.Client()
.withCredentials("xx", Some("xx"))
.database("xx")
.withSessionPool.maxSize(1)
.newRichClient("xx.eu-west-1.compute.amazonaws.com:5432")
returns an error saying ssl is off as expected
com.twitter.finagle.postgres.codec.ServerError: no pg_hba.conf entry for host "82.2.102.2", user "xx", database "xx", SSL off
I am unable to set sslmode=require on the underlying connection. I have tried
val client = Postgres.Client()
.withCredentials("xx", Some("xx"))
.database("xx")
.withSessionPool.maxSize(1)
.conditionally(true, _.withTransport.tlsWithoutValidation)
.withTransport.tls("amazonaws.com")
.newRichClient("xx.eu-west-1.compute.amazonaws.com:5432")
val client = Postgres.Client()
.withCredentials("xx", Some("xx"))
.database("xx")
.withSessionPool.maxSize(1)
.withTransport.tls("amazonaws.com")
.newRichClient("xx.eu-west-1.compute.amazonaws.com:5432")
val client = Postgres.Client()
.withCredentials("xx", Some("xx"))
.database("xx")
.withSessionPool.maxSize(1)
.conditionally(true, _.withTransport.tlsWithoutValidation)
.newRichClient("xx.eu-west-1.compute.amazonaws.com:5432")
The README has
libraryDependencies ++= Seq(
"com.github.finagle" %% "finagle-postgres" % "0.2.0"
)
But that's the wrong groupId for version 0.2.0 on Maven Central, which, unlike 0.1.0, has the groupId "io.github.finagle." See http://search.maven.org/#search%7Cga%7C1%7Cfinagle-postgres
(Btw package namespaces starting with io.
shadow scala.io => io
imports from Predef, which is kind of annoying, but perhaps that's a lost cause, because netty.)
The Finagle library is finishing up its transition to Netty4, and as such it's going to be dropping Netty3 support in its entirety at some point. However, it looks like finagle-postgres
is heavily dependent on the finagle-netty3
package.
There are at least two roads forward.
finagle-postgres
can migrate to finagle-netty4.finagle-netty3
package could be moved to the finagle org.These are not mutually exclusive, but either will address the issue at hand. Do you have a preference? There may be enough ecosystem projects to justify the second route regardless but we haven't don't an extensive survey.
Twitter has relatively recently begun work on an internal project that will be using the PostgreSQL wire protocol, and are hoping to use finagle-postgres for that since it's a great project. Obviously, @vkostyukov (and more recently @steveniemitz and I suspect others) from Twitter have been contributing to the project, but we're interested in upping our involvement a bit.
The first question, mostly for @leonmaia but really any interested contributor or user of the project, is whether you're open to us stepping in to help with the maintenance of the project more. And then related to that, we wanted to propose the idea of promoting finagle-postgres to being a core part of the finagle project alongside finagle-mysql, finagle-redis, etc. That would allow the CSL team inside of Twitter to more easily maintain it with the rest of Finagle to keep it in sync with any changes, etc.
Anyway, we really like the project and are excited about using it, and would love to help however we can. ๐
The last few releases don't have associated git tags. This makes it a lot harder to keep track of what releases there are and what code is in each one.
Postgres supports syntax like this: ... WHERE (a, b) in ((1, 2), (3, 4))
It would be nice to pass Seq of Tuple (or even a case class) as a Param in the query, currently it seems to only work with String.
A test for something like this (for sql""
syntax) might look something like this:
"Seq of tuple params for IN operator" in {
val p1 = Seq(("foo", "bar"), ("three", "four"))
expectQuery("SELECT * FROM foo WHERE (a, b) IN (($1, $2), ($3, $4))", "foo", "bar", "three", "four") {
sql"SELECT * FROM foo WHERE a IN ($p1)"
}
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.