mikeball / foundation Goto Github PK
View Code? Open in Web Editor NEWA clojure data access library for postgresql.
License: Eclipse Public License 1.0
A clojure data access library for postgresql.
License: Eclipse Public License 1.0
Placeholders in query files cause problems, and should be allowed.
Because they are so common.
When a database column allows nulls, inserts/updates can not set the value to null because it uses the value of the parameter to determine the type. JDBC requires this type. If the value is null, the sql type can't be inferred.
For logging it'd be helpful if I could print out all the sequel templates (perhaps after YesSQL interpolation) so AWS Cloudwatch can pick up on them.
When I try to execute a query that creates a database error (e.g. causing Postgres to print its own STDERR in its logs), it'd be nice to see the exception returned to be in the clj code instead of just nil
(e.g. null constraints, foreign keys, etc.). If you like this idea, where should I look to implement it?
Hi, the following code (the same as in doc) will not work:
(defn create-products-table []
(with-open [cnx (.getConnection db)]
(execute cnx "CREATE TABLE products (id serial primary key not null, name varchar(255), description text);")))
I get the error
CompilerException java.lang.RuntimeException: Unable to resolve symbol: execute in this context, compiling:(project/models/product.clj:14:3)
I fixed this by:
(:require [taoclj.foundation :as pg]
[taoclj.foundation.execution :as execution]))
...
(execution/execute cnx "CREATE TABLE products (id serial primary key not null, name varchar(255), description text);")))
Am I going wrong or I miss something in docs?
When creating a def-query, you should be able to specify a file located in the resources directory of a project. This may already be functional but need to confirm and if not add this feature.
I haven't quite wrapped my head around wrapping foundation database transactions with one of my own.
I'm trying to create a test fixture that looks like this:
(ns vsa-sql-management.test-helpers
(:require [clojure.java.jdbc :as jdbc]
[taoclj.foundation :as pg]
[environ.core :refer [env]]))
(declare ^:dynamic *store*)
(def test-db-spec
{:classname "org.postgresql.Driver"
:subprotocol "postgresql"
:subname "//127.0.0.1:5432/vsa_test"
:user "postgres"})
(defn db-transaction-fixture [f]
(jdbc/with-db-transaction [store test-db-spec]
(jdbc/db-set-rollback-only! store)
(binding [*store* store] (f))))
Ideally, I can be passing in the dynamic store variable into a pg/trx->
call so my tests know that the transaction should be rolled back. However, when I write a test like this:
(ns vsa-sql-management.seeders.vsa-questionnaire-template-test
(:require [vsa-sql-management.seeders.vsa-questionnaire-template :as sut]
[vsa-sql-management.test-helpers :refer [*store* db-transaction-fixture]]
[taoclj.foundation :as pg]
[clojure.test :as t]
[environ.core :refer [env]]))
(t/use-fixtures :each db-transaction-fixture)
(t/deftest seed-test
(t/testing "seeds database with questions"
(pg/trx-> sut/vsa-db (pg/insert :people {:name "nick"}))
(t/is (= 1 (pg/trx-> sut/vsa-db (pg/select :people {:name "nick"}))))))
I still see the created person record in my test database. I'm sure there's something simple that my stupid self is missing. Any help would be appreciated! I think if I figure out the proper place to pass the *store*
variable to, I'll be all good.
I was able to use an insert query from a template file to write to my database, but when I tried a select query, I got this result:
=> NullPointerException clojure.lang.Reflector.invokeNoArgInstanceMember (Reflector.java:301)
I tried this query:
(pg/qry-> conn (pg/select :users {}))
That query should simply evaluate to `SELECT * FROM users;`, right? I did get the same result when I used a template query:
```sql
SELECT * FROM users LIMIT :limit OFFSET :offset;
(pg/qry-> conn (get-users {:offset 0 :limit 30}))
Any idea what I'm doing wrong? For completeness, here's my DB spec:
; it actually comes from a profiles.clj thing, but these are the values:
(def datasource
{:host "localhost"
:port 5432
:database "<known-good value>"
:username "<known-good value>"
:password "<<known-good value>"
:pooled false}}
And I'm just doing:
(pg/def-datasource conn datasource)
Something in Foundation's database code is emitting debug statements to the REPL.
Run a Foundation query in the REPL.
Query runs silently if there are no errors.
On first query:
(pg/qry-> conn (get-users {:limit 10 :offset 0}))
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/amacdougall/.m2/repository/ch/qos/logback/logb
ack-classic/1.1.2/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/amacdougall/.m2/repository/org/slf4j/slf4j-sim
ple/1.7.13/slf4j-simple-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
16:05:48.528 [nREPL-worker-1] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
16:05:48.534 [nREPL-worker-1] DEBUG i.n.c.MultithreadEventLoopGroup - -Dio.netty.event
LoopThreads: 8
16:05:48.548 [nREPL-worker-1] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bu
ffer.address: available
16:05:48.548 [nREPL-worker-1] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Un
safe.theUnsafe: available
<much more output omitted>
On subsequent queries, debug statements related to thread pooling. My DB config has pooling: false
, so this is probably just related to standard JDBC.
16:08:55.584 [threadDeathWatcher-2-3] DEBUG io.netty.buffer.PoolThreadCache - Freed 8 thread-local buffer(s) from thread: PG-JDBC EventLoop (1)
Debug logging levels should be configurable, and have a high threshold (log only on warning or errors, perhaps) by default.
When I have a datasource:
(pg/def-datasource lw-db
{:host "localhost"
:port 5432
:database "andy"
:username "andy"
:pooled true})
and I try to create a jar file with lein uberjar, I get a null pointer exception during the compile:
$ lein uberjar
Compiling lwchallenge.core
WARNING: update already refers to: #'clojure.core/update in namespace: taoclj.foundation, being replaced by: #'taoclj.foundation/update
java.lang.NullPointerException, compiling:(core.clj:24:1)
Exception in thread "main" java.lang.NullPointerException, compiling:(core.clj:24:1)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3657)
at clojure.lang.Compiler$IfExpr.eval(Compiler.java:2697)
.....
Caused by: java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:459)
at com.zaxxer.hikari.HikariConfig.addDataSourceProperty(HikariConfig.java:294)
at taoclj.foundation.datasources$create_pooled_datasource.invokeStatic(datasources.clj:27)
at taoclj.foundation.datasources$create_pooled_datasource.invoke(datasources.clj:26)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3652)
... 36 more
Something going wrong at com.zaxxer.HikariConfig.
Lein is set up to use clojure 1.8.0 but the same problem occurs with clojure 1.7.0.
(defproject lwchallenge "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.8.0"]
[org.clojure/data.xml "0.0.8"]
[org.taoclj/foundation "0.1.3"]
[clj-time "0.11.0"]
[org.slf4j/slf4j-log4j12 "1.7.1"]] ; <- I seem to need this when :pooled is true as well
:main ^:skip-aot lwchallenge.core
:target-path "target/%s"
:profiles {:uberjar {:aot :all}})
Any idea what might be going on? I've not looked at Hikarai before.
Hello,
Nice work. Have you considered a core.async
interface for async connections to psql?
(If it is in fact impossible to use transactions, feel free to update the issue title, but I didn't want to assume too much.)
The JDBC functions clojure.java.jdbc/with-db-transaction
and clojure.java.jdbc/db-set-rollback-only!
expect to operate on a "database connection" object compatible with the get-connection function. It appears that most people pass in the db spec they used to set up their normal connection, whether it's a DB URI, a hash with connection parameters, or whatever.
However, Foundation expects to use a javax.sql.DataSource
-compatible object for every query. Since one of the allowed get-connection
arguments is a {:datasource <ds>, ...}
object, I tried doing this:
(clojure.java.jdbc/with-db-transaction [t-conn {:datasource db/conn}]
(jdbc/db-set-rollback-only! t-conn)
(pg/qry-> (:datasource t-conn) (db/alter-some-stuff! params)))
Where db/conn
is the result of a def-datasource
call. Although this syntax does alter the database, those changes persist.
This is not surprising: with-db-transaction
is supposed to bind t-conn
to a connection object. In this case, it binds a hash:
{:datasource
#object[com.impossibl.postgres.jdbc.PGDataSource 0x5f9231ae "com.impossibl.postgres.jdbc.PGDataSource@5f9231ae"],
:connection
#object[com.impossibl.postgres.jdbc.PGConnectionImpl 0x129eb816 "com.impossibl.postgres.jdbc.PGConnectionImpl@129eb816"],
:level 1,
:rollback #<Atom@79e53c70: true>}
My theory is that the :connection
part of that hash is the transaction connection, and if I were to operate on that, the changes would be rolled back. But Foundation expects to work on a PGDataSource, and it gets a new connection from it, one which does not have the transaction or rollback properties.
Am I right? And if so, is there a workaround for this situation, or just a better way of achieving my goal? I think you might have some unwelcome tension between Foundation's way of doing things and the bare-bones JDBC functions people may wish to apply.
I'm having an issue with the following code creating a Person record, even though the User record in the same transaction was not created:
(def post-confirmation-event
"mock event from AWS Cognito"
{"version" 1
"region" "us-west-2"
"userPoolId" "us-west-2_wiZEwc83y"
"userName" "nick"
"callerContext" {"awsSdkVersion" "aws-sdk-js-2.6.4"
"clientId" "3birc4sh4vmaoeudj50cd0vg8kd"}
"triggerSource" "PostConfirmation_Confirmation"
"request" {"userAttributes" {"sub" "0fcf454386-8cf6-40d6-b042-3e7b61a67f33"
"email_verified" "true"
"cognito:user_status" "what"
"phone_number_verified" true
"phone_number" "+123456789"
"email" "[email protected]"}}
"response" {}})
(defn add-user-to-database [user]
"adds the Cognito user to our database"
(pg/trx-> db/vsa-db
(pg/insert :people {:name (get user "userName")})
(pg/insert
:users
(pg/with-rs
[]
{:cognito-uuid (get-in user ["request" "userAttributes" "sub"])
:cognito-pool (get user "userPoolId")
:email (get-in user ["request" "userAttributes" "email"])
:username (get user "userName")
:person-id (first rs)}))))
(add-user-to-database post-confirmation-event)
In the REPL it returns the primary key for the Person record, but an empty collection for the user record. Since I'm only creating one User record, is there another way to do this? I guess there's always SQL templating
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.