kwrooijen / gungnir Goto Github PK
View Code? Open in Web Editor NEWA fully featured, data-driven database library for Clojure.
Home Page: https://kwrooijen.github.io/gungnir/
License: MIT License
A fully featured, data-driven database library for Clojure.
Home Page: https://kwrooijen.github.io/gungnir/
License: MIT License
Currently the model name will be mapped to the table name. The resulting model (before it is stored using gungnir.model/register!
) has a property :table
which contains the table name. This name is extracted from the key that you register it with.
It could be desirable to have a model name not be the same as a table name. Or possibly even have multiple models for a single table?
It would be nice to check the model structure during registration. Also things like relations (:has-one
, :has-many
:belongs-to
). And fail with a readable error. Currently if you make a typo in the relations, for example, the system will fail when you try to make a query. This should be caught as soon as possible. We can use Malli for this.
Currently Gungnir assumes all primary keys are generated through postgres. There should be an option to create it ourselves (in the case of save!
Gungnir update, instead of insert). Maybe expose insert / update separately?
[:map
{:has-many {:app/clients {:model :client}}}
[:app/id {:primary-key true} uuid?]
[:app/name [:string {:min 1}]]]
[:map
{:belongs-to {:client/app-id {:model :app}}}
[:client/id {:primary-key true} uuid?]
[:client/app-id uuid?]]
(-> (q/select :app/*)
(q/all! :app))
;; Should return
;; [{:app/id ",,," :app/name ",,," :app/clients <relation-atom>}]
;; Instead it returns:
;; [{:app/id ",,," :app/name ",,,"}]
Currently the model namespace keeps track of registered models and creates maps for "quick" mapping (e.g. mapping a model field to table column). This is mainly for performance benefits.
gungnir/src/cljc/gungnir/model.cljc
Lines 12 to 17 in 54fa3b8
These maps are stored in atoms. However it might be nice to actually store then in ^:dynamic
defs and change them with alter-var!
. This would remove the deref overhead and possibly improve performance slightly. I don't think there's much risk in changing these to defs since they are only set during model registration. Which generally only happens once at system startup.
The goal is to separate out the actual data specs, so I can re-use them in places where gungnir isn't involved. The attached example works, until you enable instrumentation.. then there is a spec violation. But it does seem to work as intended with spec disabled. You can also trigger a spec error from gungnir earlier by replacing the (m/spec User)
with User
on line 13. gungnir doesn't seem to like the referenced schema.
(def UserRegistry {:users/id uuid?
:users/email [:re {:error/message "Invalid email"} #".+@.+\..+"]
:users/created-at inst?
:users/updated-at inst?})
(def User [:map {:registry UserRegistry}
[:users/id {:auto true :primary-key true}]
:users/email
[:users/created-at {:auto true}]
[:users/updated-at {:auto true}]])
(model/register!
{:users (m/schema User)}) ;; <-- remove m/schema wrap to see spec error earlier
(def data {:users/name "Tester"
:users/email "[email protected]"
:users/id (java.util.UUID/randomUUID)
:users/created-at (j/instant)
:users/updated-at (j/instant)})
(m/validate User data) ; => true
(q/save! (changeset data)) ;; this blows up ONLY when instrumentation is enabled, otherwise it works as intended
;; the spec error
Execution error - invalid arguments to gungnir.util.malli/child-properties at (field.cljc:33).
field.cljc:33
-- Spec failed --------------------
Function arguments
(nil)
^^^
should satisfy
schema?
or
should have additional elements. The next element ":key" should satisfy
qualified-keyword?
-- Relevant specs -------
:gungnir.model/field:
(clojure.spec.alpha/or
:schema
malli.core/schema?
:gungnir.model.field/without-props
(clojure.spec.alpha/cat
:key
clojure.core/qualified-keyword?
:spec
clojure.core/any?)
:gungnir.model.field/with-props
(clojure.spec.alpha/cat
:key
clojure.core/qualified-keyword?
:props
(clojure.spec.alpha/nilable clojure.core/map?)
:spec
clojure.core/any?))
-------------------------
Detected 1 error
Hello Kevin,
Thank you for the great Gungnir presentation in the Dutch Clojure meetup.
The project is really great and inspiring.
Can you please advise if you are looking for any help or support for this project implementation (something small to start with) ?
Regards, Eugene
When creating a model with dashes in its name casting won't work. Underscores do work however. Probably related to the change regarding the :table
key.
While implementing some next.jdbc
support for postgres types (json, bytea, etc), I ran into a case where gungnir ate an exception it couldn't handle.
Gungnir is eating all Exception
s yet, the exception->map
multimethod can only handle exceptions of type SQLException
(see stack trace below).
I believe the solution to this is to catch only SQLException
s and let others bubble up. Thoughts @kwrooijen ?
gungnir.query/save! query.clj: 104
gungnir.query/save! query.clj: 108
gungnir.database/insert! database.clj: 259
gungnir.database/execute-one! database.clj: 187
gungnir.database/execute-one! database.clj: 195
...
gungnir.database/eval47644/fn database.clj: 158
java.lang.ClassCastException: class java.lang.IllegalArgumentException cannot be cast to class java.sql.SQLException (java.lang.IllegalArgumentException is in module java.base of loader 'bootstrap'; java.sql.SQLException is in module java.sql of loader 'platform')
Currently Gungnir only supports a single global datasource. We should support having multiple instances.
From Slack:
And the second thing is related to that use of a global database var: In many systems it is desirable to connect to multiple different databases, but this won't be possible with gungnir.
I think it makes lots of sense to do away with the global var, and instead return a state-map that contains the functions needed.
I'm thinking of how the sente library works here: it's equivalent function to make-datasource! returns a map of state and functions for operating on that state. (edited)
For example make-datasource! could return a map that contains the keys:
:stop-fn - a function to close the connnection pool
:changeset-fn - the changeset function bound to this particular db connection
:database - the *database* for this particular connection
:find!-fn - the query/find! function for this particular connection
... and so forth
some thought would have to be given for how gungnir.model/register! would be bound to a particular gungnir instance
For your second point, yes Gungnir only supports a single database connection at the moment. I'm still thinking of a way to handle this. What I would really like is for multiple instances to be optional, whereas a single global instance is the default. I don't often have multiple datasources but I understand that this should be supported in some way.
One way would be to map datasources to identifiers. e.g. (gungnir/make-datasource! ::datasource-1 {,,,}) . Then when you register a model you can register it with a specific datasource
(gungnir.model/register!
{:user
[:map
{:datasources [::datasource-1]}
[,,,]]})
Then, whenever you query the database, you can pass in the datasource or key
Casey 2:51 PM
If all the functions supported an arity with a datasource param, that would probably work too
kwrooijen 2:52 PM
Yeah, I think that would be difficult with the current API though. the find! find-by! and all! functions are quite flexible but also not my most elegant pieces of code. Worst case we could create a different variant of these functions which support accepting a datasource (edited)
e.g. find-with-datasource! find-by-with-datasource! all-with-datasource! or something along those lines
A library like this isn't complete without some support for transactions.
One first step, that might also be sufficient, is to expose a with-transaction
macro that wraps the next.jdbc's with-transaction macro.
Reading:
It would be nice to have some sort of data-driven migration system built in. This doesn't mean that we have to build everything from scratch. We could also decide to use a migration library such as Ragtime and expand on that.
I'm not sure what we exactly want with this. But I think it would be nice to have a syntax for basic migration action such as
And if you need to do something very specific, grant the ability to fall back to raw SQL.
Collection of things that still need to be documented.
query.md
I would like to contribute and maintain this project, is it active?
To help track what postgres errors would be good to add default exception handlers for I've created this ticket.
All postgres errors are here: https://www.postgresql.org/docs/current/errcodes-appendix.html
42703 | undefined_column
- likely to encounter this when your model field name does not match the table column... more soon ...
It would be nice if there was some sort of Seeding mechanism for creating test data. Malli already provides great support for data generation so we could easily use that. What Gungnir could do is automatically insert it in the database.
Currently in order to load a relation, you need to deref the relation atom of a record. This results in a new query being executed. It would be nice to be able to preload relations, querying them with a join
instead of separate queries.
A solution could be to add a new function gungnir.query/preload
, which will add a :gungnir/preload '(,,,)
key to the HoneySQL map. We then need to modify the query before executing it.
(-> (q/preload :user/posts)
(q/find-by! :user/email "[email protected]"))
Another thing we need to think about is how do we adjust the result of the relation? Normally we swap!
the relation atom, but in this case we'd have to do that beforehand. Maybe a function to update preloads?
(-> (q/preload :user/posts)
(q/preload-update :user/posts q/limit 5)
(q/preload-update :user/posts q/sort-by :post/score :desc)
(q/find-by! :user/email "[email protected]"))
Currently using schemas doesn't work. e.g. :table "foo.bar"
will break the qualifier-builder
function (among other things, possibly).
Currently relations are linked directly. That doesn't work well for the following use case:
user
---
id : uuid
user_organization
---
id : uuid
user_id : foreign key user(id)
organization_id : foreign key organization(id)
organization
---
id : uuid
In this case, users are linked to organizations through the user_organization
table. We need to somehow resolve a user
/ organization
relations through user_organization
.
Possibly something along the lines of:
;; User model
[:map
{:has-many {:organization {:field :user/organizations, :through :user_organization/organization_id}}
,,,
}]
;; Organizatin model
[:map
{:has-many {:user {:field :organization/users, :through :user_organization/user_id}}
,,,
}]
Consider the case when you have a user
model and a document
model.
Each document has an author-id
which is a foreign key to the user table, and a reviewer
which is also a foreign key to the user table. This results in multiple belong-to relations referencing the same table (user
).
As discussed in slack:
@kwrooijen suggested a format like this:
{:belongs-to {:document/author {:model :user :through :document/author-id}}}
{:has-many {:document/authors {:model :user :through :user/document-id}}}
{:has-one {:document/authors {:model :user :through :user/document-id}}}
So, the example given at the top would look like:
(model/register!
{:user
[:map
[:user/id {:primary-key true :auto true} int?]]})
(model/register!
{:document
[:map {:belongs-to {:document/author {:model :user :through :document/author-id}
:document/reviewer {:model :user :through :document/reviewer-id}}}
[:document/id {:primary-key true :auto true} int?]
[:document/author-id int?]
[:document/reviewer-id {:optional true} int?]
]})
When an exception is caught, gungnir is printing the honeysql form. Ref this line
gungnir/src/clj/gungnir/database.clj
Line 193 in 3d29904
This is bad for a few reasons:
#2 is actually how I discovered this, an exception during a big insert was locking up my emacs repl.
The best solution is probably to use clojure.tools.logging and use the "debug" handler. That way clients can enable/disable gungnir logs as they see fit.
(-> (q/select :%count.*)
(q/from :app)
(q/all!))
;; ERROR: {:type :malli.core/invalid-schema, :data {:schema nil}}
The reason this breaks is because the default row builder expects rows to be returned from a query. In our selection we just want to count the amount of rows that have matched, returning an int. How can we solve this?
One option would be to add "raw" query functions. e.g. q/all!!
which skips the rows builder. It might be nice to skip rows from being build if they are not part of the model. That might be challenging to do but it could be a good solution.
Gungnir is currently a single library (and gungnir.ui
), but it could be separated into three parts.
All the cljc
files are regarding the model and changeset code. This code is generic and can be used for anything. No SQL / Database code is here.
The second part is generation of SQL. This partly done by the query
namespace, and partly done by the database
namespace. This should all be moved to the query
namespace, and the namespace should be renamed to gungnir.sql.honeysql
. This would allows us to support other querying languages in Clojure.
The last part is the database / pool. As noted about, the honeysql code should be removed from here and placed in gungnir.sql.honeysql
. We use jdbc.next
, and rely heavily on it, so this should not be configurable. The database pool on the other hand should be. So the creation of the HikariCP datapool should be extracted from here and placed in gungnir.database.hikaricp
.
This would allow us to initialize gungnir in a configurable manner:
(gungnir/initialize!
{:gungnir/sql :gungnir.sql/honeysql
:gungnir/database :gungnir.database/hikaricp})
We'd probably also want to have honeysql / hikaricp as default. That if you call gungnir/initialize!
without any arguments, it will default to those two libraries. This is to ensure the "Plug & Play" feel.
A way to close the underlying connection pool would be handy when using state management systems like mount or component.
Adding gungnir.database/close-datasource!
would suffice here.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.