The goal is to separate out the actual data specs, so I can re-use them in places where gungnir isn't involved. The attached example works, until you enable instrumentation.. then there is a spec violation. But it does seem to work as intended with spec disabled. You can also trigger a spec error from gungnir earlier by replacing the (m/spec User) with User on line 13. gungnir doesn't seem to like the referenced schema.

(def UserRegistry {:users/id uuid?
                   :users/email [:re {:error/message "Invalid email"} #".+@.+\..+"]
                   :users/created-at  inst?
                   :users/updated-at  inst?})

(def User [:map {:registry UserRegistry}
           [:users/id {:auto true :primary-key true}]
           :users/email
           [:users/created-at {:auto true}]
           [:users/updated-at {:auto true}]])

(model/register!
  {:users (m/schema User)}) ;; <-- remove m/schema wrap to see spec error earlier

(def data {:users/name       "Tester"
           :users/email      "[email protected]"
           :users/id         (java.util.UUID/randomUUID)
           :users/created-at (j/instant)
           :users/updated-at (j/instant)})
(m/validate User data)                  ; => true

(q/save! (changeset data)) ;; this blows up ONLY when instrumentation is enabled, otherwise it works as intended

;; the spec error
Execution error - invalid arguments to gungnir.util.malli/child-properties at (field.cljc:33).
field.cljc:33

-- Spec failed --------------------

Function arguments

  (nil)
   ^^^

should satisfy

  schema?

or

should have additional elements. The next element ":key" should satisfy

  qualified-keyword?

-- Relevant specs -------

:gungnir.model/field:
  (clojure.spec.alpha/or
   :schema
   malli.core/schema?
   :gungnir.model.field/without-props
   (clojure.spec.alpha/cat
    :key
    clojure.core/qualified-keyword?
    :spec
    clojure.core/any?)
   :gungnir.model.field/with-props
   (clojure.spec.alpha/cat
    :key
    clojure.core/qualified-keyword?
    :props
    (clojure.spec.alpha/nilable clojure.core/map?)
    :spec
    clojure.core/any?))

-------------------------
Detected 1 error

It would be nice to check the model structure during registration. Also things like relations (:has-one, :has-many :belongs-to). And fail with a readable error. Currently if you make a typo in the relations, for example, the system will fail when you try to make a query. This should be caught as soon as possible. We can use Malli for this.

A library like this isn't complete without some support for transactions.

One first step, that might also be sufficient, is to expose a with-transaction macro that wraps the next.jdbc's with-transaction macro.

Reading:

• https://hexdocs.pm/ecto/Ecto.Multi.html
• https://curiosum.dev/blog/elixir-ecto-database-transactions

When creating a model with dashes in its name casting won't work. Underscores do work however. Probably related to the change regarding the :table key.

Currently the model namespace keeps track of registered models and creates maps for "quick" mapping (e.g. mapping a model field to table column). This is mainly for performance benefits.

These maps are stored in atoms. However it might be nice to actually store then in ^:dynamic defs and change them with alter-var!. This would remove the deref overhead and possibly improve performance slightly. I don't think there's much risk in changing these to defs since they are only set during model registration. Which generally only happens once at system startup.

Consider the case when you have a user model and a document model.

Each document has an author-id which is a foreign key to the user table, and a reviewer which is also a foreign key to the user table. This results in multiple belong-to relations referencing the same table (user).

As discussed in slack:

• ecto does it like this: https://elixirforum.com/t/belongs-to-multiple-foreign-keys/8127/2
• ecto docs https://hexdocs.pm/ecto/Ecto.Schema.html#belongs_to/3

@kwrooijen suggested a format like this:

{:belongs-to {:document/author  {:model :user :through :document/author-id}}}
{:has-many   {:document/authors {:model :user :through :user/document-id}}}
{:has-one    {:document/authors {:model :user :through :user/document-id}}}

So, the example given at the top would look like:

(model/register!
  {:user
   [:map
    [:user/id {:primary-key true :auto true} int?]]})

(model/register!
  {:document
   [:map {:belongs-to {:document/author   {:model :user :through :document/author-id}
                       :document/reviewer {:model :user :through :document/reviewer-id}}}
    [:document/id {:primary-key true :auto true} int?]
    [:document/author-id int?]
    [:document/reviewer-id {:optional true} int?]
    ]})

When an exception is caught, gungnir is printing the honeysql form. Ref this line

This is bad for a few reasons:

1. It pollutes the stdout of the program
2. If the SQL query is very long (inserting a lot of data), then it can lock up the dev's REPL.

#2 is actually how I discovered this, an exception during a big insert was locking up my emacs repl.

The best solution is probably to use clojure.tools.logging and use the "debug" handler. That way clients can enable/disable gungnir logs as they see fit.

While implementing some next.jdbc support for postgres types (json, bytea, etc), I ran into a case where gungnir ate an exception it couldn't handle.

Gungnir is eating all Exceptions yet, the exception->map multimethod can only handle exceptions of type SQLException (see stack trace below).

I believe the solution to this is to catch only SQLExceptions and let others bubble up. Thoughts @kwrooijen ?

                                                      gungnir.query/save!                query.clj:  104
                                                      gungnir.query/save!                query.clj:  108
                                                 gungnir.database/insert!             database.clj:  259
                                            gungnir.database/execute-one!             database.clj:  187
                                            gungnir.database/execute-one!             database.clj:  195
                                                                      ...                               
                                            gungnir.database/eval47644/fn             database.clj:  158
java.lang.ClassCastException: class java.lang.IllegalArgumentException cannot be cast to class java.sql.SQLException (java.lang.IllegalArgumentException is in module java.base of loader 'bootstrap'; java.sql.SQLException is in module java.sql of loader 'platform')

To help track what postgres errors would be good to add default exception handlers for I've created this ticket.

All postgres errors are here: https://www.postgresql.org/docs/current/errcodes-appendix.html

• 42703 | undefined_column - likely to encounter this when your model field name does not match the table column

... more soon ...

Collection of things that still need to be documented.

query.md

• gungnir.query.insert!
• gungnir.query.update!

Currently relations are linked directly. That doesn't work well for the following use case:

user
---
id : uuid

user_organization
---
id : uuid
user_id : foreign key user(id)
organization_id : foreign key organization(id)

organization
---
id : uuid

In this case, users are linked to organizations through the user_organization table. We need to somehow resolve a user / organization relations through user_organization.

Possibly something along the lines of:

;; User model
[:map 
 {:has-many {:organization {:field :user/organizations, :through :user_organization/organization_id}}
 ,,,
 }]

;; Organizatin model
[:map 
 {:has-many {:user {:field :organization/users, :through :user_organization/user_id}}
 ,,,
 }]

I would like to contribute and maintain this project, is it active?

Currently Gungnir only supports a single global datasource. We should support having multiple instances.

From Slack:

@Ramblurr

And the second thing is related to that use of a global database var: In many systems it is desirable to connect to multiple different databases, but this won't be possible with gungnir.

I think it makes lots of sense to do away with the global var, and instead return a state-map that contains the functions needed.

I'm thinking of how the sente library works here: it's equivalent function to make-datasource! returns a map of state and functions for operating on that state. (edited)
For example make-datasource! could return a map that contains the keys:

:stop-fn - a function to close the connnection pool
:changeset-fn - the changeset function bound to this particular db connection
:database  - the *database* for this particular connection
:find!-fn  - the query/find! function for this particular connection
... and so forth

some thought would have to be given for how gungnir.model/register! would be bound to a particular gungnir instance

@kwrooijen

For your second point, yes Gungnir only supports a single database connection at the moment. I'm still thinking of a way to handle this. What I would really like is for multiple instances to be optional, whereas a single global instance is the default. I don't often have multiple datasources but I understand that this should be supported in some way.
One way would be to map datasources to identifiers. e.g. (gungnir/make-datasource! ::datasource-1 {,,,}) . Then when you register a model you can register it with a specific datasource

(gungnir.model/register!
 {:user
  [:map
   {:datasources [::datasource-1]}
   [,,,]]})

Then, whenever you query the database, you can pass in the datasource or key

Casey 2:51 PM
If all the functions supported an arity with a datasource param, that would probably work too

kwrooijen 2:52 PM
Yeah, I think that would be difficult with the current API though. the find! find-by! and all! functions are quite flexible but also not my most elegant pieces of code. Worst case we could create a different variant of these functions which support accepting a datasource (edited)

e.g. find-with-datasource! find-by-with-datasource! all-with-datasource! or something along those lines

(-> (q/select :%count.*)
    (q/from :app)
    (q/all!))
;; ERROR: {:type :malli.core/invalid-schema, :data {:schema nil}}

The reason this breaks is because the default row builder expects rows to be returned from a query. In our selection we just want to count the amount of rows that have matched, returning an int. How can we solve this?

One option would be to add "raw" query functions. e.g. q/all!! which skips the rows builder. It might be nice to skip rows from being build if they are not part of the model. That might be challenging to do but it could be a good solution.

Currently using schemas doesn't work. e.g. :table "foo.bar" will break the qualifier-builder function (among other things, possibly).

Hello Kevin,

Thank you for the great Gungnir presentation in the Dutch Clojure meetup.
The project is really great and inspiring.

Can you please advise if you are looking for any help or support for this project implementation (something small to start with) ?

Regards, Eugene

Currently in order to load a relation, you need to deref the relation atom of a record. This results in a new query being executed. It would be nice to be able to preload relations, querying them with a join instead of separate queries.

A solution could be to add a new function gungnir.query/preload, which will add a :gungnir/preload '(,,,) key to the HoneySQL map. We then need to modify the query before executing it.

(-> (q/preload :user/posts)
    (q/find-by! :user/email "[email protected]"))

Another thing we need to think about is how do we adjust the result of the relation? Normally we swap! the relation atom, but in this case we'd have to do that beforehand. Maybe a function to update preloads?

(-> (q/preload :user/posts)
    (q/preload-update :user/posts q/limit 5)
    (q/preload-update :user/posts q/sort-by :post/score :desc)
    (q/find-by! :user/email "[email protected]"))

Gungnir is currently a single library (and gungnir.ui), but it could be separated into three parts.

Model / Changeset

All the cljc files are regarding the model and changeset code. This code is generic and can be used for anything. No SQL / Database code is here.

SQL (HoneySQL)

The second part is generation of SQL. This partly done by the query namespace, and partly done by the database namespace. This should all be moved to the query namespace, and the namespace should be renamed to gungnir.sql.honeysql. This would allows us to support other querying languages in Clojure.

Database / Pool (HikariCP)

The last part is the database / pool. As noted about, the honeysql code should be removed from here and placed in gungnir.sql.honeysql. We use jdbc.next, and rely heavily on it, so this should not be configurable. The database pool on the other hand should be. So the creation of the HikariCP datapool should be extracted from here and placed in gungnir.database.hikaricp.

This would allow us to initialize gungnir in a configurable manner:

(gungnir/initialize!
 {:gungnir/sql :gungnir.sql/honeysql
  :gungnir/database :gungnir.database/hikaricp})

We'd probably also want to have honeysql / hikaricp as default. That if you call gungnir/initialize! without any arguments, it will default to those two libraries. This is to ensure the "Plug & Play" feel.

It would be nice if there was some sort of Seeding mechanism for creating test data. Malli already provides great support for data generation so we could easily use that. What Gungnir could do is automatically insert it in the database.

A way to close the underlying connection pool would be handy when using state management systems like mount or component.

Adding gungnir.database/close-datasource! would suffice here.

Currently Gungnir assumes all primary keys are generated through postgres. There should be an option to create it ourselves (in the case of save! Gungnir update, instead of insert). Maybe expose insert / update separately?

Currently the model name will be mapped to the table name. The resulting model (before it is stored using gungnir.model/register!) has a property :table which contains the table name. This name is extracted from the key that you register it with.

It could be desirable to have a model name not be the same as a table name. Or possibly even have multiple models for a single table?

It would be nice to have some sort of data-driven migration system built in. This doesn't mean that we have to build everything from scratch. We could also decide to use a migration library such as Ragtime and expand on that.

I'm not sure what we exactly want with this. But I think it would be nice to have a syntax for basic migration action such as

• create / update / delete table
• create / update / delete field
• setting the type of the field
  ...

And if you need to do something very specific, grant the ability to fall back to raw SQL.