Would you consider an integrant2 to support extensible build steps? about integrant HOT 10 CLOSED

Integrant is currently only version 0.8.0, so it might be a little early to think about Integrant 2.

I'm afraid I don't understand what you mean by "steps" here. What's the difference between a step and a function that transforms the configuration or system map?

Also, are you aware of the build, run! and reverse-run! functions?

from integrant.

Integrant is currently only version 0.8.0, so it might be a little early to think about Integrant 2.

I more mean that this design proposal is probably not backwards compat (Although, in theory it could be...). So it could go in a different integrant namespace.

I'm afraid I don't understand what you mean by "steps" here. What's the difference between a step and a function that transforms the configuration or system map?

Depends what you mean as well but in my head steps are chained together. The result of your previous step is passed to the next build step of the component. In practice, someone can combine any number of steps into a single step which does it all, but my thought is that breaking it apart makes it easier to reason about.

A small example, let's pretend we have 3 steps: [:load-defaults, :build-objects, :run]
Let's suppose we are configuring a http server to bind to and address on our machine and it includes a third party worker pool library.

(def config (read "my-file-location.edn"))

;; let's suppose config looks like
{:my.server/default {:addr "localhost",
                     :worker #ref [:my.worker/default],
                     :heartbeat #ref [:my.heatbeat.clientpool/default]},
 :my.worker.default {:heartbeat #ref [:my.heatbeat.clientpool/default]},
 :my.heartbeat.clientpool/default {:addr "https://internal.heartbeats.not.real.example"},

;; after the :load-defaults
;; it would look like
;; notice we added a default port
;; and max-threads
{:my.server/default {:addr "localhost",
                     :port 8080,
                     :worker #ref [:my.worker/default],
                     :heartbeat #ref [:my.heatbeat.clientpool/default]},
 :my.worker.default {:max-threads 8,
                     :heartbeat #ref [:my.heatbeat.clientpool/default]},
 :my.heartbeat.clientpool/default {:addr "https://internal.heartbeats.not.real.example",
                     :max-threads 4}}

;; after :build-objects
;; it would look like
;; we converted our config data
;; to java objects which will run
;; our program
{:my.server/default #object [some.jetty.server 0xdeadbeef ...],
 :my.worker/default #object [thirdparty.worker.library 0xdeadbeef ...],
 :my.heartbeat.clientpool/default #object [thirdparty.heartbeat.client 0xdeadbeef ...]}

;; after the :run step
;; the map would look the same
;; but the step function :run
;; would be called for both
;; :my.server/default and :my.worker/default

;; sidenote, it would be useful to get the config of one of the components
;; example
(fashionable/input-of system :build-objects [:my.server/default]) =>
{:my.server/default {:addr "localhost",
                     :port 8080,
                     :worker #object [thirdparty.worker.library 0xdeadbeef ...],
                     :heartbeat #object [thirdparty.heartbeat.client 0xdeadbeef ...]}}

In think you can accomplish the above example, using integrant today using: prep and init but I think there is value in being able to customize and break apart those steps.

Also, are you aware of the build, run! and reverse-run! functions?

Let me take a look. I think integrant works well for most cases but on more than one occasion I had the desire to separate out different build steps. Which I think is the main proposal (being able to extend/customize the build steps), not that you can't accomplish the same goal w/ integrant as it is now.

from integrant.

Maybe, combining something like aero and integrant as it is now is good enough.

from integrant.

Depends what you mean as well but in my head steps are chained together. The result of your previous step is passed to the next build step of the component.

Can't you just chain the functions? e.g.

(-> config foo ig/prep bar ig/init baz)

Where foo, bar and baz are your custom steps.

You can use the build, run! and reverse-run! functions to create "step" functions with different dependency resolutions. Behind the scenes, init uses build, and halt! uses reverse-run!

from integrant.

I will take take a look at build and run!

(-> config foo ig/prep bar ig/init baz)

That's a good example. I imagine creating foo, bar, and baz would miss out on the key resolution configuration that ig/prep and ig/init is aware of.

Example, let's suppose foo needs to add a default port for the keys:

• my.database/primary
• my.database/backup
  They both derive database/postgresql

Example 1: I would imagine the definition of foo would look something like:

(defn foo
  [config]
  (reduce
    (fn [c [k defaults]
      (update c k merge defaults)
    config
   [[:my.database/primary {:port 4336}]
    [:my.database/backup {:port 4337}]])))

Example 2: Another example (if the developer is feeling more ambitious) is to mimic how prep or init work:

(defmulti prep-foo (fn [k config] k))

(defmethod prep-foo :default [k config] config)

(defmethod prep-foo :database/postgresql [k config]
  (let [defaults {:my.database/primary {:port 4336}
                     :my.database/backup {:port 4337}}
  (merge defaults config))

(defn foo
  [config]
  (reduce
    (fn [c k]
      (prep-foo k c))
    config
    (keys config)))

I would prefer to be able to leverage the existing resolution code in integrant and only have to write:

;; let's assume I defined the original method :database/postrgresql
;; if someone else defined it, we can create a new keyword (e.g. :my.database/postgresql)
;; and have :my.database/primary or :my.database/default derive
;; from the new keyword instead
(defmethod prep :foo :database/postgresql [k config]
  (let [defaults {:my.database/primary {:port 4336}
                     :my.database/backup {:port 4337}}
  (merge defaults config))

I would prefer not to do Example 1 because it is aware of each key (across different components) that it is updating.
And I would also not to do Example 2 because it seems like I'm reimplementing something which has already been solved in order to make foo work.

Maybe run! and build solve this.

from integrant.

I think build

may solve this exact issue

(defmethod prep-foo :default [k config] config)

(defmethod prep-foo :database/postgresql [k config]
  (let [defaults {:my.database/primary {:port 4336}
                     :my.database/backup {:port 4337}}
  (merge defaults config))

(defn foo
  [config keys]
  (ig/build config keys prep-foo))

from integrant.

I'm going to close this and experiment if this works.

from integrant.

Thank you!

from integrant.

No problem.

from integrant.

Yeah, I think having separate build steps might be "overkill". It looks like having something like aero pass a config to integrant works well. Thanks again

from integrant.