Giter Club home page Giter Club logo

drift's Introduction

drift

Drift is a migration library written in Clojure. Drift works much like Rails migrations where a directory in your project contains all of the migration files. Drift will detect which migration files need to be run and run them as appropriate.

Migration files must contain an up and down function in which you can perform the migration using your prefered database library.

Usage

To use Drift you'll need to add drift to your Leiningen project. Simply add the following to your project.clj file:

[drift "x.x.x"]

Where "x.x.x" is the latest version of drift which you can find on clojars: http://clojars.org/drift

If you want to use the lein commands, you'll need to add the above vector to your :dev-dependencies vector. If you want to access drift code directly in your project, you'll need to add it to your :dependencies vector. You can add drift to both vectors without any problems.

If you're using drift with lein2, you'll need to add it to the :plugins list in the project.clj file.

To set your Drift migration directory, simply add a clojure file called "migrate_config.clj" to the directory "src/config".

Your migrate_config.clj file should look something like:

 (ns config.migrate-config)

 (defn migrate-config []
   { :directory "/src/migrations"
     :current-version current-db-version-fn
     :update-version update-db-version-fn })

directory must be a directory included in the classpath. Most likely you want your migrations somewhere under src.

current-db-version-fn and update-db-version-fn are both functions which you must implement to let Drift read and set the current db version.

current-db-version-fn does not take any parameters, and returns the current version of the database. If no version has been set, then current-db-version-fn must return 0.

update-db-version-fn takes only a new-version parameter. After update-db-version-fn is run with a given new-version, current-db-version-fn must return that version.

For example, here is an in memory db version (note, you do not want to do this):

 (ns config.migrate-config)
 
 (def version (atom nil))
 
 (defn memory-current-db-version []
   (or @version 0)) 
 
 (defn memory-update-db-version [new-version]
   (reset! version new-version))
 
 (defn migrate-config []
   { :directory "/test/migrations"
     :current-version memory-current-db-version
     :update-version memory-update-db-version })

Here is an example database version using a table named "schema_migrations" that has one version column with a single value holding the current database version:

(ns config.migrate-config
  (:require [clojure.contrib.sql :as sql])
  (:use warehouse.core))

(defn db-version []
  (sql/with-connection DB
    (sql/with-query-results res 
      ["select version from schema_migrations"]
      (or (:version (last res)) 0))))

(defn update-db-version [version]
  (sql/with-connection DB
    (sql/insert-values :schema_migrations [:version] [version])))

Your migration files must contain an up and down function in which you perform the migration and rollback using your prefered database library.

For example:

 (ns migrations.001-create-authors-table
  (:require [clojure.java.jdbc :as j])

(def mysql-db { :dbtype "mysql" :dbname "my_db" :user "root" })

(defn up []
  (j/query mysql-db
    ["CREATE TABLE authors(id INT PRIMARY KEY)"]))

(defn down []
  (j/query mysql-db
    ["DROP TABLE authors"]))

Initialization

If you need to run some initialization code, add :init to your migrate-config. For example:

(defn migrate-config []
   { :directory "/test/migrations"
     :init init
     :current-version memory-current-db-version
     :update-version memory-update-db-version })

The above migrate-config will call the function "init" before any migrations are run. If you pass parameters to lein migrate, they will be passed along to the init function.

If you want to include a special use or require section in the namespace function of all migration files you can use the :ns-content key.

For example:

(defn migrate-config []
   { :directory "/test/migrations"
     :ns-content "\n  (:use database.util)"
     :current-version memory-current-db-version
     :update-version memory-update-db-version })

The above migrate-config will add "\n (:use database.util)" to the namespace function call at the top of every migration file.

Using Leiningen

To migrate to the most recent version:

$ lein migrate

To migrate to a specific version, pass the version as a parameter to migrate. For example, to migrate to version 1:

$ lein migrate -version 1

To undo all migrations and start with a clean database, simply pass 0 as the migration number. For example:

$ lein migrate -version 0

To create a new migration file which you can then edit:

$ lein create-migration <migration name>

Calling Drift From Java

You can call Drift from any java application. Simply create an instance of the Drift object:

Drift myDrift = new Drift();

After you create an instance of the Drift object, you must initialize it, or your Drift migrations may run in a bad state.

myDrift.init(Collections.EMPTY_LIST);

The one argument to init is a list which will be passed on to the init function set in your migrate_config. In the above example, the init function does not require any parameters, so an empty list is passed in.

You can get the current database version with:

myDrift.currentVersion();

You can get the maximum migration number with:

myDrift.maxMigrationNumber();

Finally, to run the migrations, you can use the function migrate:

myDrift.migrate(myDrift.maxMigrationNumber(), Collections.EMPTY_LIST);

The migrate function takes the migration number to migrate to, and a parameter list which is passed onto the init function. In the above example, we tell Drift to migrate to the maximum version and pass no arguments since our init function does not need any.

License

Copyright (C) 2009 Matthew Courtney and released under the Apache 2.0 license.

drift's People

Contributors

andrewmacquarrie avatar gfredericks avatar ghostandthemachine avatar joegallo avatar krisleech avatar loganlinn avatar macourtney avatar maxbrunsfeld avatar mccraigmccraig avatar rosejn avatar rplevy-draker avatar sardtok avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

drift's Issues

lein migrate not working

Hello guys, It seems this is no longer maintained, if so please add that to the README.
Created new project with lein new,
my project.clj looks like

(defproject test-drift "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url "https://www.eclipse.org/legal/epl-2.0/"}
  :dependencies [[org.clojure/clojure "1.10.0"]
                 [mysql/mysql-connector-java "8.0.26"]
                 [org.clojure/java.jdbc "0.7.12"]]
  :plugins [[drift "1.5.3"]]
  :repl-options {:init-ns test-drift.core})

added the configs to src/config/migrate_config.clj

(ns config.migrate-config
  (:require [clojure.java.jdbc :as j]))

(def db-spec {:connection-uri "jdbc:mysql://localhost:3306/sot?user=root&password="})

(defn get-db-version
  []
  (let [version (try (first
                      (j/query db-spec
                               ["select max(version) as version from schema_migrations"]
                               {:row-fn :version}))
                     (catch Exception e 0))]
    (or version 0)))

(defn update-db-version
  [version]
  (try (j/db-do-commands db-spec
                        [(j/create-table-ddl :schema_migrations
                                             [[:version :int]])])
       (catch Exception e nil))
  (j/insert! db-spec :schema_migrations {:version version}))

(defn migrate-config
  []
  {:directory "/resources/migrations"
   :current-version get-db-version
   :update-version update-db-version})

lein create-migration creates new file but lein migrate does not identify files for migration and returns always 0 version :

Aug 22, 2021 9:16:14 PM clojure.lang.Reflector invokeMatchingMethod
INFO: Current database version: 0
Aug 22, 2021 9:16:14 PM clojure.lang.Reflector invokeMatchingMethod
INFO: Updating to version: 0
Aug 22, 2021 9:16:14 PM clojure.lang.Reflector invokeMatchingMethod
INFO: No changes were made to the database.

Edit: The file for migration was created with create-migration

Not really an "issue" but some questions

I see several uses of (defn ^{:doc "..."} foobar ...) Is there a reason for it being that way as opposed to (defn foobar "..." ...)?

Also, would it be amenable to have the generated migrations be named after a timestamp (e.g. "20111121105223" like they are in rails) instead of an incrementing number? This is a lesson learned from Rails where you have multiple developers committing migrations at the same time. It could maybe be done as an option that you can turn on or off.

I'd be glad to work up a pull request or two around this.

Issue running migrations on heroku

Hi, when I'm running lein migrate on heroku I get following error:

Caused by: java.io.FileNotFoundException: Could not locate leiningen/core/eval__init.class or leiningen/core/eval.clj on classpath:

Any advice?
Running something like lein run -m some.ns works fine.

Thanks

ClassNotFoundException running lein tasks

This is almost certainly user error, but I'm not sure what I'm doing wrong.

$ lein version
Leiningen 1.6.1 on Java 1.6.0_26 Java HotSpot(TM) 64-Bit Server VM

project.clj:
(defproject clj-contact "1.0.0-SNAPSHOT"
:description "Example Contact Manager Component"
:dependencies [[org.clojure/clojure "1.2.1"]
[org.clojure/clojure-contrib "1.2.0"]
[mysql/mysql-connector-java "5.1.17"]]
:dev-dependencies [ [drift "1.2.1"] ])

$ lein deps
Copying 3 files to /Users/james/bitmechanic/clj-contact/lib
Copying 5 files to /Users/james/bitmechanic/clj-contact/lib/dev

So far so good. But then:

$ lein create-migration create_contact_table
Exception in thread "main" java.lang.ClassNotFoundException: drift.generator (NO_SOURCE_FILE:1)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:5376)
at clojure.lang.Compiler.analyze(Compiler.java:5190)
at clojure.lang.Compiler.analyze(Compiler.java:5151)
Caused by: java.lang.ClassNotFoundException: drift.generator
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)

lib/dev has drift-1.2.1.jar -- and jar tvf shows that the drift files are in there. I tried unjarring it and moving the drift dir into my src dir, but I got the same error.

thoughts?

-- James

Ability to redefine *config-fn-symbol* in lein project

Spent about hour or so, it looks impossible. It's impossible to use :global-vars - as it expands to set! which can't re-set dynamic and there is no other way to interfere.\

For some legacy purposes I need my migration config in src/migrations/config.clj (don't even ask). Symlink worked fine so far, but now I need to support windows as well ;)

Simple :drift-config option in lein plugin 'd be really helpfull.

drift.execute and drift.generator FileNotFound erro with Lein 2

When running
$ lein migrate
of
$ lein create_migration

I get

$ lein migrate
All namespaces already :aot compiled.
Exception in thread "main" java.io.FileNotFoundException: Could not locate drift/execute__init.class or drift/execute.clj on classpath:
at clojure.lang.RT.load(RT.java:432)
at clojure.lang.RT.load(RT.java:400)
at clojure.core$load$fn__4890.invoke(core.clj:5415)
at clojure.core$load.doInvoke(core.clj:5414)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$load_lib.doInvoke(core.clj:5264)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$load_libs.doInvoke(core.clj:5298)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:603)
at clojure.core$require.doInvoke(core.clj:5381)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at user$eval1.invoke(NO_SOURCE_FILE:1)
at clojure.lang.Compiler.eval(Compiler.java:6511)
at clojure.lang.Compiler.eval(Compiler.java:6500)
at clojure.lang.Compiler.eval(Compiler.java:6477)
at clojure.core$eval.invoke(core.clj:2797)
at clojure.main$eval_opt.invoke(main.clj:297)
at clojure.main$initialize.invoke(main.clj:316)
at clojure.main$null_opt.invoke(main.clj
...

No warnings on invalid command line

This may be just an annoyance, but it wasted us a bunch of time.

If you run "lein migrate [version number]" by accident (instead of "lein migrate -version [version number]") the version number is silently ignored, and instead drift migrates to the latest version.

This is rather confusing!

Are there any guides or tutorials available?

I’m interested in using Drift but I’m having trouble getting a sense of how to really use it with drift-db. I’ve never used any migrations framework before, and I get the sense that Drift’s current documentation assumes that one has. For example, I don’t understand where to store the migration files, and what form the files should be, and what they should contain. I don’t understand how I am supposed to indicate to the tool what migration files map to what version, or in what order they should be applied, etc. It’s not even clear to me how I initialize a “flavor”.

Missing migrations directory throws unhelpful NPE

While working on another project, I ended up spending a long time chasing down a NullPointerException inside Drift. I found the culprit being that the configured migrations directory (as indicated by the :directory key in the config map) was missing. As a new user, the NPE was unhelpful, since I had inherited my code, which I had presumed ran correctly in the past. The NPE didn't provide useful breadcrumbs.

The location in question in Drift that raised the unhelpful NPE is here:

drift/src/drift/core.clj

Lines 114 to 119 in 1acfe11

(defn default-migration-namespaces []
(->> (find-migrate-directory)
.listFiles
(map #(.getName ^File %))
(filter #(re-matches #".*\.clj$" %))
(map namespace-string-for-file)))

I'd imagine that a null check that throws a useful missing directory error might help someone else in the future.

Last migration is being run first by `lein migrate` for new (version 0) database

Here's the output when I run lein migrate:

Jan 06, 2016 5:32:53 PM sun.reflect.NativeMethodAccessorImpl invoke0
INFO: Current database version: 0
Jan 06, 2016 5:32:54 PM sun.reflect.NativeMethodAccessorImpl invoke0
INFO: Updating to version: 20151013202740
Jan 06, 2016 5:32:54 PM sun.reflect.NativeMethodAccessorImpl invoke0
INFO: Running ledgeries.migrations.20151013202740-modify-table-transaction-entries-allow-null-description-values up...
Exception in thread "main" java.sql.BatchUpdateException: Batch entry 0 ALTER TABLE transaction_entries ALTER COLUMN description DROP NOT NULL was aborted.  Call getNextException to see the cause.
    at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2598)
    at org.postgresql.core.v3.QueryExecutorImpl$1.handleError(QueryExecutorImpl.java:459)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:407)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2737)
    at clojure.java.jdbc$execute_batch.invoke(jdbc.clj:319)
    at clojure.java.jdbc$db_do_commands$fn__5268.invoke(jdbc.clj:520)
    at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:480)
    at clojure.java.jdbc$db_do_commands.doInvoke(jdbc.clj:519)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invoke(core.clj:623)
    at clojure.java.jdbc$db_do_commands.doInvoke(jdbc.clj:513)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at clojure.core$apply.invoke(core.clj:619)
    at clojure.java.jdbc$do_commands.doInvoke(jdbc.clj:847)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invoke(core.clj:617)
    at ledgeries.db$do_commands$fn__5396.invoke(db.clj:55)
    at clojure.java.jdbc$with_connection_STAR_.invoke(jdbc.clj:770)
    at ledgeries.db$do_commands.doInvoke(db.clj:54)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at ledgeries.migrations.20151013202740_modify_table_transaction_entries_allow_null_description_values$up.invoke(20151013202740_modify_table_transaction_entries_allow_null_description_values.clj:7)
    at clojure.lang.Var.invoke(Var.java:411)
    at drift.runner$run_migrate_up.invoke(runner.clj:41)
    at clojure.core$map$fn__4207.invoke(core.clj:2487)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:60)
    at clojure.lang.RT.seq(RT.java:484)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
    at clojure.core.protocols$fn__6026.invoke(protocols.clj:54)
    at clojure.core.protocols$fn__5979$G__5974__5992.invoke(protocols.clj:13)
    at clojure.core$reduce.invoke(core.clj:6177)
    at drift.runner$migrate_up_all.invoke(runner.clj:67)
    at drift.runner$migrate_up.invoke(runner.clj:86)
    at drift.runner$update_to_version.invoke(runner.clj:120)
    at drift.execute$migrate$fn__717.invoke(execute.clj:28)
    at drift.core$with_init_config$fn__579.invoke(core.clj:30)
    at clojure.lang.AFn.applyToHelper(AFn.java:159)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:617)
    at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1788)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at drift.config$with_config_map.invoke(config.clj:16)
    at drift.core$with_init_config.invoke(core.clj:28)
    at drift.execute$migrate.invoke(execute.clj:26)
    at drift.execute$run$fn__721.invoke(execute.clj:35)
    at clojure.lang.AFn.applyToHelper(AFn.java:159)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:617)
    at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1788)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at drift.config$with_config_fn_symbol.invoke(config.clj:12)
    at drift.execute$run.invoke(execute.clj:33)
    at user$eval23.invoke(form-init7623668639265027663.clj:1)
    at clojure.lang.Compiler.eval(Compiler.java:6619)
    at clojure.lang.Compiler.eval(Compiler.java:6609)
    at clojure.lang.Compiler.load(Compiler.java:7064)
    at clojure.lang.Compiler.loadFile(Compiler.java:7020)
    at clojure.main$load_script.invoke(main.clj:294)
    at clojure.main$init_opt.invoke(main.clj:299)
    at clojure.main$initialize.invoke(main.clj:327)
    at clojure.main$null_opt.invoke(main.clj:362)
    at clojure.main$main.doInvoke(main.clj:440)
    at clojure.lang.RestFn.invoke(RestFn.java:421)
    at clojure.lang.Var.invoke(Var.java:419)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.main.main(main.java:37)
Subprocess failed

Very strangely, 20151013202740-modify-table-transaction-entries-allow-null-description-values is the last migration, of 27 total migrations.

I'm running Drift 1.5.2. Output of lein -v:

Leiningen 2.5.3 on Java 1.8.0_65 Java HotSpot(TM) 64-Bit Server VM

Init function called by "lein create-migration ..." command?

I'm not sure that it shouldn't be called by that command, but I don't understand why it would be, and I'd prefer either that it didn't.

Specifically, my init function adds a row to a table migrations in my database with an auto-generated timestamp for a column started_at. The update function should update that row with both the version of the migration that was just completed and the date that the migration completed. [This isn't necessary, but I wanted to be able to know how long migrations are taking.]

README

should indicate that drift needs to be added to the :plugins list in project.clj in a lein2 project

this stumped me for a little while :)

Problem getting correct max migration version

I set up the following directory: /project_root/db/migrations.

In (migrate-config), :directory was "/db/migrations".

I ran lein create-migration add_blah. The file appeared in /db/migrations as expected.

I ran lein migrate.

Output:
INFO: Current database version: 0
(...)
INFO: Updating to version: 0
Jul 30, 2012 7:10:15 PM sun.reflect.NativeMethodAccessorImpl invoke0
INFO: No changes were made to the database.
(process stops)

I then tried changing the directory to /project_root/src/migrations. Now everything works as expected.

There seems to be a problem with some code called by (core/max-migration-number).

v1.5.3 tag is missing

Version 1.5.3 isn't tagged, as far as I can tell from looking on GitHub.

Not a major issue but I encountered this when looking through the source for the version I was using.

proposal : lein cmdline arg to specify config function

would you accept a pull request which allows the passing of an (optional) extra cmdline arg to the lein commands create-migration and migrate, permitting the config function to be explicitly named : e.g.

lein create-migration -c config.migrate/cassandra new-migration
lein migrate -c config.migrate/postgresql -v 12345

i haven't done it yet, but it looks straightforward, and i will probably do it in the next few days

thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.