Would be nice if dbmigrate would use SSL if specified in the url (sslmode=require)

Hello!

Consider myself a noobie, please.

I am (successfully) using your awesome migration tool (thanks for that!) programmatically in an application of mine.

To get it working I had a look at your implementation of dbmigrate (as suggested in the readme) and basically more or less re-implemented the same macro and functions myself.

dbmigrate is not a library (correct me if I am wrong and simply too dumb to get the crate somehow).

That leaves me with 2 questions/proposals:

Can it be done that functionality (e.g. migrate macro and up etc. functions) be moved to dbmigrate-lib?

If not (I would be curious as to the why then, actually really curious why they were done in the dbmigrate - as said, consider myself a noobie!), could dbmigrate be also made a lib?

Thanks again for your time and efforts!

Similar, but different from #26.

It would be great if there was the ability to create {up/down}.rs migrations alongside {up/down}.sql migrations. It is not uncommon to have data migrations, which are distinct from schema migrations. Using .rs (or even shell scripts) would make it possible to create migrations that SQL alone is incapable of.

Thanks for considering!

Hey!

I see that a few years ago dbmigrate used to provide release binaries in the GitHub repo.

Any reason for the stop - or was it just a hassle?

Any chance of publishing a new release with a recent version - if not I can totally just host my own somewhere - so it's really not an issue if it's a hassle.

Cheers!

The way Migration and MigrationFile are currently structured, a lot of handling of different cases is required, which could be prevented in the type system directly.

Migration

Let's start with Migration. Since up and down are Options, there are 4 possible states:

• up and down (makes sense)
• only up (might make sense)
• only down (probably wrong
• none (certainly wrong)
  I would argue that you should either only allow cases where up and down exist (1), or if you want to support the case where there is no down, at least make only down an Option (2).

1. up and down required: There is only 1 possible state
2. up required, down optional. Valid states:
   • up and down
   • only up

MigrationFile

I see several issues here:

1. Why is content an option? If there is no content, there shouldn't be a MigrationFile in the first place. If the file is empty, content should be an empty string, not None.
2. Putting the direction in the MigrationFile duplicates that information, making it potentially not match. One could for example put a MigrationFile with direction up into the down-Part of a Migration.
3. Both name and number should be the same between up and down migrations (or the number at least). Otherwise they can get out of sync which doesn't really make any sense.

Possible Solution

I suggest flattening everything down into the Migration struct:

struct Migration {
    pub up: String,
    pub down: Option<String>,
    pub number: u32, //negative migration numbers don't make any sense
    pub name: String, //the filenames can be calculated from the name if needed
}

This would greatly simplify the use of this library.

Consider the following folder structure:

migrations
migrations/base
migrations/dummy_data

Now, depending on some input either only migration files in base are to be executed or also the ones that insert some "dummy data".

From the top of my head there are two options to solve a situation like that:

1. Either make https://github.com/Keats/dbmigrate/blob/master/dbmigrate-lib/src/files.rs#L94 recursive
2. Allow multiple folders to be passed to the function

The workaround is copying everything from base also to dummy_data which is not really nice.

I wanted to discuss this, before starting any work.

Hi!

I can’t compile my application if I import something from the dbmigrate_lib. Compilation fails with this error:

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/sivakov512/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib"
// many letters
"-Wl,-Bdynamic" "-lsqlite3" "-lssl" "-lcrypto" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /bin/ld: cannot find -lsqlite3
          /bin/ld: /home/sivakov512/Devel/Projects/yafas/auth-server/target/debug/deps/libopenssl-fd2ddb3820966926.rlib(openssl-fd2ddb3820966926.openssl.ajlwxbfs-cgu.3.rcgu.o): in function `openssl::symm::Crypter::finalize':
          /home/sivakov512/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.10.24/src/symm.rs:581: warning: EVP_CipherFinal is often misused, please use EVP_CipherFinal_ex and EVP_CIPHER_CTX_cleanup
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

My Cargo.toml looks like this

[package]
name = "auth-server"
version = "0.1.0"
authors = ["Nikita Sivakov <[email protected]>"]
edition = "2018"

[dependencies]
clap = "2.33"
dbmigrate-lib = { version = "0.1", features = ["postgres_support"] }
futures = "0.1"
hyper = "0.12"
tokio = "0.1"
tokio-postgres = { version = "0.4.0-rc.3", features = ["with-uuid-0_7"] }
bb8 = "0.3"
bb8-postgres = "0.3"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
uuid = { version = "0.7", features = ["serde"] }

[dev-dependencies]
test-tools = { path = "test-tools" }
uuid = { version = "0.7", features = ["serde", "v4"] }

How to fix this error?

When migration script contains a mistake, then part of commands get executed. The second attempt will rerun script from the beginning, which results in executing the commands for the second time. This behavior is wrong.

Background

Say there are two developers named Jane and Rob working on the same codebase.

Main Branch

0001.first-migration.up.sql
0001.first-migration.down.sql

Jane's branch

0001.first-migration.up.sql
0001.first-migration.down.sql
0002.janes-migration.up.sql
0002.janes-migration.down.sql

Rob's branch

0001.first-migration.up.sql
0001.first-migration.down.sql
0002.robs-migration.up.sql
0002.robs-migration.down.sql

Merged

After Jane and Rob merge their branches into the main branch, the main branch looks like this:

0001.first-migration.up.sql
0001.first-migration.down.sql
0002.janes-migration.up.sql
0002.janes-migration.down.sql
0002.robs-migration.up.sql
0002.robs-migration.down.sql

The Problem

Right now dbmigrate does not return an error in this case, which can lead to:

• Not realizing that a migration never actually got ran
• When migrating up a different migrations might run than when migrating down.
  • For example when migrating up janes-migration.up.sql might run, but when migrating down robs-migration.down.sql might run.

Potential Fix

Make dbmigrate return an error if there are two or more migrations that have the same number.

If I PR a fix can you merge it and deploy a new version?

Cheers!

So i can use a path like ~/code/migrations

I'm running dbmigrate on a server with Python's Fabric library and I get this:

[174.138.34.126] out: thread 'main' panicked at 'called Result::unwrap() on an Err value: ColorOutOfRange', /checkout/src/libcore/result.rs:860
[174.138.34.126] out: note: Run with RUST_BACKTRACE=1 for a backtrace.

I solved it by sshing in and running dbmigrate manually, but it would be nice to be able to run dbmigrate from Fabric.

Any ideas?

I cannot have multiple mysql statements in one migration. The weird thing is it doesn't give an error or something, but it silently only executes the first statement. Am I doing something wrong or is this a bug?

Right now dbmigrate creates its __dbmigrate_table without specifying a schema, which by default will use the public schema.

In my use-case where I have schema1 and schema2 for two different projects. When I now use dbmigrate, the migrations of both projects try to use public.__dbmigrate_table, which causes them to mix up.

To avoid that, it would be nice to be able to specify a schema where dbmigrate looks for the migration table so that both migration tables can live separately in their respective schemas.

Build is failing on stable Rust (1.20.0):

error: type `io::IoPack + 'static` is private
    --> /home/mentalbrew/.cargo/registry/src/github.com-1ecc6299db9ec823/mysql-10.0.0/src/conn/mod.rs:1044:30
     |
1044 |         let (data, seq_id) = self.get_mut_stream().as_mut().read_packet(old_seq_id)?;
     |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: type `io::IoPack + 'static` is private
    --> /home/mentalbrew/.cargo/registry/src/github.com-1ecc6299db9ec823/mysql-10.0.0/src/conn/mod.rs:1051:22
     |
1051 |         let seq_id = self.get_mut_stream().as_mut().drop_packet(old_seq_id)?;
     |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: type `io::IoPack + 'static` is private
    --> /home/mentalbrew/.cargo/registry/src/github.com-1ecc6299db9ec823/mysql-10.0.0/src/conn/mod.rs:1059:23
     |
1059 |         self.seq_id = self.get_mut_stream().as_mut().write_packet(data, seq_id, max_allowed_packet)?;
     |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to 3 previous errors

Searching for this error revealed this support issue for the compiler:
rust-lang/rust#43662

The simplest path seems to be upgrading to a newer version of mysql crate. Is there any issue with doing so?

Adding dbmigrate-lib = "0.1.3" to my toml results in the following:

error: failed to run custom build command for `openssl v0.9.24`
process didn't exit successfully: `/home/jon/projects/example/target/debug/build/openssl-57d227ac03cd5782/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'Unable to detect OpenSSL version', /home/jon/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.9.24/build.rs:16:14
note: Run with `RUST_BACKTRACE=1` for a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

From what I understand, this issue is caused by using an older version of native-tls, although I'm not certain. If it is, it's also not clear to me which dependencies are bringing it in. The dependencies in my project are:

[dependencies]
chrono = "0.4"
clap = "2.32"
dbmigrate-lib = "0.1.3"
dotenv = "0.13"
dotenv_codegen = "0.11"
postgres = { version = "0.15", features = ["with-uuid", "with-chrono"] }
uuid = { version = "0.5", features = ["v4"] }

Hi,

any thoughts on this? I have an automated build/deploy case in which creating the database manually beforehand is painful. Given the already provided credentials, it would be lovely if dbmigrate up would take care of the initial database creation for me.

If the idea catches up, I'd like to give it a shot.

Best regards,
lwk

I think it might be reasonable to have for example a .gitignore file in the migrations directory. What do you think about just letting the directory walker skip invalid entries?

Why does creating 2 files require a database connection? It isn't running them.

Really like what you've done so far. Keep up the great work!

I won't have time to work on that anytime soon but will take a PR on the next branch.

This issue was automatically generated. Feel free to close without ceremony if
you do not agree with re-licensing or if it is not possible for other reasons.
Respond to @cmr with any questions or concerns, or pop over to
#rust-offtopic on IRC to discuss.

You're receiving this because someone (perhaps the project maintainer)
published a crates.io package with the license as "MIT" xor "Apache-2.0" and
the repository field pointing here.

TL;DR the Rust ecosystem is largely Apache-2.0. Being available under that
license is good for interoperation. The MIT license as an add-on can be nice
for GPLv2 projects to use your code.

Why?

The MIT license requires reproducing countless copies of the same copyright
header with different names in the copyright field, for every MIT library in
use. The Apache license does not have this drawback. However, this is not the
primary motivation for me creating these issues. The Apache license also has
protections from patent trolls and an explicit contribution licensing clause.
However, the Apache license is incompatible with GPLv2. This is why Rust is
dual-licensed as MIT/Apache (the "primary" license being Apache, MIT only for
GPLv2 compat), and doing so would be wise for this project. This also makes
this crate suitable for inclusion and unrestricted sharing in the Rust
standard distribution and other projects using dual MIT/Apache, such as my
personal ulterior motive, the Robigalia project.

Some ask, "Does this really apply to binary redistributions? Does MIT really
require reproducing the whole thing?" I'm not a lawyer, and I can't give legal
advice, but some Google Android apps include open source attributions using
this interpretation. Others also agree with
it.
But, again, the copyright notice redistribution is not the primary motivation
for the dual-licensing. It's stronger protections to licensees and better
interoperation with the wider Rust ecosystem.

How?

To do this, get explicit approval from each contributor of copyrightable work
(as not all contributions qualify for copyright, due to not being a "creative
work", e.g. a typo fix) and then add the following to your README:

## License

Licensed under either of

 * Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
 * MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.

### Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any
additional terms or conditions.

and in your license headers, if you have them, use the following boilerplate
(based on that used in Rust):

// Copyright 2016 dbmigrate Developers
//
// Licensed under the Apache License, Version 2.0, <LICENSE-APACHE or
// http://apache.org/licenses/LICENSE-2.0> or the MIT license <LICENSE-MIT or
// http://opensource.org/licenses/MIT>, at your option. This file may not be
// copied, modified, or distributed except according to those terms.

It's commonly asked whether license headers are required. I'm not comfortable
making an official recommendation either way, but the Apache license
recommends it in their appendix on how to use the license.

Be sure to add the relevant LICENSE-{MIT,APACHE} files. You can copy these
from the Rust repo for a plain-text
version.

And don't forget to update the license metadata in your Cargo.toml to:

license = "MIT OR Apache-2.0"

I'll be going through projects which agree to be relicensed and have approval
by the necessary contributors and doing this changes, so feel free to leave
the heavy lifting to me!

Contributor checkoff

To agree to relicensing, comment with :

I license past and future contributions under the dual MIT/Apache-2.0 license, allowing licensees to chose either at their option.

Or, if you're a contributor, you can check the box in this repo next to your
name. My scripts will pick this exact phrase up and check your checkbox, but
I'll come through and manually review this issue later as well.

• @Keats

Remove error-chain in favour of plain Error and update the rest of the dependencies.

It would be nice to be able to specify a stopping point when migrating. For instance, suppose we have 3 migrations.

0001.create_users
0002.add_profile_pic
0003.add_birthday

If none of them have been applied, we could apply just the first two like so:

dbmigrate -u <DATBASE_URL> -p <MIGRATION_FOLDER_PATH> up 0002

Similarly, if we had migrated all 3 and realized we wanted to roll back only the last one, we could do:

dbmigrate -u <DATBASE_URL> -p <MIGRATION_FOLDER_PATH> down 0002

IMHO, the selective rollback is something that really needs to exist before I'd want to use something like this in production.

What do you think?

So you can choose to only have one database if you want rather than all of them