lukaszbudnik / migrator Goto Github PK

implement loader interface for Azure Blob

The Azure Blob implementation should use similar concept to AWS default chain credentials provider so that there are no other additional cloud-vendor specific configuration needed.

current performance test framework creates migrations in one go.

for a better performance tests we should be able to add new migrations into existing ones to replicate more real life test cases

A little bit more fancier name of xcli would be core.

Also, xcli is used by server module. So core is a better name for it.

At the moment migrator is quite taciturn.

Add logging for:

• current migration
• current tenant

@lukaszbudnik One suggestion to make the application easier to use.

In order to make it easier to monitor how Migrator connects with a DB server would be good set application_name upon connecting to the server. This would allow a user to monitor on the DB's side when Migrator is working on.
That feature can come helpful when there is a problem with applying a migration.

Steps to reproduce

I'm testing Migrator against a PostgreSQL server.

1. Start a migration.
2. List open DB connections.

SELECT
  application_name,
  query
FROM pg_catalog.pg_stat_activity
WHERE state IS NOT NULL;

    application_name    |                                                                       query                                                                       
  ------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------
   PostgreSQL JDBC Driver | SET application_name = 'PostgreSQL JDBC Driver'
   PostgreSQL JDBC Driver | SET application_name = 'PostgreSQL JDBC Driver'
                          | select name, source_dir as sd, filename, type, db_schema, created, contents, checksum from migrator.migrator_migrations order by name, source_dir
                          | create schema if not exists migrator

3. The listing shows JDBC connections and lib/pq connections.

Actual results

There is no application_name set for Migrator therefore it is hard to list activity of Migrator only.
On the other hand JDBC connections are labelled and it is easier to track them.

Expected

For a connection with Migrator application_name is set.

SET application_name = 'Migrator'

References

Documentation: https://www.postgresql.org/docs/9.5/runtime-config-logging.html#GUC-APPLICATION-NAME
Proposed function to change:

Currently migrator supports Slack web hooks.

This should be re-worked to support any type of POST web hooks.

In order to compare migrator to other DB migration frameworks a simple performance test suite is required

I've applied DB migrations and the operation ended successfully.
The migration was suspended for some time due to a secondary transaction.
After the successful migration I got the following summary.

      "summary": {                                                                                                                                                                                
        "startedAt": "2021-09-15T11:08:00.162242499Z",                                                                                                                                            
        "duration": -1948086662,                                                                                                                                                                  
        "tenants": 6,                                                                                                                                                                             
        "singleMigrations": 0,                                                                                                                                                                    
        "tenantMigrationsTotal": 0,                                                                                                                                                               
        "migrationsGrandTotal": 0,                                                                                                                                                                
        "singleScripts": 8,                                                                                                                                                                       
        "tenantScriptsTotal": 246,                                                                                                                                                                
        "scriptsGrandTotal": 254                                                                                                                                                                  
      }                                                                                                                                                                                           

where the duration is obviously wrong - it's negative.

I was wondering why exactly we cast to int32 in the following passage:

Currently MS SQL Server is not provided out of the box on Travis. However, Travis supports running docker containers. We could port test/docker/scripts/mssql-create-and-setup-container.sh

Currently both CreateVersion and CreateTenant call GetVersionByID to return version with DB migrations.

This could be reworked by simply returning the created Version and DB Migrations directly. The bit missing is that after inserting the DB migration we need to fetch ID of that migration and add to version. There is already mechanism in place in migrator for fetching Version ID so we can replicate it there too.

Removal of CLI is a major change planned for v3.0.

The default port is currently implemented in config module.

Default value should be moved to server module.

Currently "response" can be:

• summary - only summary of executed migrations and scripts
• full - contains full results including summary with a list of all applied migrations and scripts, the later includes migration/script contents which proved to be too verbose for large systems

The idea is to provide a third, backward compatible, response type:

• list - this will return same response as in full but without the actual migration/script contents

The https://github.com/aws/aws-sdk-go is still actively maintained and new versions are released daily.

For Go 1.15+ there is an AWS SDK Go v2 available: https://github.com/aws/aws-sdk-go-v2.

With added support for 1.17, migrator moved supported versions to 1.15+. migrator can now make the switch to new AWS SDK Go.

Coordinator.GetSourceMigrations(*SourceMigrationFilters) operation accepts optional filters inSourceMigrationFilters struct.

Under the hood it is implemented as getting all source migrations from and then filtering them:

func (c *coordinator) GetSourceMigrations(filters *SourceMigrationFilters) []types.Migration {
	allSourceMigrations := c.loader.GetSourceMigrations()
	filteredMigrations := c.filterMigrations(allSourceMigrations, filters)
	return filteredMigrations
}

The implementation could be improved and the specific loader (disk, AWS S3, Azure Blob Storage) could accept*SourceMigrationFilters and return already filtered results - better performance and faster operation.

Merge migrator-docker project into migrator

migrator should allow reloads of single and tenant scripts as these are special types of migrations which are expected to change over a time.

currently if you update single or tenant script migrator will error and return a list of offending migrations.

when creating new tenant we should also apply all known migrations for it

At the moment printing tables and String() methods are defined in types module. These should be placed in separate utils module. Also String() method formatting output as table row is too much and looks odd when used in for example tests.

There are some technical challenges for implementing MSSQL (and they are all mentioned in https://github.com/lukaszbudnik/migrator/projects/2).

MSSQL has different syntax for:

• create table if not exists
• different way of handling timestamp/date/times

Currently MySQL and PostgreSQL vary only in prepared statement parameter syntax. For MSSQL there much bigger challenges.

Knowing that other DB engines have different syntax for creating dialects (migrator uses syntax which works for both MySQL and PostgreSQL) migrator should allow overriding default create table syntax.

Implement it in dialects.

There should be one script for creating and destroying containers. Name of DB could be passed as argument to create/destroy scripts.

Currently migrator loads migrations from disk. Implement loader interface for AWS S3

The https://github.com/lib/pq is still maintained however there is a newer and more regularly updated Go driver: https://github.com/jackc/pgx. This new driver is also recommended by the authors of lib/pq.

migrator should make the switch to this new PostgreSQL Go driver.