influitive / blueshift Goto Github PK

Right now I'd have to have a .env file in order for blueshift to setup my test database. Running RAILS_ENV=test rake pg:schema:load won't actually do anything against my test db as blueshift, nor dotenv doesn't know about Rails, nor should they.

I get this error:

rake aborted!
Sequel::DatabaseError: PG::UndefinedColumn: ERROR:  column "name" does not exist
LINE 1: ...ttribute"."attnotnull" AS "allow_null", COALESCE(("name" = '...
                                                             ^

e.g.

Blueshift.migration do 
  change { add_column :products, :price, Float }
  redchange :ditto
end

It seems as if we can get metadata about redshift tables in order to figure out what the primary key is.

http://stackoverflow.com/questions/21273439/how-do-i-get-table-and-columns-information-from-redshift/29775745#29775745

this isn't high priority, so don't worry about it now, just logging for future considerations.

This should be the default because up {} down {} redup {} reddown {} is too verbose.

± br |br/group_contact_company_id ✓| → rake redshift:schema:dump
rake aborted!
Sequel::DatabaseError: PG::UndefinedColumn: ERROR: column "name" does not exist
LINE 1: ...ttribute"."attnotnull" AS "allow_null", COALESCE(("name" = '...
^
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:186:in async_exec' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:186:inblock in execute_query'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/database/logging.rb:48:in log_connection_yield' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:186:inexecute_query'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:173:in block in execute' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:149:incheck_disconnect_errors'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:173:in execute' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:525:in_execute'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:341:in block (2 levels) in execute' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:546:incheck_database_errors'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:341:in block in execute' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/database/connecting.rb:251:inblock in synchronize'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/connection_pool/threaded.rb:105:in hold' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/database/connecting.rb:251:insynchronize'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:341:in execute' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/dataset/actions.rb:970:inexecute'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/adapters/postgres.rb:669:in fetch_rows' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/dataset/actions.rb:141:ineach'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/dataset/actions.rb:392:in map' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/dataset/actions.rb:392:inmap'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/blue-shift-0.3.1/lib/sequel/adapters/redshift.rb:109:in schema_parse_table' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/database/query.rb:161:inschema'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/extensions/schema_dumper.rb:246:in dump_table_generator' /Users/dev/.rvm/gems/ruby-2.3.0/gems/blue-shift-0.3.1/lib/sequel/extensions/redshift_schema_dumper.rb:10:indump_table_schema'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/extensions/schema_dumper.rb:138:in block in dump_schema_migration' /Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/extensions/schema_dumper.rb:138:inmap'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/sequel-4.35.0/lib/sequel/extensions/schema_dumper.rb:138:in dump_schema_migration' /Users/dev/.rvm/gems/ruby-2.3.0/gems/blue-shift-0.3.1/lib/tasks/schema.rake:51:inblock (4 levels) in <top (required)>'
/Users/dev/.rvm/gems/ruby-2.3.0/gems/blue-shift-0.3.1/lib/tasks/schema.rake:51:in open' /Users/dev/.rvm/gems/ruby-2.3.0/gems/blue-shift-0.3.1/lib/tasks/schema.rake:51:inblock (3 levels) in <top (required)>'
/Users/dev/.rvm/gems/ruby-2.3.0/bin/ruby_executable_hooks:15:in eval' /Users/dev/.rvm/gems/ruby-2.3.0/bin/ruby_executable_hooks:15:in

This line will break in development (the app won't even load) if you don't have these env vars set.

This shouldn't be required to just start the app in development mode. One thought is to lazily load the connections, but these should be thread safe as doing a

def self.redshift_db
  @redshift_db ||= Sequel.connect()
end

is NOT thread safe. Perhaps assign them to a thread local var.

Right now Suuid type assumes NOT NULL. This means I can't add new columns, backfill, and later make NOT NULL

This is actually Sequel's default behaviour to include trailing whitespace , but that doesn't mean we have to live with it.

I get this after migration

-
+
     create_table!(:schema_migrations) do
-      String :filename, :null=>false
-
-      primary_key [:filename]
+      String :filename, :size=>255, :null=>false
     end

see more here