Giter Club home page Giter Club logo

Comments (8)

mingodad avatar mingodad commented on May 29, 2024

Another possibility is to create a parallel data dictionary where we can have more freedom to add metadata as needed, I did something like this here https://github.com/mingodad/db-api-server/blob/master/dbapi.db.sql

from cg-sql.

ricomariani avatar ricomariani commented on May 29, 2024

There's probably two different things going on here. This one is particularly tricky.

If you declare your schema with CQL when it actually emits the create statements it rewrites long_int into just integer. It does this because in SQLite an integer is big enough to hold int64. So the declaration only serves to tell you what data type to fetch from the db. There's actually special logic for this exact case because in SQLite you must say exactly "id integer primary key autoincrement" -- it's quite specific about this. That means if you use LONG_INT in your CQL input you can't then send that directly to SQLite. But that's ok... that's not usually how you do this.

First let me show you what I mean about the codegen:

/*
CREATE PROC foo ()
BEGIN
  CREATE TABLE t4(
    id LONG_INT PRIMARY KEY AUTOINCREMENT,
    data BLOB
  );
END;
*/

CQL_WARN_UNUSED cql_code foo(sqlite3 *_Nonnull _db_) {
  cql_code _rc_ = SQLITE_OK;
  _rc_ = cql_exec(_db_,
    "CREATE TABLE t4( "
      "id INTEGER PRIMARY KEY AUTOINCREMENT, "  // <----- LOOK AT THAT <<---- <-----
      "data BLOB "
    ")");
  if (_rc_ != SQLITE_OK) goto cql_cleanup;
  _rc_ = SQLITE_OK;

cql_cleanup:
  return _rc_;
}

See how the codegen actually changed?

Now if you're going to deploy schema with CQL you can do this with the schema maintenance features.

Here's an example with just one table:

sources % cat x.sql

  CREATE TABLE t4(
    id LONG_INT PRIMARY KEY AUTOINCREMENT,
    data BLOB
  );

sources % out/cql --in x.sql --rt schema_upgrade --cg upgrader.sql --global_proc upgrader
% cat upgrader.sql
-- @generated SignedSource<<deadbeef8badf00ddefec8edfacefeed>>

-- no columns will be considered hidden in this script
-- DDL in procs will not count as declarations
@SCHEMA_UPGRADE_SCRIPT;

-- schema crc -5635641207918476938

-- declare sqlite_master -- 
CREATE TABLE sqlite_master (
  type TEXT NOT NULL,
  name TEXT NOT NULL,
  tbl_name TEXT NOT NULL,
  rootpage INTEGER NOT NULL,
  sql TEXT NOT NULL
);

-- declare full schema of tables and views to be upgraded and their dependencies -- 
CREATE TABLE t4(
  id LONG_INT PRIMARY KEY AUTOINCREMENT,
  data BLOB
);

-- facets table declaration --
CREATE TABLE IF NOT EXISTS upgrader_cql_schema_facets(
  facet TEXT NOT NULL PRIMARY KEY,
  version LONG INTEGER NOT NULL
);

-- saved facets table declaration --
CREATE TEMP TABLE upgrader_cql_schema_facets_saved(
  facet TEXT NOT NULL PRIMARY KEY,
  version LONG INTEGER NOT NULL
);

-- helper proc for testing for the presence of a column/type
CREATE PROCEDURE upgrader_check_column_exists(table_name TEXT NOT NULL, decl TEXT NOT NULL, OUT present BOOL NOT NULL)
BEGIN
  SET present := (SELECT EXISTS(SELECT * FROM sqlite_master WHERE tbl_name = table_name AND sql GLOB decl));
END;

-- helper proc for creating the schema version table
CREATE PROCEDURE upgrader_create_cql_schema_facets_if_needed()
BEGIN
  CREATE TABLE IF NOT EXISTS upgrader_cql_schema_facets(
    facet TEXT NOT NULL PRIMARY KEY,
    version LONG INTEGER NOT NULL
  );
END;

-- helper proc for saving the schema version table
CREATE PROCEDURE upgrader_save_cql_schema_facets()
BEGIN
  DROP TABLE IF EXISTS upgrader_cql_schema_facets_saved;
  CREATE TEMP TABLE upgrader_cql_schema_facets_saved(
    facet TEXT NOT NULL PRIMARY KEY,
    version LONG INTEGER NOT NULL
  );
  INSERT INTO upgrader_cql_schema_facets_saved
    SELECT * FROM upgrader_cql_schema_facets;
END;

-- helper proc for setting the schema version of a facet
CREATE PROCEDURE upgrader_cql_set_facet_version(_facet TEXT NOT NULL, _version LONG INTEGER NOT NULL)
BEGIN
  INSERT OR REPLACE INTO upgrader_cql_schema_facets (facet, version) VALUES(_facet, _version);
END;

-- helper proc for getting the schema version of a facet
CREATE PROCEDURE upgrader_cql_get_facet_version(_facet TEXT NOT NULL, out _version LONG INTEGER NOT NULL)
BEGIN
  BEGIN TRY
    SET _version := (SELECT version FROM upgrader_cql_schema_facets WHERE facet = _facet LIMIT 1);
  END TRY;
  BEGIN CATCH
    SET _version := -1;
  END CATCH;
END;

-- helper proc for getting the schema version CRC for a version index
CREATE PROCEDURE upgrader_cql_get_version_crc(_v INTEGER NOT NULL, out _crc LONG INTEGER NOT NULL)
BEGIN
  BEGIN TRY
    SET _crc := (SELECT version FROM upgrader_cql_schema_facets WHERE facet = 'cql_schema_v'||_v LIMIT 1);
  END TRY;
  BEGIN CATCH
    SET _crc := -1;
  END CATCH;
END;

-- helper proc for setting the schema version CRC for a version index
CREATE PROCEDURE upgrader_cql_set_version_crc(_v INTEGER NOT NULL, _crc LONG INTEGER NOT NULL)
BEGIN
  INSERT OR REPLACE INTO upgrader_cql_schema_facets (facet, version) VALUES('cql_schema_v'||_v, _crc);
END;

-- helper proc to reset any triggers that are on the old plan --
DECLARE PROCEDURE cql_exec_internal(sql TEXT NOT NULL) USING TRANSACTION;
CREATE PROCEDURE upgrader_cql_drop_legacy_triggers()
BEGIN
  DECLARE C CURSOR FOR SELECT name from sqlite_master
     WHERE type = 'trigger' AND name GLOB 'tr__*';
  LOOP FETCH C
  BEGIN
    call cql_exec_internal(printf('DROP TRIGGER %s;', C.name));
  END;
END;

CREATE PROCEDURE upgrader_cql_install_baseline_schema()
BEGIN
  CREATE TABLE IF NOT EXISTS t4(
    id LONG_INT PRIMARY KEY AUTOINCREMENT,
    data BLOB
  );
  
END;
-- declared upgrade procedures if any

-- drop all the triggers we know
CREATE PROCEDURE upgrader_cql_drop_all_triggers()
BEGIN
  CALL upgrader_cql_drop_legacy_triggers();
END;

CREATE PROCEDURE upgrader_perform_needed_upgrades()
BEGIN
  DECLARE column_exists BOOL NOT NULL;
  DECLARE facet_version LONG INTEGER NOT NULL;
  DECLARE schema_version LONG INTEGER NOT NULL;

  -- fetch current schema version --
  CALL upgrader_cql_get_facet_version('cql_schema_version', schema_version);

  -- check for downgrade --

  IF schema_version > 0 THEN
    SELECT 'downgrade detected' facet;
  ELSE

    -- save the current facets so we can diff them later --
    CALL upgrader_save_cql_schema_facets();
    -- dropping condemned or changing triggers --
    CALL upgrader_cql_drop_all_triggers();

    ---- install baseline schema if needed ----

    CALL upgrader_cql_get_version_crc(0, schema_version);
    IF schema_version != 4355262830921719841 THEN
      CALL upgrader_cql_install_baseline_schema();
      CALL upgrader_cql_set_version_crc(0, 4355262830921719841);
    END IF;

    CALL upgrader_cql_set_facet_version('cql_schema_version', 0);
    CALL upgrader_cql_set_facet_version('cql_schema_crc', -5635641207918476938);

    -- finally produce the list of differences
    SELECT T1.facet FROM
      upgrader_cql_schema_facets T1
      LEFT OUTER JOIN upgrader_cql_schema_facets_saved T2
        ON T1.facet = T2.facet
      WHERE T1.version is not T2.version;
  END IF;
END;

CREATE PROCEDURE upgrader()
BEGIN
  DECLARE schema_crc LONG INTEGER NOT NULL;

  -- create schema facets information table --
  CALL upgrader_create_cql_schema_facets_if_needed();

  -- fetch the last known schema crc, if it's different do the upgrade --
  CALL upgrader_cql_get_facet_version('cql_schema_crc', schema_crc);

  IF schema_crc <> -5635641207918476938 THEN
    -- save the current facets so we can diff them later --
    CALL upgrader_perform_needed_upgrades();
  ELSE
    -- some canonical result for no differences --
    SELECT 'no differences' facet;
  END IF;
END;

The upgrader produces a new set of stored procedures you can then compile to create that schema. There are more features that let you add/delete columns but importantly. You compile the above again.

out/cql --in upgrader.sql --cg u.h u.c

When you do that the long int will be mapped back to integer.

Now if you don't want to this by using CQL's upgrade facility another easy thing you can do is use the C-Preprocessor to fix it.

cc -E -x c x.sql -DLONG_INT=integer | sed "/^#/d"

  CREATE TABLE t4(
    id integer PRIMARY KEY AUTOINCREMENT,
    data

You can give SQLite the above.

The thing is none of this explains why stage 14 failed for you. My guess there is that the test case uses a feature that is only in a more modern version of sqlite

sources % sqlite3
SQLite version 3.28.0 2019-04-15 14:49:49

That's what I have.

from cg-sql.

ricomariani avatar ricomariani commented on May 29, 2024

If you don't want to use the schema maintenance features you can just make one big proc to create your schema. That procedure will get the integer re-write and you're all good.

from cg-sql.

ricomariani avatar ricomariani commented on May 29, 2024

In the context of the tests (any stage) you don't have to worry about the integer primary key thing because CQL automatically fixes it.

from cg-sql.

ricomariani avatar ricomariani commented on May 29, 2024

common/test_common.sh line 1025 begins that block, there should be more specific output to help you see which step failed.

query_plan_test() {
  echo '--------------------------------- STAGE 15 -- TEST QUERY PLAN'

if you get this far:

  # linux build of sqlite doesn't have explain query plan
  echo run query plan in c

That comment is actually stale, but for a time the linux had an older sqlite and EXPLAIN QUERY PLAN was absent.

If you can do this:

sqlite> create table foo(id integer);
sqlite> explain query plan select * from foo;
0|0|0|SCAN TABLE foo (~1000000 rows)

That's a good sanity check.

from cg-sql.

mingodad avatar mingodad commented on May 29, 2024

Hello !
Thanks for reply !
I'm using Ubuntu 18.04 and it comes with sqlite3 3.22 so a lot of people will have this issue, when linking against the latest sqlite3 it pass the test.

sqlite3/bionic-updates,bionic-security 3.22.0-1ubuntu0.4 amd64
  Command line interface for SQLite 3

The error reported with sqlite3 3.22 (default on ubuntu 18.04):

--------------------------------- STAGE 15 -- TEST QUERY PLAN
C preprocessing
semantic analysis
codegen query plan
semantic analysis
validating test
Verification results: 1 tests matched 32 patterns of which 0 were errors.
validating query plan codegen
  computing diffs (empty if none)
validate udf test
validating udf codegen
  computing diffs (empty if none)
build query plan c code
compile query plan code
cc -I./ -Iout -g -std=c99 -D_Nullable= -D_Nonnull= -c query_plan_test.c -o out/query_plan_test.o
cc -I./ -Iout -g -std=c99 -D_Nullable= -D_Nonnull=   -c -o out/query_plan.o out/query_plan.c
cc -I./ -Iout -g -std=c99 -D_Nullable= -D_Nonnull=   -c -o out/udf.o out/udf.c
cc -I./ -Iout -g -std=c99 -D_Nullable= -D_Nonnull= -o out/query_plan_test out/query_plan_test.o out/cqlrt.o out/query_plan.o out/udf.o -lsqlite3 -pthread -ldl
run query plan in c
out/query_plan_test returned a failure code
failed populating query 9
sqlite error: near "ON": syntax error
--------------------------------- FAILED

Cheers !

from cg-sql.

ricomariani avatar ricomariani commented on May 29, 2024

OK we got this sorted, it's the Ubuntu SQLite version. We should probably do some diagnostics on this or something.

from cg-sql.

ricomariani avatar ricomariani commented on May 29, 2024

I added notes for this in the README.

from cg-sql.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.