Comments (8)
Another possibility is to create a parallel data dictionary
where we can have more freedom to add metadata as needed, I did something like this here https://github.com/mingodad/db-api-server/blob/master/dbapi.db.sql
from cg-sql.
There's probably two different things going on here. This one is particularly tricky.
If you declare your schema with CQL when it actually emits the create statements it rewrites long_int into just integer. It does this because in SQLite an integer is big enough to hold int64. So the declaration only serves to tell you what data type to fetch from the db. There's actually special logic for this exact case because in SQLite you must say exactly "id integer primary key autoincrement" -- it's quite specific about this. That means if you use LONG_INT in your CQL input you can't then send that directly to SQLite. But that's ok... that's not usually how you do this.
First let me show you what I mean about the codegen:
/*
CREATE PROC foo ()
BEGIN
CREATE TABLE t4(
id LONG_INT PRIMARY KEY AUTOINCREMENT,
data BLOB
);
END;
*/
CQL_WARN_UNUSED cql_code foo(sqlite3 *_Nonnull _db_) {
cql_code _rc_ = SQLITE_OK;
_rc_ = cql_exec(_db_,
"CREATE TABLE t4( "
"id INTEGER PRIMARY KEY AUTOINCREMENT, " // <----- LOOK AT THAT <<---- <-----
"data BLOB "
")");
if (_rc_ != SQLITE_OK) goto cql_cleanup;
_rc_ = SQLITE_OK;
cql_cleanup:
return _rc_;
}
See how the codegen actually changed?
Now if you're going to deploy schema with CQL you can do this with the schema maintenance features.
Here's an example with just one table:
sources % cat x.sql
CREATE TABLE t4(
id LONG_INT PRIMARY KEY AUTOINCREMENT,
data BLOB
);
sources % out/cql --in x.sql --rt schema_upgrade --cg upgrader.sql --global_proc upgrader
% cat upgrader.sql
-- @generated SignedSource<<deadbeef8badf00ddefec8edfacefeed>>
-- no columns will be considered hidden in this script
-- DDL in procs will not count as declarations
@SCHEMA_UPGRADE_SCRIPT;
-- schema crc -5635641207918476938
-- declare sqlite_master --
CREATE TABLE sqlite_master (
type TEXT NOT NULL,
name TEXT NOT NULL,
tbl_name TEXT NOT NULL,
rootpage INTEGER NOT NULL,
sql TEXT NOT NULL
);
-- declare full schema of tables and views to be upgraded and their dependencies --
CREATE TABLE t4(
id LONG_INT PRIMARY KEY AUTOINCREMENT,
data BLOB
);
-- facets table declaration --
CREATE TABLE IF NOT EXISTS upgrader_cql_schema_facets(
facet TEXT NOT NULL PRIMARY KEY,
version LONG INTEGER NOT NULL
);
-- saved facets table declaration --
CREATE TEMP TABLE upgrader_cql_schema_facets_saved(
facet TEXT NOT NULL PRIMARY KEY,
version LONG INTEGER NOT NULL
);
-- helper proc for testing for the presence of a column/type
CREATE PROCEDURE upgrader_check_column_exists(table_name TEXT NOT NULL, decl TEXT NOT NULL, OUT present BOOL NOT NULL)
BEGIN
SET present := (SELECT EXISTS(SELECT * FROM sqlite_master WHERE tbl_name = table_name AND sql GLOB decl));
END;
-- helper proc for creating the schema version table
CREATE PROCEDURE upgrader_create_cql_schema_facets_if_needed()
BEGIN
CREATE TABLE IF NOT EXISTS upgrader_cql_schema_facets(
facet TEXT NOT NULL PRIMARY KEY,
version LONG INTEGER NOT NULL
);
END;
-- helper proc for saving the schema version table
CREATE PROCEDURE upgrader_save_cql_schema_facets()
BEGIN
DROP TABLE IF EXISTS upgrader_cql_schema_facets_saved;
CREATE TEMP TABLE upgrader_cql_schema_facets_saved(
facet TEXT NOT NULL PRIMARY KEY,
version LONG INTEGER NOT NULL
);
INSERT INTO upgrader_cql_schema_facets_saved
SELECT * FROM upgrader_cql_schema_facets;
END;
-- helper proc for setting the schema version of a facet
CREATE PROCEDURE upgrader_cql_set_facet_version(_facet TEXT NOT NULL, _version LONG INTEGER NOT NULL)
BEGIN
INSERT OR REPLACE INTO upgrader_cql_schema_facets (facet, version) VALUES(_facet, _version);
END;
-- helper proc for getting the schema version of a facet
CREATE PROCEDURE upgrader_cql_get_facet_version(_facet TEXT NOT NULL, out _version LONG INTEGER NOT NULL)
BEGIN
BEGIN TRY
SET _version := (SELECT version FROM upgrader_cql_schema_facets WHERE facet = _facet LIMIT 1);
END TRY;
BEGIN CATCH
SET _version := -1;
END CATCH;
END;
-- helper proc for getting the schema version CRC for a version index
CREATE PROCEDURE upgrader_cql_get_version_crc(_v INTEGER NOT NULL, out _crc LONG INTEGER NOT NULL)
BEGIN
BEGIN TRY
SET _crc := (SELECT version FROM upgrader_cql_schema_facets WHERE facet = 'cql_schema_v'||_v LIMIT 1);
END TRY;
BEGIN CATCH
SET _crc := -1;
END CATCH;
END;
-- helper proc for setting the schema version CRC for a version index
CREATE PROCEDURE upgrader_cql_set_version_crc(_v INTEGER NOT NULL, _crc LONG INTEGER NOT NULL)
BEGIN
INSERT OR REPLACE INTO upgrader_cql_schema_facets (facet, version) VALUES('cql_schema_v'||_v, _crc);
END;
-- helper proc to reset any triggers that are on the old plan --
DECLARE PROCEDURE cql_exec_internal(sql TEXT NOT NULL) USING TRANSACTION;
CREATE PROCEDURE upgrader_cql_drop_legacy_triggers()
BEGIN
DECLARE C CURSOR FOR SELECT name from sqlite_master
WHERE type = 'trigger' AND name GLOB 'tr__*';
LOOP FETCH C
BEGIN
call cql_exec_internal(printf('DROP TRIGGER %s;', C.name));
END;
END;
CREATE PROCEDURE upgrader_cql_install_baseline_schema()
BEGIN
CREATE TABLE IF NOT EXISTS t4(
id LONG_INT PRIMARY KEY AUTOINCREMENT,
data BLOB
);
END;
-- declared upgrade procedures if any
-- drop all the triggers we know
CREATE PROCEDURE upgrader_cql_drop_all_triggers()
BEGIN
CALL upgrader_cql_drop_legacy_triggers();
END;
CREATE PROCEDURE upgrader_perform_needed_upgrades()
BEGIN
DECLARE column_exists BOOL NOT NULL;
DECLARE facet_version LONG INTEGER NOT NULL;
DECLARE schema_version LONG INTEGER NOT NULL;
-- fetch current schema version --
CALL upgrader_cql_get_facet_version('cql_schema_version', schema_version);
-- check for downgrade --
IF schema_version > 0 THEN
SELECT 'downgrade detected' facet;
ELSE
-- save the current facets so we can diff them later --
CALL upgrader_save_cql_schema_facets();
-- dropping condemned or changing triggers --
CALL upgrader_cql_drop_all_triggers();
---- install baseline schema if needed ----
CALL upgrader_cql_get_version_crc(0, schema_version);
IF schema_version != 4355262830921719841 THEN
CALL upgrader_cql_install_baseline_schema();
CALL upgrader_cql_set_version_crc(0, 4355262830921719841);
END IF;
CALL upgrader_cql_set_facet_version('cql_schema_version', 0);
CALL upgrader_cql_set_facet_version('cql_schema_crc', -5635641207918476938);
-- finally produce the list of differences
SELECT T1.facet FROM
upgrader_cql_schema_facets T1
LEFT OUTER JOIN upgrader_cql_schema_facets_saved T2
ON T1.facet = T2.facet
WHERE T1.version is not T2.version;
END IF;
END;
CREATE PROCEDURE upgrader()
BEGIN
DECLARE schema_crc LONG INTEGER NOT NULL;
-- create schema facets information table --
CALL upgrader_create_cql_schema_facets_if_needed();
-- fetch the last known schema crc, if it's different do the upgrade --
CALL upgrader_cql_get_facet_version('cql_schema_crc', schema_crc);
IF schema_crc <> -5635641207918476938 THEN
-- save the current facets so we can diff them later --
CALL upgrader_perform_needed_upgrades();
ELSE
-- some canonical result for no differences --
SELECT 'no differences' facet;
END IF;
END;
The upgrader produces a new set of stored procedures you can then compile to create that schema. There are more features that let you add/delete columns but importantly. You compile the above again.
out/cql --in upgrader.sql --cg u.h u.c
When you do that the long int will be mapped back to integer.
Now if you don't want to this by using CQL's upgrade facility another easy thing you can do is use the C-Preprocessor to fix it.
cc -E -x c x.sql -DLONG_INT=integer | sed "/^#/d"
CREATE TABLE t4(
id integer PRIMARY KEY AUTOINCREMENT,
data
You can give SQLite the above.
The thing is none of this explains why stage 14 failed for you. My guess there is that the test case uses a feature that is only in a more modern version of sqlite
sources % sqlite3
SQLite version 3.28.0 2019-04-15 14:49:49
That's what I have.
from cg-sql.
If you don't want to use the schema maintenance features you can just make one big proc to create your schema. That procedure will get the integer re-write and you're all good.
from cg-sql.
In the context of the tests (any stage) you don't have to worry about the integer primary key thing because CQL automatically fixes it.
from cg-sql.
common/test_common.sh line 1025 begins that block, there should be more specific output to help you see which step failed.
query_plan_test() {
echo '--------------------------------- STAGE 15 -- TEST QUERY PLAN'
if you get this far:
# linux build of sqlite doesn't have explain query plan
echo run query plan in c
That comment is actually stale, but for a time the linux had an older sqlite and EXPLAIN QUERY PLAN was absent.
If you can do this:
sqlite> create table foo(id integer);
sqlite> explain query plan select * from foo;
0|0|0|SCAN TABLE foo (~1000000 rows)
That's a good sanity check.
from cg-sql.
Hello !
Thanks for reply !
I'm using Ubuntu 18.04 and it comes with sqlite3 3.22 so a lot of people will have this issue, when linking against the latest sqlite3 it pass the test.
sqlite3/bionic-updates,bionic-security 3.22.0-1ubuntu0.4 amd64
Command line interface for SQLite 3
The error reported with sqlite3 3.22 (default on ubuntu 18.04):
--------------------------------- STAGE 15 -- TEST QUERY PLAN
C preprocessing
semantic analysis
codegen query plan
semantic analysis
validating test
Verification results: 1 tests matched 32 patterns of which 0 were errors.
validating query plan codegen
computing diffs (empty if none)
validate udf test
validating udf codegen
computing diffs (empty if none)
build query plan c code
compile query plan code
cc -I./ -Iout -g -std=c99 -D_Nullable= -D_Nonnull= -c query_plan_test.c -o out/query_plan_test.o
cc -I./ -Iout -g -std=c99 -D_Nullable= -D_Nonnull= -c -o out/query_plan.o out/query_plan.c
cc -I./ -Iout -g -std=c99 -D_Nullable= -D_Nonnull= -c -o out/udf.o out/udf.c
cc -I./ -Iout -g -std=c99 -D_Nullable= -D_Nonnull= -o out/query_plan_test out/query_plan_test.o out/cqlrt.o out/query_plan.o out/udf.o -lsqlite3 -pthread -ldl
run query plan in c
out/query_plan_test returned a failure code
failed populating query 9
sqlite error: near "ON": syntax error
--------------------------------- FAILED
Cheers !
from cg-sql.
OK we got this sorted, it's the Ubuntu SQLite version. We should probably do some diagnostics on this or something.
from cg-sql.
I added notes for this in the README.
from cg-sql.
Related Issues (20)
- doc bug in cql-guide example HOT 2
- cql "--cg" first argument semantics make it awkward to generate code to an "output" directory HOT 10
- demo_todo.sql bug, and cg-sql language question: should mixed int / long operations warn? HOT 1
- cqlrt: Would it be worth adding a prepared statement cache? HOT 6
- Somehow related project "urweb" HOT 1
- There is some "[unsigned] long" types around, trouble on windows HOT 5
- Nice to have better code generation for prepared statements inside loops HOT 24
- Non reserved keywords not accepted by CG-SQL HOT 4
- Error `-Werror=maybe-uninitialized` when trying to build with `cosmopolitan` HOT 6
- Missing '--rt lua' in 'static void cql_usage()' HOT 1
- Missing indentation for Lua/C generated code HOT 1
- A better prepared statement generated code ? HOT 6
- Generated code needs a prefix to prevent clash HOT 12
- cgsql.dev links are pointing to bad paths in Github. HOT 1
- CG-SQL-Lua online playground HOT 14
- unsub entries do not need a name HOT 1
- Simplified CQL Syntax HOT 9
- Simplified Shared Fragment Calls
- facebook version of CG/SQL is no longer pushing commits (unlinked from internal) HOT 2
- This fork is no longer maintained: See https://github.com/ricomariani/CG-SQL-author
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cg-sql.