araddon / qlbridge Goto Github PK
View Code? Open in Web Editor NEWA golang expression evaluator & Library to build SQL query engine based functionality.
License: MIT License
A golang expression evaluator & Library to build SQL query engine based functionality.
License: MIT License
Variety of improvements for FilterQL:
WITH
clause was not available in String()
functions and couldn't roundtripMisc usability improvements on the underlying expression Node's
IN
statement from MultiArg -> BinaryNode w ArrayNode typeAs a proxy, expressions in SQL could potentially be evaluated in one of several points of the lifecycle:;
col +1
or set type operations in cassandra, n1ql or postgres or mongoHow to express these 3 different contexts? Currently I think only 1 is supported for update/insert/delete and 2 is not supported at all and 3 for select only.
UPDATE counterks.page_view_counts
SET counter_value = counter_value + 2
WHERE url_name='www.datastax.com' AND page_name='home';
CQL set operation
UPDATE users
SET emails = emails + {'[email protected]'} WHERE user_id = 'frodo';
We need a way of having a source-backed-source. So that a rest-api or some high-latency, low query/index feature set can be fronted by a richer query engine.
In implementation of the new Node -> Expr -> Json we should add round-tripping to our tests.
Additionally, we are having problems with left.right identities.
expr
expr.ToPb
Currently the function-registry is assumed to be global scope. This is flawed, each table
in a schema would have its own function registry, so that a elasticsearch backed table would not have the same available functions as say a mysql one, etc.
need a traditional/simple ast for non-vm usage
Expression needs to support to/from json to allow json api usage.
Node
interfaces
Include
BooleanNode
Expr
for Jsontable.field name
type identities.Expr
for ProtobufReally convenient to choose position from end.
Currently in Elasticsearch Generator #158 the Right side of a left-right binary expressio must be a value, not an identity. We can epress right side identities using es5 scripting.
To be useful, it must expose schema info to tools such as Metabase, Looker, Tableau, etc. Make the currently in memory structs available as schema like traditional db, ie for show, create, like statements.
db
.table
syntaxtemp
LIKE '%'@@info
dataux
.jobstate
;table
FROM db
;Create a cloud storage file reader https://github.com/lytics/cloudstorage that allows the file-parsing to be pluggable, and the pre-fetching, and supports partitioning api.
Some dialects literals are escaped with ', others are ", Identties similarly are different. Pass through dialect writers to the Node
and Statement
's to allow dialect-specific outputs. This implies that qlbridge is starting to implement sql-builder functionality, for reasons of being able to take a query, and transform it into a different query. (most builders are to aid construction of sql by using non-sql native golang functions). See https://github.com/umisama/go-sqlbuilder and https://github.com/elgris/golang-sql-builder-benchmark
expr
pkgrel
sql, filterql libCurrently queries like FILTER x > 5
are invalid and must be written as FILTER AND (x > 5)
.
It would be nice to support single expression queries without a boolean operator.
allow a MustParseFilterQL(query) that panics if not successful, useful for init() methods and testing.
Panic on Equal()
panic(0x7ff8d34a8ba0, 0xc4200100c0)
/usr/local/go/src/runtime/panic.go:458 +0x243
github.com/lytics/lio/vendor/github.com/araddon/qlbridge/expr.(*ArrayNode).Equal(0xc48bbfc390, 0x7ff8d454e780, 0xc48bbfc4b0, 0x1)
/home/go/src/github.com/lytics/lio/vendor/github.com/araddon/qlbridge/expr/node.go:1979 +0x11a
github.com/lytics/lio/vendor/github.com/araddon/qlbridge/expr.(*BinaryNode).Equal(0xc4a2f15da0, 0x7ff8d454e800, 0xc4a2f15740, 0x1)
/home/go/src/github.com/lytics/lio/vendor/github.com/araddon/qlbridge/expr/node.go:1387 +0x193
github.com/lytics/lio/vendor/github.com/araddon/qlbridge/expr.(*BooleanNode).Equal(0xc4a2f15620, 0x7ff8d454e880, 0xc4a2f15c80, 0x1c)
/home/go/src/github.com/lytics/lio/vendor/github.com/araddon/qlbridge/expr/node.go:1529 +0x190
github.com/lytics/lio/vendor/github.com/araddon/qlbridge/rel.(*FilterStatement).Equal(0xc49e9eb290, 0xc49e9ebc30, 0x0)
/home/go/src/github.com/lytics/lio/vendor/github.com/araddon/qlbridge/rel/filter.go:147 +0x225
The old one is deprecated, and more importantly can't be used for benchmarks.
Many functions can have per-function shared state as many arguments are static for that parsed expression.
Maybe the Validate
can return the func?
// The last arg "," is static, and the function should be able to pre-calculate that value
join(arg1,arg2, ",")
Error messages on parse are not very good.
Need a persistent data source to persist schema info etc. Obviously just a default to test easily, but most would upgrade to mysql etc.
Originally, i made a big mistake on left.right namespacing identities, and allowed a bad format. Need to find a way to back this out. utility to find/remove?
particularly in generators/elasticsearch
# legacy
`key_name.field value` -> "key_name", "field value"
# correct future state
`key_name.field value` -> "", "key_name.field value"
The schema discovery/setup layer is horrible code, clean it up.
Need nested/scoped contexts, so first context we read is row, then up a level (session), then global (@@vars)
mysql> SELECT @@max_allowed_packet;
Some sources (think json files) have sparse columns within a row, that is vs the entire possible keys within a keyspace (row) any one row may have a subset. The messages that now require schema enforce creation of non-sparse rows representing these sparse rows as they go through sql exec tasks. This is really slow for very, very sparse data sets.
See
Line 971 in 2548fa7
value.Value
returned from functions?Args
type that supports variadics so we don't have different function args?TODO
implement a database/sql driver
There is too much overlap.
Lex/Parse, DataTypes for user-defined types
http://docs.datastax.com/en/cql/3.3/cql/cql_using/useCreateUDT.html
Implement groupby and aggregations Tasks. Just simple builtin aggregation functions (sum, avg, max)
user_id
as uid from users group by user_id"select count(distinct(field)) from users
Allow functions in XQL to pass through to structs, instead of only supporting builtin funcs.
for example FirstName()
below would pass through to context-reader instead of builtins
SELECT FirstName("--") FROM user
Currently parser doesn't support the 3 joins, sub-query joins
Ensure logically equivalent statements equate to Equal() in code
by collapsing un-used logical statements:
FILTER OR (x = "y") = FILTER x = "y"
FILTER NOT (x = "y") = FILTER X != "y"
Equal()
for FilterQLAllow a distributed planner (dataux) to replace built in qlbridge execution planner (in exec
) . Currently there is no separation of planning, execution, and no strong interface to allow the distributed executor to replace the native in-process qlbridge executor. Also, no serialization interfaces/capability to allow tasks to be deterministically distributed, and no primitives (partitions, tablets) to allow work to be broken down.
Phase 1 usecase: Query Mongo with 3 nodes participating, a groub-by aggregate query select AVG(CHAR_LENGTH(CAST(
titleAS CHAR))) as title_avg from article GROUP BY category
plan.Visitor
with full suite of plan structs instead of using rel
sql structstaskparent.Add(task)
) to be interfaces not implementationTaskMaker
factory for overriding built in task implementationsplan.PlannerDefault
and all new plan typesCustom
)depended on by dataux/dataux#9
moving until later:
MarshalJson()
or Explain()
?The sql vm interpreter, needs to be separated with a very, very clear api/interface for implementing the Storage Layers (currently ContextWriter, ContextReader
)
LeftRight has a bug for back-tick escaping with periods in it
FILTER `last.event`.`has.period` == "12/18/2015"
Build out the common function library.
CAST
function http://www.cheatography.com/davechild/cheat-sheets/mysql/Currently the expression Walk logic, and context.Get() are not consistent with respect to:
nil, true
for could not be evaluated, some ErrValue, false
, some return NilValue, false
TODO:
currently select * FROM table
does not support any other columns, implement the following
SELECT *, now() AS cur_time, yymm(now()) as yymm FROM table
datasource that will be good for files-api
Job/Exec engine pipeline/tasks need complete refactor.
Revamp the Plan/Exec with the following Goals
SourceTask
interface in dataux move to Accept
visitor pattern w poly-fillTasks
from
in query is its own source & discrete task
Where
implemented in each SubSelectJoinKey
function/Task
JoinExpr
to create key used in join expression (ie, for multi-condition join conditions, create composite key similar to cql)joinNodes
during sql rewriteprojection
w unit testA few enhancements
SELECT COLUMNS
Add unit tests for repeatability and fix.
INSERT INTO cas_table (title, revid, last_modified) VALUES (?, ?, ?) IF NOT EXISTS
DELETE FROM cas_table WHERE title = ? and revid = ? IF last_modified = ?
support the standard CREATE
statements
SOURCE
, TABLE
and create colsexec
query engineGenerators convert AST statements into underlying native query syntax for predicate pushdown.
Implement in elasticsearch and later mongo.
Create a datasource for json
Construction of Nodes directly, not through lexing should be easier
Pos
String
and StringAST
into just String
New
functionsThe SHOW/Describe code is not great. Replace it with query-rewrite, so that show statements become normal Select statements that operate against normal data sources.
TODO
like '%xyz'
add where-clause support for show/describeplan
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.