staab / janet-pqutils Goto Github PK
View Code? Open in Web Editor NEWA utility library for Janet and janet-pq.
License: MIT License
A utility library for Janet and janet-pq.
License: MIT License
The current "lazy" implementation only lazily grabs results from libpq's result object which lives in memory, not from the server, so it doesn't actually help anything. In order to do that, we need to use cursors.
With different nesting modes:
For stuff not handled directly in core.c, we should add a multimethod that can be populated with user-defined functionality for unpacking rows. We should run all results through an unpack function by default, but allow turning it off with a dynamic binding.
I think when reading a serial or big serial for example, it might be better to use s64 than a plain janet number for correctness sake.
Matching on strings is a performance problem since it's in an inner loop. We should convert this to some constant-time match functionality in C for custom types, and for built-in types bring in pg_types.c. Once we do that, #7 may just go away.
For additional background on getting and installing libpq, check out this SO answer.
This is just a dumping place for my progress in compiling/linking my code with postgres. Unless otherwise specified, I'm using jpm test
to trigger the errors.
After getting the right -I
flag to include libpq-fe.h
, I started getting:
creating static library build/pg.a...
running test/pg.janet ...
error: could not load native build/pg.so: dlopen(build/pg.so, 2): Symbol not found: _PQconnectdb
Referenced from: build/pg.so
Expected in: flat namespace
in build/pg.so
in native
in <anonymous> [boot.janet] on line 1934, column 26
in require [boot.janet] on line 1955, column 16
in import* [boot.janet] on line 1967, column 15
in _thunk [test/pg.janet] (tailcall) on line 1, column 1
I was able to get that error to go away by including all the c
files from libpq:
(declare-native
:name "pg"
:source @["staab.pg/pg.c"
"postgresql/src/interfaces/libpq/fe-auth-scram.c"
"postgresql/src/interfaces/libpq/fe-auth.c"
"postgresql/src/interfaces/libpq/fe-connect.c"
"postgresql/src/interfaces/libpq/fe-exec.c"
"postgresql/src/interfaces/libpq/fe-gssapi-common.c"
"postgresql/src/interfaces/libpq/fe-lobj.c"
"postgresql/src/interfaces/libpq/fe-misc.c"
"postgresql/src/interfaces/libpq/fe-print.c"
"postgresql/src/interfaces/libpq/fe-protocol2.c"
"postgresql/src/interfaces/libpq/fe-protocol3.c"
"postgresql/src/interfaces/libpq/fe-secure-common.c"
"postgresql/src/interfaces/libpq/fe-secure-gssapi.c"
"postgresql/src/interfaces/libpq/fe-secure-openssl.c"
"postgresql/src/interfaces/libpq/fe-secure.c"
"postgresql/src/interfaces/libpq/legacy-pqsignal.c"
"postgresql/src/interfaces/libpq/libpq-events.c"
"postgresql/src/interfaces/libpq/pqexpbuffer.c"
"postgresql/src/interfaces/libpq/pthread-win32.c"
"postgresql/src/interfaces/libpq/win32.c"]
:cflags @["-Ipostgresql/src/interfaces/libpq"
"-Ipostgresql/src/include"])
I then got:
compiling build/postgresql___src___interfaces___libpq___fe-connect.o...
postgresql/src/interfaces/libpq/fe-connect.c:32:10: fatal error: 'pg_config_paths.h' file not found
#include "pg_config_paths.h"
^~~~~~~~~~~~~~~~~~~
1 error generated.
Which I was able to get rid of by running cd postgresql/src/interfaces/libpq && make ../../../src/port/pg_config_paths.h
and adding -Ipostgresql/src/port
to my :cflags
section.
After looking into it a bit more, that seemed like a bandaid, so I ended up installing the client libraries to postgresql/build
, using:
cd postgresql
rm -rf build
make distclean
./configure --prefix=$(pwd)/build
make -C src/bin install
make -C src/include install
make -C src/interfaces install
make -C doc install
This let me to change my project.janet to:
(declare-native
:name "pg"
:source @["staab.pg/pg.c"
"postgresql/src/interfaces/libpq/fe-auth-scram.c"
"postgresql/src/interfaces/libpq/fe-auth.c"
"postgresql/src/interfaces/libpq/fe-connect.c"
"postgresql/src/interfaces/libpq/fe-exec.c"
"postgresql/src/interfaces/libpq/fe-gssapi-common.c"
"postgresql/src/interfaces/libpq/fe-lobj.c"
"postgresql/src/interfaces/libpq/fe-misc.c"
"postgresql/src/interfaces/libpq/fe-print.c"
"postgresql/src/interfaces/libpq/fe-protocol2.c"
"postgresql/src/interfaces/libpq/fe-protocol3.c"
"postgresql/src/interfaces/libpq/fe-secure-common.c"
"postgresql/src/interfaces/libpq/fe-secure-gssapi.c"
"postgresql/src/interfaces/libpq/fe-secure-openssl.c"
"postgresql/src/interfaces/libpq/fe-secure.c"
"postgresql/src/interfaces/libpq/legacy-pqsignal.c"
"postgresql/src/interfaces/libpq/libpq-events.c"
"postgresql/src/interfaces/libpq/pqexpbuffer.c"
"postgresql/src/interfaces/libpq/pthread-win32.c"
"postgresql/src/interfaces/libpq/win32.c"]
:cflags @["-Ipostgresql/build/include"
"-Ipostgresql/build/include/libpq"
"-Ipostgresql/build/include/internal"
"-Ipostgresql/build/include/internal/libpq"
"-Ipostgresql/build/include/server"
"-Lpostgresql/build/lib"])
At this point, I'm now getting a bunch of errors having to do with types that aren't defined:
compiling build/postgresql___src___interfaces___libpq___fe-gssapi-common.o...
clang: warning: argument unused during compilation: '-Lpostgresql/build/lib' [-Wunused-command-line-argument]
postgresql/src/interfaces/libpq/fe-gssapi-common.c:26:6: error: unknown type name 'OM_uint32'; did you mean 'uint32'?
OM_uint32 stat, int type)
^~~~~~~~~
uint32
postgresql/build/include/internal/c.h:359:22: note: 'uint32' declared here
typedef unsigned int uint32; /* == 32 bits */
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:28:2: error: unknown type name 'OM_uint32'; did you mean 'uint32'?
OM_uint32 lmin_s;
^~~~~~~~~
uint32
postgresql/build/include/internal/c.h:359:22: note: 'uint32' declared here
typedef unsigned int uint32; /* == 32 bits */
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:29:2: error: use of undeclared identifier 'gss_buffer_desc'
gss_buffer_desc lmsg;
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:30:2: error: unknown type name 'OM_uint32'; did you mean 'uint32'?
OM_uint32 msg_ctx = 0;
^~~~~~~~~
uint32
postgresql/build/include/internal/c.h:359:22: note: 'uint32' declared here
typedef unsigned int uint32; /* == 32 bits */
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:34:3: warning: implicit declaration of function
'gss_display_status' is invalid in C99 [-Wimplicit-function-declaration]
gss_display_status(&lmin_s, stat, type,
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:35:10: error: use of undeclared identifier 'GSS_C_NO_OID'
GSS_C_NO_OID, &msg_ctx, &lmsg);
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:35:35: error: use of undeclared identifier 'lmsg'
GSS_C_NO_OID, &msg_ctx, &lmsg);
(several more lines)
If I search postgresql for e.g., OM_uint32
, I get a lot of uses of it, but no definition, so I wonder if it's a preprocessor thing. Maybe I need to reference different c
files, or use a -L
option?
In summary, I have no idea what I'm doing. I'll keep this updated as I work on it.
Three layers:
{:select [:x :y] :where [{:x := 1}]}
, which allows us to target clauses by type when compiling the query. This might be redundant and unnecessary given representation number 2.[{:t :select :v :x} {:t :select :v :y} {:t :where :k :x :op := :v 1}]
. This allows the user to easily compose/modify queries, since a query is a (recursive, with subqueries) collection of clauses.(q (select x y z) (where= x 1))
. The q
function should compile directly to the collection representation above, flattening values returned from query building functions/macros into a single list. This should recur into subqueries as well.A compile
function should take any of the three representations above and turn it into a string, escaping literals and identifiers as necessary. If the input is a string already, it should just return it.
Execution functions shouldn't know about this layer, but should expose a configuration option for preparing queries, which can be configured by setting it to compile
.
We catch an error for retrieving next rows, but we should check the message.
As far as I can tell, the most general purpose way to return results would be to create a fiber and yield from within it. This would require creating a result wrapper that would call PQclear
when garbage collected. It would also require figuring out how the heck to yield from a c function ๐ค.
Using e.g., (base16/encode (os/cryptorand 16))
Both when collecting results and when passing parameters (see #3 for that).
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==11071==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x7f94ecbfb240 (pc 0x7fff633066f2 bp 0x7ffeea954ae0 sp 0x7ffeea954ae0 T7799094)
==11071==The signal is caused by a READ memory access.
#0 0x7fff633066f1 in _platform_strlen (libsystem_platform.dylib:x86_64+0x16f1)
#1 0x1052d2c02 in janet_cstring (janet:x86_64+0x100028c02)
#2 0x10532c1b3 in populate_oids (core.so:x86_64+0x11b3)
#3 0x10532c74f in cfun_exec (core.so:x86_64+0x174f)
#4 0x1052dcea3 in run_vm (janet:x86_64+0x100032ea3)
#5 0x1052ddbdd in janet_continue (janet:x86_64+0x100033bdd)
#6 0x1052dd14e in run_vm (janet:x86_64+0x10003314e)
#7 0x1052ddbdd in janet_continue (janet:x86_64+0x100033bdd)
#8 0x1052cf981 in janet_dobytes (janet:x86_64+0x100025981)
#9 0x1052df7f4 in main (janet:x86_64+0x1000357f4)
#10 0x7fff6311e3d4 in start (libdyld.dylib:x86_64+0x163d4)
==11071==Register values:
rax = 0x415ad6da00000000 rbx = 0x415ad6da00000000 rcx = 0x00007f94ecbfb240 rdx = 0x00007f94ecbfb240
rdi = 0x00007f94ecbfb240 rsi = 0x00007f94ec400000 rbp = 0x00007ffeea954ae0 rsp = 0x00007ffeea954ae0
r8 = 0x0000000000000006 r9 = 0x0000000000000000 r10 = 0x0000000000000007 r11 = 0x0000000000000007
r12 = 0x00007f94ec634ec0 r13 = 0x0000000000000011 r14 = 0x00007f94ecbfb240 r15 = 0x00007f94ec502d01
UndefinedBehaviorSanitizer can not provide additional info.
==11071==ABORTING
We're already using PQexecParams
, just gotta pass params through.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.