Giter Club home page Giter Club logo

emacs-libpq's Introduction

An Emacs 25 module for accessing PostgreSQL via the libpq client library.

Using libpq for client connections has various advantages over the wire-protocol speaking pure elisp implementations. For example, it has better performance and supports all features of the protocol like full TLS support and new authentication methods like scram-sha-256.

It doesn’t expose many libpq features yet, but what’s there should be crash-safe no matter what you do in the lisp world. I’ve been using it for three years now for reading mail through my custom Gnus backend without incidents. If you make it crash, please report.

Basic usage

ELISP> (setq *pq* (pq:connectdb "dbname=andreas"))
#<user-ptr ptr=0x55b466c02780 finalizer=0x7f7d50112236>
ELISP> (pq:query *pq* "select version()")
("PostgreSQL 9.6.7 on x86_64-pc-linux-gnu, compiled by gcc (Debian 6.3.0-18) 6.3.0 20170516, 64-bit")
ELISP> (pq:query *pq* "create table local_variables(name text, value text)")
nil
ELISP> (dolist (el (buffer-local-variables))
	 (pq:query *pq* "insert into local_variables values ($1, $2)"
		   (car el) (cdr el)))
nil
ELISP> (pq:query *pq* "select name, length(value) from local_variables where value ~ 'mode'")
(["major-mode" 24]
 ["change-major-mode-hook" 86]
 ["hi-lock-mode-major-mode" 24]
 ["eldoc-mode-major-mode" 24]
 ["font-lock-major-mode" 24]
 ["font-lock-mode-major-mode" 24])

Error Handling

pq raises SQL errors as error signal pq:error. This provides the SQLSTATE error code in an additional string in the error data list. For example, you can reliably catch unique violations like this:

(condition-case err (pq:query *pq* "insert into t values (666)")
  (pq:error
   (if (string= "23505" (nth 2 err))
       (progn
	 (message "Caught a unique violation"))
     ;; re-throw anything else
     (signal (car err) (cdr err)))))

Conversion of data types from SQL to Emacs Lisp

pq converts bigints and numerics your queries return to lisp floats because they don’t fit into a lisp integer. This looses precision on big values. If you need the full precision, cast them to text and use, e.g., calc-eval to do arbitrary precision things with them. All other data types are returned as utf-8 strings.

Conversion of data types from Emacs Lisp to SQL

Strings and the query text itself is converted to utf-8 by the module interface. If this conversion fails, the behavior is undefined by the module interface. If you want to send strings that are not valid utf-8, you need to work around this. For example, I’m using code like the following to store raw bytes into a table with a bytea column:

 (pq:query *con*
   "insert into t (blob) values (decode($1, 'base64'))"
   (base64-encode-string my-random-bytes))

Any non-string parameter to pq:query is turned into an emacs string using prin1-to-string first. This works quite well to store arbitrary lisp data and read it back with read. All other aspects of prin1-to-string apply too. For example, when print-length or print-level are set to non-nil, these would be applied as well.

Notifications

After a =LISTEN= statement, PostgreSQL will deliver notifications asynchronously over the connection. Since the emacs-module interface does not allow for asynchronous callbacks, you have to check for these periodically after a LISTEN statement by calling pq:notifies. Calling it will not cause any traffic on the connection itself.

See the testsuite ./test.el for more implemented features.

https://api.travis-ci.org/anse1/emacs-libpq.svg

emacs-libpq's People

Contributors

anse1 avatar marsam avatar tgbugs avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

emacs-libpq's Issues

"peculiar error"s appear again

In some contexts, SQL errors are still not properly propagated and yield a generic "peculiar error" by the module interface instead. I've not found a recipe to reproduce these yet.

pg:connect: Backend error: FATAL: unsupported frontend protocol 1.0: server supports 2.0 to 3.0

Since PostgreSQL version 10.1 the protocol version 1.0 has been removed, and such is used by pg.el and each package, just as this one.

See the change here:
https://www.postgresql.org/docs/current/static/release-10.html

As protocol 1.0 has been dropped, the pg.el on which this package depends does not work any more with the PostgreSQL latest version 10.1

If you can do anything to handle this problem, please do, as of now, all PostgreSQL programs with Emacs are broken on my side.

Auto-compile `pq-core.c` when compiling/using the package

I pushed to the scratch/pq-compile branch of elpa.git my proposed patch which should let Emacs auomatically compile pq-core.c for the user when the package is installed/compiled or when it's used. Feel free to make it a pull request here if you prefer, of course.

Getting true FALSE instead of nil

If I have column like variable_boolean and I want to know if it is FALSE, the module returns nil, which I understand, however, it does not say if column is with the value FALSE or it is empty. There is particular situation where it is important. I hope you understand me.

Provide support for PostgreSQL arrays

Table:

admin=# \d sal_emp
                   Table "public.sal_emp"
     Column     |   Type    | Collation | Nullable | Default 
----------------+-----------+-----------+----------+---------
 name           | text      |           |          | 
 pay_by_quarter | integer[] |           |          | 
 schedule       | text[]    |           |          | 

Result:

(rcd-sql "SELECT schedule from sal_emp;" *admin*)
("{{meeting,lunch},{training,presentation}}" "{{meeting,lunch},{training,presentation},{Assigned,Jean}}")

In my opinion that is poor for Emacs Lisp to receive just textual representation of PostgreSQL arrays. I was expecting to receive some better structure.

Is it feasible to support PostgreSQL arrays so that Emacs Lisp receives better structure out of it?

progn: SQL error: "ERROR: invalid regular expression: quantifier operand invalid

I do expect that library automatically escapes whatever is necessary when it comes to providing parameters. Here below I get error as "+" is considered quantifier. Am I supposed to escape this before providing it as parameter?

(pq:query cf-db "SELECT commlines_people FROM commlines WHERE commlines_value ~ $1 AND (SELECT addresses_people FROM addresses WHERE addresses_countries = $2) != commlines_people" "+256" 224)

progn: SQL error: "ERROR:  invalid regular expression: quantifier operand invalid

pq:escapeLiteral does it need conn-db?

Does escaping really needs conn-db and why? Should not escaping be independent of connection?

Finally escaping function could be used to generate SQL without having a database connection.

Could you help me? Emacs Bug: Invalid datatype (0x01)

(defun rcd-sql (sql)
  (pq:query *rcd-pg* sql))

(rcd-sql "SELECT a.attname, pg_catalog.format_type(a.atttypid, a.atttypmod), (SELECT substring(pg_catalog.pg_get_expr(d.adbin, d.adrelid) for 128) FROM pg_catalog.pg_attrdef d WHERE d.adrelid = a.attrelid AND d.adnum = a.attnum AND a.atthasdef), a.attnotnull, a.attnum, (SELECT c.collname FROM pg_catalog.pg_collation c, pg_catalog.pg_type t WHERE c.oid = a.attcollation AND t.oid = a.atttypid AND a.attcollation <> t.typcollation) AS attcollation, a.attidentity, NULL AS indexdef, NULL AS attfdwoptions, a.attstorage, CASE WHEN a.attstattarget=-1 THEN NULL ELSE a.attstattarget END AS attstattarget, pg_catalog.col_description(a.attrelid, a.attnum) FROM pg_catalog.pg_attribute a WHERE a.attrelid = '33748' AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum")

(["tlds_id" "integer" "nextval('tlds_tlds_id_seq'::regclass)" "" 1 #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "p" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "ID"] ["tlds_tld" "text" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "" 2 #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "x" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "TLD"] ["tlds_nic" "integer" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> 3 #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "p" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "NIC Company"] ["tlds_description" "text" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> 4 #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "x" #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug> "Description"] . #<EMACS BUG: INVALID DATATYPE (0x01) Save your buffers immediately and please report this bug>)

     attname      | format_type |               substring               | attnotnull | attnum | attcollation | attidentity | indexdef | attfdwoptions | attstorage | attstattarget | col_description 
------------------+-------------+---------------------------------------+------------+--------+--------------+-------------+----------+---------------+------------+---------------+-----------------
 tlds_id          | integer     | nextval('tlds_tlds_id_seq'::regclass) | t          |      1 |              |             |          |               | p          |               | ID
 tlds_tld         | text        |                                       | t          |      2 |              |             |          |               | x          |               | TLD
 tlds_nic         | integer     |                                       | f          |      3 |              |             |          |               | p          |               | NIC Company
 tlds_description | text        |                                       | f          |      4 |              |             |          |               | x          |               | Description
(4 rows)


Basically, I have git upgraded to newest development Emacs version. Recompiled your module which worked so well over last months. And now I cannot run it, as I am crashing whole emacs by the above SQL query.

Do you know maybe, do you have some clue, how to avoid the crash?

(pq:query db "SELECT $1" 100) returns string ⇒ ("100") instead of integer

I am expecting that

(pq:query db "SELECT $1" 100) ⇒ ("100")

returns integer instead of a string. I suspect bug here.

Only way it would work is:

(pq:query db "SELECT $1::integer" 100) ⇒ (100)

which makes everything cumbersome. It forces me to use function format which is then unsafe for users' input. I would rather like to switch to using pq:query parameters, but this problem is here.

Do you recommend using handle in each function using database?

At first I have made libraries that connect and fetch and handle was &optional. If there was no handle standard *db* handle was used. Programs could access one database.

Then I have developed separate Emacs package that uses the same database but due to its nature could also use separate database and other databases outside of my local computer.

As then I had 2 programs connecting to local database and complexities arrive in future when using multiple databases in the same time, I am of opinion that each call to function that uses database should also transfer the handle, so that handle is not just global variable.

Do you agree on this?

Handle was here optional and it worked well until I started making two connections to database:

(defun rcd-sql (sql &optional handle)
  "Sends SQL queries to PostgreSQL database and returns results"
  (let ((handle (if handle handle *cf*)))
    (condition-case err
	(pq:query handle sql)
      (error
       (if (string-match "^ERROR:  syntax error" (cdr err))
	   (progn
	     (if (fboundp 'speak) (speak (cdr err)))
	     (message (cdr err)))
	 ;; re-throw
	 (signal (car err) (cdr err)))))))

(defun rcd-sql-list (sql &optional handle)
  "Returns list of lists instead of vectors"
  (let ((list '()))
    (dolist (i (apply 'rcd-sql (list sql handle)) (reverse list))
      (cond ((eq (type-of i) 'vector) (push (append i '()) list))
	    (t (push i list))))))

(defun rcd-sql-first (sql &optional handle)
  "Returns first entry from SQL query"
  (car (apply 'rcd-sql (list sql handle))))

My opinion is that I should not make it optional but require handle each time. Is that what you also recommend?

Queries not interruptible from Emacs with Ctrl-G

I have noticed that queries that take long time are not interruptible by Emacs. Maybe something can be done on that?

I had to stop server to interrupt it.

Maybe there is some function that can be implemented to send some signal to interrupt it?

Error handling

I should be able to get the database erorr programmatically and not just enter debugger on something like

(pq:query *rcd-pg* "SEjnjn")

so how do I do it?

I cannot find description for these optional parameters:

pq:query is a module function.

(pq:query ARG1 ARG2 &optional ARG3 ARG4 ARG5 ARG6
ARG7 ARG8 ARG9 ARG10 ARG11 ARG12 ARG13 ARG14)

Execute QUERY on CONNECTION with optional PARAMETERS.

I was thinking there is some error handling option, I cannot find it. Please help. Imagine editing a table field and it should be unique, and database is raising error. I think I should be able to capture, understand error and do further programming based on that.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.