Giter Club home page Giter Club logo

chicken-lmdb's Introduction

chicken-lmdb

Chicken Scheme bindings for the Lightning Memory-Mapped Database Manager (LMDB) (http://symas.com/mdb/doc/index.html) database management library.

Library procedures

(db-open filename [key: enckey] [mapsize: size]) Opens or creates LMDB database with optional encryption key and map size.

(db-close db) Closes LMDB database handle.

(db-begin db [dbname: dbname] [flags: 0]) Begins LMDB transaction with optional database name and flags returned by db-flags. Commits and ends LMDB transaction.

(db-flags flag-name ...) Returns a bit mask that represents LMDB environment flags. The flag name is one of the following keywords:

  • #:fixed-map : mmap at a fixed address
  • #:no-subdirectory : no environment directory
  • #:read-only : read only
  • #:write-map : use writable mmap
  • #:no-meta-sync : don't fsync metapage after commit
  • #:no-sync : don't fsync after commit
  • #:map-async : use asynchronous sync with #:write-map is used
  • #:no-lock : don't do locking
  • #:no-read-ahead : don't do readahead

(db-abort db) Aborts LMDB transaction.

(db-delete db key) Removes a key from the database.

(db-delete-database dbname) Deletes LMDB database.

(db-ref db key) Looks up key in database.

(db-set! db key value) Sets a key-value pair in the database.

(db-count db) Returns number of key-value pairs in database.

(db-keys db) Returns a list of database keys.

(db-values db) Returns a list of database values.

(db-key-len m) Returns the length of the current key.

(db-value-len m) Returns the length of the current value.

(db-key m buf) Copies the current key to the specified blob.

(db-value m buf) Copies the current value to the specified blob.

(db-fold f init db) Fold over the keys and values in the database.

(db-for-each f db) Iterate over the keys and values in the database.

(hash-table->db t dbfile [enckey]) Saves SRFI-69 hash table to database.

(db->hash-table dbfile [enckey]) Load database into SRFI-69 hash table.

Examples

;; lmdb encrypted key-value creation and lookup

(let* ((fname (make-pathname "." "mydb.mdb"))
       (keys (list "k1" 'k2 '(k3)))
       (values (list 'one 2 "three"))
       (cryptokey (string->blob "1234"))
       (mm (db-open fname key: cryptokey)))
  (db-begin mm)
  (let loop ((ks keys) (vs values))
    (if (> (length ks) 0) 
        (begin
          (db-set! mm (string->blob (->string (car ks))) (string->blob (->string (car vs))))
          (loop (cdr ks) (cdr vs)))))
  (db-end mm)
  (db-begin mm)
  (let ((res (let loop ((ks keys) (vs values))
               (if (= (length ks) 0) #t
                   (let ((v (db-ref mm (string->blob (->string (car ks))))))
                     (if (not (equal? (string->blob (->string (car vs))) v))  #f
                         (loop (cdr ks) (cdr vs)))))))
        )
    (db-end mm)
    (db-close mm)
    res)
  )
;; lmdb readonly test

(let* ((fname (make-pathname "." "unittest.mdb")))
       (db-delete-database fname)
  (let ((mm (db-open fname maxdbs: 2)))
       (db-begin mm)
		;; set foo
		(db-set! mm (string->blob "foo") (string->blob "one"))
		(db-end mm)
        ;; reopen as readonly
		(db-begin mm flags: (db-flags #:read-only))
		;; foo is still set
		(test (string->blob "one")
		      (db-ref mm (string->blob "foo")))
		;; cannot set bar
		(test 'unknown
		      (condition-case (db-set! mm (string->blob "bar") (string->blob "two"))
			((exn lmdb unknown) 'unknown)))
                (db-close mm)))

chicken-lmdb's People

Contributors

alicemaz avatar iraikov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

caolan alicemaz

chicken-lmdb's Issues

Separate encryption layer

For my own project, I don't need the encryption of keys/values and would like to avoid the rabbit egg dependency. How do you feel about me splitting that out into a wrapper module, something like lmdb-rabbit?

My vague plan would be to keep most the code the same and use a functor module to make the key/value encoding a parameter (which defaults to identity).

too many failed db-ref calls break the gc(?)

this one is perplexing to me, was hoping you could reproduce and might have some insight. minimal example:

(module testy *

(import scheme)
(import chicken.base)
(import chicken.format)
(import chicken.condition)
(import lmdb-ht)

(define (loopy db n)
  (printf "MAZ begin ~A\n" n)
  (db-begin db)
  (printf "MAZ catching bad ref\n")
  (condition-case
    (db-ref db #${1234567890})
    ((exn) (printf "MAZ caught it\n")))
  (printf "MAZ end\n")
  (db-end db)
  (if (= n 0) '() (loopy db (- n 1))))


(printf "MAZ open\n")
(define db (db-open "testy-db"))
(loopy db 10000)
(printf "MAZ close\n")
(db-close db)

)

so when I run this, what happens is it blows through the first ~1200ish calls to db-ref no problem but then the program freezes on a db-ref invocation. here's the top few frames from a representative gdb backtrace:

Starting program: /home/alice/work/misc/lmdb-test/.pontiff-work/testy
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Program received signal SIGINT, Interrupt.
0x00007ffff7ebf5b0 in C_reclaim () from /usr/lib/libchicken.so.11
#0  0x00007ffff7ebf5b0 in C_reclaim () from /usr/lib/libchicken.so.11
#1  0x00007ffff7ec0324 in C_save_and_reclaim () from /usr/lib/libchicken.so.11
#2  0x00007ffff7cb3721 in ?? () from /usr/lib/libchicken.so.11
#3  0x00007ffff7ebd16b in C_callback () from /usr/lib/libchicken.so.11
#4  0x00007ffff7ebd201 in C_callback_wrapper () from /usr/lib/libchicken.so.11
#5  0x00007ffff78e3cd5 in chicken_lmdb_exception () from /usr/lib/chicken/11/lmdb-ht.so
#6  0x00007ffff78e41b6 in _mdb_read () from /usr/lib/chicken/11/lmdb-ht.so
#7  0x00007ffff78e4215 in stub181 () from /usr/lib/chicken/11/lmdb-ht.so
#8  0x00007ffff78e2262 in f_1026 () from /usr/lib/chicken/11/lmdb-ht.so
#9  0x00007ffff7eb8d7d in ?? () from /usr/lib/libchicken.so.11
#10 0x00007ffff7ebd16b in C_callback () from /usr/lib/libchicken.so.11
#11 0x00007ffff7ebd201 in C_callback_wrapper () from /usr/lib/libchicken.so.11
#12 0x00007ffff78e3cd5 in chicken_lmdb_exception () from /usr/lib/chicken/11/lmdb-ht.so
#13 0x00007ffff78e41b6 in _mdb_read () from /usr/lib/chicken/11/lmdb-ht.so

playing around with compiler options doesn't change much, though I have also seen it freeze on the longjmp call too. neither -fsanitize=undefined nor valgrind memcheck indicate anything amiss. behavior the same on clang and gcc. other experimental results...

  • still freezes with db-begin and db-end outside the loop
  • still freezes with db-end changed to db-abort
  • finishes flawlessly if the key does exist
  • finishes flawlessly with db-ref changed to srfi-18 raise
  • the effect is cumulative somehow... if you have a case where it freezes on say iteration 1000, if you insert the record immediately before the db-ref call, it can proceed through arbitrarily many more. but then if you delete the record on say iteration 5000 it will immediately freeze on the next db-ref

as I said this one is very strange to me and probably requires more familiarity with the runtime than I have at present, though I'll try to look at it in more detail later this week

Rename `lmdb-delete` and add support for deleting keys?

The lmdb-delete procedure deletes the files associated with a database. I'd like to suggest we use lmdb-delete! to delete keys from a database instead, as this would be in line with the srfi-69 naming style used elsewhere. The original lmdb-delete might be renamed to lmdb-destroy or lmdb-delete-database or similar.

If you agree with this change I'm happy to make a start on it. We currently don't have a way to delete keys.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.