Giter Club home page Giter Club logo

rust-rocksdb's Introduction

rust-rocksdb

RocksDB build crates.io documentation license Gitter chat rust 1.70.0 required

GitHub commits (since latest release)

Requirements

  • Clang and LLVM

Contributing

Feedback and pull requests welcome! If a particular feature of RocksDB is important to you, please let me know by opening an issue, and I'll prioritize it.

Usage

This binding is statically linked with a specific version of RocksDB. If you want to build it yourself, make sure you've also cloned the RocksDB and compression submodules:

git submodule update --init --recursive

Compression Support

By default, support for Snappy, LZ4, Zstd, Zlib, and Bzip2 compression is enabled through crate features. If support for all of these compression algorithms is not needed, default features can be disabled and specific compression algorithms can be enabled. For example, to enable only LZ4 compression support, make these changes to your Cargo.toml:

[dependencies.rocksdb]
default-features = false
features = ["lz4"]

Multithreaded ColumnFamily alternation

RocksDB allows column families to be created and dropped from multiple threads concurrently, but this crate doesn't allow it by default for compatibility. If you need to modify column families concurrently, enable the crate feature multi-threaded-cf, which makes this binding's data structures use RwLock by default. Alternatively, you can directly create DBWithThreadMode<MultiThreaded> without enabling the crate feature.

Switch between /MT or /MD run time library (Only for Windows)

The feature mt_static will request the library to be built with /MT flag, which results in library using the static version of the run-time library. This can be useful in case there's a conflict in the dependecy tree between different run-time versions.

rust-rocksdb's People

Contributors

akrylysov avatar aleksuss avatar alexreg avatar andreyg avatar busyjay avatar calavera avatar derekdreery avatar dgrnbrg avatar ekmartin avatar garyttierney avatar isynaptic avatar kaedroho avatar linxgnu avatar methyl avatar mina86 avatar niklasf avatar ordian avatar pavel-mukhanov avatar rnarubin avatar rohitjoshi avatar romanz avatar rrichardson avatar siddontang avatar spacejam avatar stanislav-tkach avatar valeriansaliou avatar vmx avatar xxuejie avatar zaidoon1 avatar zhangjinpeng87 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rust-rocksdb's Issues

Are `&mut self` methods necessary?

Since all these methods don't seem to be mutating any Rust data in self, I think we can get away with just &self parameters. This is advantageous when it comes to using a DB value in a multithreaded environment, since you can then get proper read-write concurrency without a Mutex/RwLock.

Custom Comparators

Hello, I'm interested in custom comparator support. I would like to be able to specify keys as lists of sorted keys. This way, I could do prefix lookups on multiple levels of variable-length prefixes. This indexing scheme is critical for building EAV database indices like Datomic (a functional DB) use.

EXC_BAD_ACCESS with iterator

I seem to be hitting a weird corner case: I created a DB with 10000 keys and then trying to iterate over them. Everything works when I create iterator directly in a function, however when I create a wrapper which returns DBIterator, I get EXC_BAD_ACCESS when trying to dump the db with returned iterator.

Here's a minimal demo which reproduces the issue:

https://github.com/nitingupta910/iter_demo
(checkout with_upstream branch)

Note that output stops after printing just 320 keys while 10000 keys where inserted. Running it under gdb shows EXC_BAD_ACCESS as the reason for early termination.

Uncomment call to db.iterator() in dump() and the bug disappears.

Can only store utf8 values using this crate?

I can write arbitrary byte values to the rocksdb store using the put() method but when I call get() I get a DBVector for the result, and I can see it has a to_utf8() method but I can't see how to get the raw bytes (my data is not utf8).

I'm still new to Rust and I tried calling the deref() method too, but couldn't see how to get that to work.

Would appreciate some pointers.

WriteBatch and Iterator Support

Hello, I'm very interested in adding support for these features. I've just started learning Rust, but I'd be happy to help if you could outline what it would take for me.

These features together add read and write transactions to the DB, which enables new use cases.

Messy tests

When a test fails, it's a bit hard to see what went wrong, you just get a panic for unwrapping a result.

Suggestion: split larger tests into smaller tests, explicitly check error conditions.

I am happy to have a crack at it if you think it's a good idea.

add support for delete_range

would be cool to have the support for this feature to be able to delete in a range directly instead of launching an iterator then delete.

Doesn't work for me on OS X

iMac:noise damienkatz$ cargo install rocksdb --verbose
Updating registry https://github.com/rust-lang/crates.io-index
Fresh libc v0.2.12
Compiling rocksdb v0.3.5
Running rustc /Users/damienkatz/.cargo/registry/src/github.com-88ac128001ac3a9a/rocksdb-0.3.5/src/main.rs --crate-name rocksdb --crate-type bin -C opt-level=3 --cfg feature=\"default\" --out-dir /Users/damienkatz/noise/target-install/release --emit=dep-info,link -L dependency=/Users/damienkatz/noise/target-install/release -L dependency=/Users/damienkatz/noise/target-install/release/deps --extern libc=/Users/damienkatz/noise/target-install/release/deps/liblibc-d9dba2869ce64308.rlib --extern rocksdb=/Users/damienkatz/noise/target-install/release/librocksdb.rlib
error: linking with cc failed: exit code: 1
note: "cc" "-m64" "-L" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib" "/Users/damienkatz/noise/target-install/release/rocksdb.0.o" "-o" "/Users/damienkatz/noise/target-install/release/rocksdb" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/Users/damienkatz/noise/target-install/release" "-L" "/Users/damienkatz/noise/target-install/release/deps" "-L" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib" "/Users/damienkatz/noise/target-install/release/librocksdb.rlib" "/Users/damienkatz/noise/target-install/release/deps/liblibc-d9dba2869ce64308.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libstd-d16b8f0e.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libcollections-d16b8f0e.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/librustc_unicode-d16b8f0e.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/librand-d16b8f0e.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/liballoc-d16b8f0e.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/liballoc_jemalloc-d16b8f0e.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/liblibc-d16b8f0e.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libcore-d16b8f0e.rlib" "-l" "rocksdb" "-l" "System" "-l" "pthread" "-l" "c" "-l" "m" "-l" "compiler-rt"
note: ld: library not found for -lrocksdb
clang: error: linker command failed with exit code 1 (use -v to see invocation)

error: aborting due to previous error
error: failed to compile rocksdb v0.3.5, intermediate artifacts can be found at /Users/damienkatz/noise/target-install

Caused by:
Could not compile rocksdb.

Caused by:
Process didn't exit successfully: rustc /Users/damienkatz/.cargo/registry/src/github.com-88ac128001ac3a9a/rocksdb-0.3.5/src/main.rs --crate-name rocksdb --crate-type bin -C opt-level=3 --cfg feature="default" --out-dir /Users/damienkatz/noise/target-install/release --emit=dep-info,link -L dependency=/Users/damienkatz/noise/target-install/release -L dependency=/Users/damienkatz/noise/target-install/release/deps --extern libc=/Users/damienkatz/noise/target-install/release/deps/liblibc-d9dba2869ce64308.rlib --extern rocksdb=/Users/damienkatz/noise/target-install/release/librocksdb.rlib (exit code: 101)

[Windows] Unicode path issue

On Windows, creating a database with unicode characters in the filename creates two directories: an empty one with the correct name, and one with the actual database in it but with a mangled name.

For example:

DB::open_default("_rust_rocksdb_u͔n͈̰̎i̙̮͚̦c͚̉o̼̩̰͗d͔̆̓ͥé");

Creates two folders:

  • _rust_rocksdb_u͔n͈̰̎i̙̮͚̦c͚̉o̼̩̰͗d͔̆̓ͥé (empty)
  • _rust_rocksdb_uÍ”n͈̰̎i̙̮͚̦c͚̉o̼̩̰͗d͔̆̓ͥeÌ (with data inside it)

Linux works as expected (creates one folder with correct name)

streamlining DB api with regard to CFs

This is a bit of a follow-on from #144 but is different enough in scope

The native RocksDB column family API is a bit clunky and seems like an after thought. If users of rust-rocksdb prefer it this way, then that's fine, you can stop reading now.

If you're still with me, I am thinking that we can remove all PIGMED operations and remove the *_cf. The way we do this is make ColumnFamily the source for PIGMED operations. Or, we can shorten "ColumnFamily" to "Table" . So the API would look something like :

DB::open() -> DBResult   //which would enumerate and use all CFs by default
DB.get_table(cf_name: string) -> TableResult . // it would fetch the named table if it exists
DB.get_default_table() -> TableResult // if the user is living in the dark ages, use the default table

Table.put(key, val) 
Table.get(key)...
...

I have a crate rocks-cache that implements this API and it works quite nicely. We've been using it at work for a bit.

Make ffi public?

I'd like to propose re-exporting the FFI bindings through the rocksdb crate to allow users access the underlying bindings without having to import the rocksdb-sys crate directly (this could cause version mismatches when the rocksdb crate is updated later. It's not hard to resolve, but would be ideal for devlopers to no have to think about this).

  • Make the ffi module public (I, obviously, could just use rocksdb-sys directly, but I think it's nicer to access it through rocksdb::ffi)
  • Make db::ReadOptions public
  • Add .inner() getter methods to all the types that have an .inner attribute. Eg, DB, Snapshot, DBIterator, etc.

What do you think?

better testing

Most of the major features have functional tests which occasionally blow up when misusing pointers, but we need to expand coverage to other parts of the codebase so that we can improve the confidence gained when linking against new versions of RocksDB down the road.

Cannot use snapshot DbIterators across threads

Hit this today, attempting to query database and find matching elements using a snapshot DbIterator on one thread, then use DbIterator from same snapshot on another thread. Because of the lifetime requirements it's not possible.

Potential solution: use std::sync::Arc to hold references to snapshot or inner members. Each DbIterator from a single snapshot will also hold a the same arc references.

rustdocs

While the README.md contains a nice high-level overview, we may want to add rustdoc style documentation to reduce the time some people will spend fighting their compiler while trying this out for the first time.

add support for prefix seek

This feature has been requested, and has a ton of different use cases in higher-level database abstractions built on top of this.

Can't compile rust-rocksdb

Followed the instructions but had the following error when compiling:

$ cargo build
   Compiling rocksdb v0.0.8
/home/ferris/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:168:27: 168:36 error: type mismatch resolving `<&[u8] as core::iter::IntoIterator>::Item == u8`:
 expected &-ptr,
    found u8 [E0271]
/home/ferris/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:168         Some(v) => result.extend(v),
                                                                                                                               ^~~~~~~~~
/home/ferris/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:168:27: 168:36 help: run `rustc --explain E0271` to see a detailed explanation
/home/ferris/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:172:16: 172:26 error: type mismatch resolving `<&[u8] as core::iter::IntoIterator>::Item == u8`:
 expected &-ptr,
    found u8 [E0271]
/home/ferris/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:172         result.extend(op);
                                                                                                                    ^~~~~~~~~~
note: in expansion of for loop expansion
/home/ferris/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:171:5: 173:6 note: expansion site
/home/ferris/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:172:16: 172:26 help: run `rustc --explain E0271` to see a detailed explanation
error: aborting due to 2 previous errors
Could not compile `rocksdb`.

Any idea what the issue is? I'm happy to do a PR and fix it if needed :)

TxnDB : Transactions via rocksdb_transaction_t

Transaction functionality is exposed in rocksdb via a parallel implementation of the rocksdb_t called rocksdb_transaction_t . Now that we have bindgen (#143) we get access to it for "free".

First, for brevity, I propose a new acronym that is the RocksDB parallel of CRUD, I call it
PIGMED , that's Put Iterate MErge Delete. Not catchy, I know.

The question is how to best expose the Transaction functionality while :

  1. Not duplicating the entire DB impl for TxnDB
  2. Maintaining type safety.

There are quite a few new methods for TxnDb that would need to be wrapped, some of them might make sense as guards or even closures.

I don't think there is a way to unify the PIGMED operations of DB and TxnDB while retaining type safety or even ergonomics. But there isn't actually a ton of PIGMED functionality in DB.

I propose we move all of the utility/environment functionality, that is anything that doesn't require a rocksdb_t into a separate utility module or modules (e.g. rocksdb::cf::create for rocksdb_create_cf)

We then slim down DB to just the PIGMED operations.

My hope is that we would be able to leverage the same Iterator functionality. There is a rocksdb_transaction_create_iterator but the iterator_t is the same.

Thoughts?

32-bit support

How about backporting the changes from here?

Currently rust-rocksdb doesn't build on 32-bit:

src/rocksdb_options.rs:76:69: 76:73 error: mismatched types:
 expected `u32`,
    found `u64` [E0308]
src/rocksdb_options.rs:76                                                                     size);
                                                                                              ^~~~
src/rocksdb_options.rs:76:69: 76:73 help: run `rustc --explain E0308` to see a detailed explanation
src/rocksdb_options.rs:250:64: 250:68 error: mismatched types:
 expected `u64`,
    found `u32` [E0308]
src/rocksdb_options.rs:250                                                                size);
                                                                                          ^~~~
src/rocksdb_options.rs:250:64: 250:68 help: run `rustc --explain E0308` to see a detailed explanation
src/rocksdb_options.rs:76:69: 76:73 error: mismatched types:
 expected `u32`,
    found `u64` [E0308]
src/rocksdb_options.rs:76                                                                     size);
                                                                                              ^~~~
src/rocksdb_options.rs:76:69: 76:73 help: run `rustc --explain E0308` to see a detailed explanation
src/rocksdb_options.rs:250:64: 250:68 error: mismatched types:
 expected `u64`,
    found `u32` [E0308]
src/rocksdb_options.rs:250                                                                size);
                                                                                          ^~~~
src/rocksdb_options.rs:250:64: 250:68 help: run `rustc --explain E0308` to see a detailed explanation
error: aborting due to 2 previous errors

Using casts locally works OK but is not the correct cross-platform solution.

32-bit build fails

On i686 there's a mismatched types failure:

  Compiling rocksdb v0.3.0
/tmp/.cargo/registry/src/github.com-48ad6e4054423464/rocksdb-0.3.0/src/rocksdb_options.rs:65:69: 65:73 error: mismatched types:
 expected `u32`,
    found `u64` [E0308]
/tmp/.cargo/registry/src/github.com-48ad6e4054423464/rocksdb-0.3.0/src/rocksdb_options.rs:65                                                                     size);
                                                                                                                                                                 ^~~~
/tmp/.cargo/registry/src/github.com-48ad6e4054423464/rocksdb-0.3.0/src/rocksdb_options.rs:65:69: 65:73 help: run `rustc --explain E0308` to see a detailed explanation
/tmp/.cargo/registry/src/github.com-48ad6e4054423464/rocksdb-0.3.0/src/rocksdb_options.rs:234:64: 234:68 error: mismatched types:
 expected `u64`,
    found `u32` [E0308]
/tmp/.cargo/registry/src/github.com-48ad6e4054423464/rocksdb-0.3.0/src/rocksdb_options.rs:234                                                                size);
                                                                                                                                                             ^~~~
/tmp/.cargo/registry/src/github.com-48ad6e4054423464/rocksdb-0.3.0/src/rocksdb_options.rs:234:64: 234:68 help: run `rustc --explain E0308` to see a detailed explanation
error: aborting due to 2 previous errors
Build failed, waiting for other jobs to finish...
Could not compile `rocksdb`.

Naturally, ARM is affected too.

Segfaults when attempting to create column family that already exits

System:

$ uname -a
Linux 4.2.5-1-ARCH #1 SMP PREEMPT Tue Oct 27 08:13:28 CET 2015 x86_64 GNU/Linux
$ pacman -Q rocksdb
rocksdb 3.13-1

Code:

let mut iter = try!(db.iterator_cf("intern"));

Where intern is a column family that exists in the database (but even if it didn't, reading the lib code, It should just error out at me when it tries to read the handle from DB::cfs).

Backtrace:

#0  0x00007fe120295c69 in rocksdb::DBImpl::NewIterator(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*) () from /usr/lib/librocksdb.so.3.13
#1  0x00007fe1202600a5 in rocksdb_create_iterator_cf () from /usr/lib/librocksdb.so.3.13
#2  0x000055b44ebc7bf6 in rocksdb::rocksdb::DBIterator::new_cf (db=0x7ffd44c1b320, cf_name=..., readopts=0x7ffd44c1acf0)
    at /home/ben/.cargo/registry/src/github.com-88ac128001ac3a9a/rocksdb-0.2.1/src/rocksdb.rs:138
#3  0x000055b44ebdba99 in rocksdb::rocksdb::DB::iterator_cf (self=0x7ffd44c1b320, cf=...) at /home/ben/.cargo/registry/src/github.com-88ac128001ac3a9a/rocksdb-0.2.1/src/rocksdb.rs:527
#4  0x000055b44ebbe5b2 in triplicate::triplicate::store::Store::new (path=...) at src/triplicate/store.rs:30
#5  0x000055b44ebbabd5 in triplicate_server::main () at src/main.rs:104
#6  0x000055b44ebe9395 in sys_common::unwind::try::try_fn::h6322428861471041419 ()
#7  0x000055b44ebe6d49 in __rust_try ()
#8  0x000055b44ebe9037 in rt::lang_start::h45458ea0787b3e00e0x ()
#9  0x000055b44ebbd99a in main ()

Any thoughts?

I am kind of new to gdb in general!

[windows] Can not handle error message when OS is not in utf8

In my Windows PC (Chinese Version), when RocksDB get a failure with OS message, it will be bytes encoded in GBK. so str::from_utf8 will fail.

d:\dws\rust-rocksdb>cargo test -- errors_do_stuff --nocapture
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target\debug\deps\rocksdb-c7627b31eefdcf68.exe

running 1 test
thread 'rocksdb::errors_do_stuff' panicked at 'called `Result::unwrap()` on an `Err` value: Utf8Error { valid_up_to: 64
}', ../src/libcore\result.rs:837
stack backtrace:
   0:        0x13f5a0c9a - std::panicking::default_hook::{{closure}}
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libstd\panicking.rs:252
   1:        0x13f59ecca - std::panicking::default_hook
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libstd\panicking.rs:263
   2:        0x13f59f6ad - std::panicking::rust_panic_with_hook
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libstd\panicking.rs:451
   3:        0x13f59f548 - std::panicking::begin_panic<collections::string::String>
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libstd\panicking.rs:413
   4:        0x13f59f464 - std::panicking::begin_panic_fmt
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libstd\panicking.rs:397
   5:        0x13f59f3f9 - std::panicking::rust_begin_panic
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libstd\panicking.rs:373
   6:        0x13f5aaad7 - core::panicking::panic_fmt
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libcore\panicking.rs:69
   7:        0x13f53f4bc - core::result::unwrap_failed<core::str::Utf8Error>
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libcore\macros.rs:29
   8:        0x13f53ccff - core::result::Result<&str, core::str::Utf8Error>::unwrap<&str,core::str::Utf8Error>
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libcore\result.rs:737
   9:        0x13f547015 - rocksdb::ffi_util::error_message
                        at d:\dws\rust-rocksdb\src\ffi_util.rs:22
  10:        0x13f54a5d4 - rocksdb::rocksdb::DB::destroy<&str>
                        at d:\dws\rust-rocksdb\src\rocksdb.rs:413
  11:        0x13f54c176 - rocksdb::rocksdb::errors_do_stuff
                        at d:\dws\rust-rocksdb\src\rocksdb.rs:912
  12:        0x13f55eb7a - test::{{impl}}::call_box<(),closure>
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libtest\lib.rs:141
  13:        0x13f5a37c1 - panic_unwind::__rust_maybe_catch_panic
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libpanic_unwind\lib.rs:97
  14:        0x13f551aab - std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()>
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libstd\panicking.rs:352
  15:        0x13f5a37c1 - panic_unwind::__rust_maybe_catch_panic
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libpanic_unwind\lib.rs:97
  16:        0x13f55978a - alloc::boxed::{{impl}}::call_box<(),closure>
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\liballoc\boxed.rs:595
  17:        0x13f59d97e - std::sys::imp::thread::{{impl}}::new::thread_start
                        at C:\bot\slave\nightly-dist-rustc-win-msvc-64\build\src\libstd\sys\windows\thread.rs:50
  18:         0x76c059ec - BaseThreadInitThunk
test rocksdb::errors_do_stuff ... FAILED

failures:

failures:
    rocksdb::errors_do_stuff

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured

error: test failed

which is caused by https://github.com/spacejam/rust-rocksdb/blob/master/src/ffi_util.rs#L22

A workground is:
change

pub fn error_message(ptr: *const c_char) -> String {
    let cstr = unsafe { CStr::from_ptr(ptr as *const _) };
    let s = str::from_utf8(cstr.to_bytes()).unwrap().to_owned();
    unsafe {
        libc::free(ptr as *mut c_void);
    }
    s
}

to

pub fn error_message(ptr: *const c_char) -> String {
    let cstr = unsafe { CStr::from_ptr(ptr as *const _) };
    let s = String::from_utf8_lossy(cstr.to_bytes()).into();
    unsafe {
        libc::free(ptr as *mut c_void);
    }
    s
}

ReadOptions are not exposed?

New to rust so apologies if this is a misfile, but it seems that ReadOptions only exist in db.rs and aren't exposed. The docs also don't reference the type (ie. http://docs.rs).

Sorry for the lack of PR - like I said, I'm not quite sure on the preferred fix / if this issue is valid.

column family support

Column families are placed into their own store files, but share the same WAL with the rest of the DB - allowing for cross-CF atomicity and maximizing locality for certain types of workloads. One thing I want to use in some situations is a lookaside table that maps prefixes or attributes to a compact corresponding byte sequence for certain time series operations. This allows for a number of nice logical patterns as well as performance optimizations.

Error Compiling the documentation

I'm trying to build the documentation with cargo doc and I get the following:

error: cannot document a package where a library and a binary have the same name. Consider renaming one or marking the target as 'doc = false'

Adding

[lib]
name = "rocksdb"

[[bin]]
name = "rocksdb_bin"

or

[[bin]]
name = "rocksdb"
doc = false

to cargo.toml fixes this issue and allows me to compile. Would it be worth adding some form of one of the two?

Not compilable in Rust 1.1.0

$ cargo run
   Compiling rocksdb v0.0.8
/Users/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:168:27: 168:36 error: type mismatch resolving `<&[u8] as core::iter::IntoIterator>::Item == u8`:
 expected &-ptr,
    found u8 [E0271]
/Users/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:168         Some(v) => result.extend(v),
                                                                                                                               ^~~~~~~~~
/Users/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:168:27: 168:36 help: run `rustc --explain E0271` to see a detailed explanation
/Users/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:172:16: 172:26 error: type mismatch resolving `<&[u8] as core::iter::IntoIterator>::Item == u8`:
 expected &-ptr,
    found u8 [E0271]
/Users/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:172         result.extend(op);
                                                                                                                    ^~~~~~~~~~
note: in expansion of for loop expansion
/Users/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:171:5: 173:6 note: expansion site
/Users/robin/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.0.8/src/merge_operator.rs:172:16: 172:26 help: run `rustc --explain E0271` to see a detailed explanation
error: aborting due to 2 previous errors
Could not compile `rocksdb`.

To learn more, run the command again with --verbose.
$ rustc --version
rustc 1.1.0 (35ceea399 2015-06-19)

Using same jemalloc as Rust

There may now exist an opportunity to use the same jemalloc as Rust, thanks to jemalloc/jemalloc@2319152. This is just a heads up. @kaedroho, you may be interested. I haven't checked your custom compilation code yet, but I presume it uses the default system allocator when building, since having two versions of jemalloc creates big problems.

A few questions about DBResult

Hey there,

I had a few questions about DBResult.

It's unfortunate that it's not an actual Result, because then you don't get the nice combinator methods. I was working on some code with @petehunt today, and I was assuming we could use unwrap_or(), for example, but that doesn't exist.

Is there some specific reason why it's not DbResult<Option<T>, E>, for example? that'd still get you the Some/None distinction, yet inherit all the goodies. I haven't spent a ton of time with this library yet though, so I might be missing something important!

system rocksdb feature flag

It's good to have librocksdb-sys as the default, but some folks require fast updates of the underlying rocksdb library because production is breaking RIGHT NOW and don't have time to wait on us to bump. we should make life easier for these stressed out folks with a feature flag that skips librocksdb-sys

They may also have a rocksdb version that is packaged on their systems and managed by an internal team, and we currently get in their way.

snapshot doesn't support Get function

Hi @spacejam

I find that snapshot has only iterator function, but I think get or getcf is very useful and important too.
If not, I have to use iterator to seek the key, call next(), and then check whether the return key matches.

Although I can wrap this in a function, iterator seek is still slower than using get in rocksdb.

Build fails on gcc 7+

It seems librocksdb-sys v0.4.1 has issues compiling its rocksdb dependency on gcc7+ with issues such as the following:

cargo:warning=                 from rocksdb/db/auto_roll_logger.h:15,
cargo:warning=                 from rocksdb/db/auto_roll_logger.cc:6:
cargo:warning=rocksdb/util/thread_local.h:66:16: error: ‘function’ in namespace ‘std’ does not name a template type
cargo:warning=   typedef std::function<void(void*, void*)> FoldFunc;

It's to be directly related to this issue, which was fixed within rocksdb:

#facebook/rocksdb#1492

Would suggest the rocksdb dependency needs to be updated to a newer one.

[Question] Any chance bringing the forks together?

Hey there!

First, thanks for making this all happen, it helps me a lot in my daily work!

Secondly: Is there a chance, that you/we can get all forks on one table and see how this can be beneficial for the project? For example: You and also https://github.com/ngaut/rust-rocksdb seem to have made pretty good progress, but the branches are too diverged and I lack of knowledge to merge them my self.

So I try and shoutout to @ngaut, @BusyJay and @zhangjinpeng1987 to join the discussion if those two forks should be one again?

Hope that helps somehow ...

Cheers,
Mario

Should Writable take &mut references?

I'm wondering if taking immutable references was intentional (since RocksDB is threadsafe) or unintentional (since you can have unintentional data races despite RocksDB being threadsafe).

I'm still a Rust newbie, but using things like RwLock with rust-rocksdb doesn't feel natural because everything takes an immutable self reference.

Options.set_filter_deletes calls non-existant RocksDB function

Options.set_filter_deletes calls a function in the ffi bindings called rocksdb_options_set_filter_deletes. However, no function of that name is exported by RocksDB itself, leading to a link error (when using static linking).

https://github.com/spacejam/rust-rocksdb/blob/c0e0331d0cb1f652c541d35cebd66de00df26992/src/rocksdb_options.rs#L539
https://github.com/facebook/rocksdb/blob/a10e8a056d569acf6a52045124e6414ad33bdfcd/db/c.cc

Should we remove this method? is there another way to implement it?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.