Giter Club home page Giter Club logo

rustbreak's Introduction

Rustbreak

Build Status Crates Link

Documentation

Rustbreak is an Daybreak inspired self-contained file database. It is meant to be fast and simple to use. You add it to your application and it should just work for you. The only thing you will have to take care of is saving.

When to use it

This library started out because of a need to be able to quickly write an application in rust that needed some persistence while still being able to write arbitrary data to it.

In Ruby there is Daybreak however for Rust there was no similar crate, until now!

When not to use it

Rustbreak makes several trade-offs to be easy to use and extend, so knowing of these drawbacks is important if you wish to use the library:

  • The Database needs to fit into memory (Rustbreak cannot do partial loads/saves, so if the Database exceeds your available memory you will run OOM)
  • Not all backends support atomic saves, so if your program crashes while it is saving you might save incomplete data (Notably only PathBackend supports atomic saves)

Features

  • Simple To Use, Fast, Secure
  • Threadsafe
  • Serde compatible storage (ron, bincode, or yaml included)

Quickstart

Add this to your Cargo.toml:

[dependencies.rustbreak]
version = "2"
features = ["ron_enc"] # You can also use "yaml_enc" or "bin_enc"
                       # Check the documentation to add your own!
extern crate rustbreak;
use std::collections::HashMap;
use rustbreak::{MemoryDatabase, deser::Ron};

fn main() -> rustbreak::Result<()> {
    let db = MemoryDatabase::<HashMap<u32, String>, Ron>::memory(HashMap::new())?;

    println!("Writing to Database");
    db.write(|db| {
        db.insert(0, String::from("world"));
        db.insert(1, String::from("bar"));
    });

    db.read(|db| {
        // db.insert("foo".into(), String::from("bar"));
        // The above line will not compile since we are only reading
        println!("Hello: {:?}", db.get(&0));
    })?;

    Ok(())
}

Usage

Usage is quite simple:

  • Create/open a database using one of the Database constructors:
    • Create a FileDatabase with FileDatabase::from_path.
    • Create a MemoryDatabase with MemoryDatabase::memory.
    • Create a MmapDatabase with MmapDatabase::mmap or MmapDatabase::mmap_with_size with mmap feature.
    • Create a Database with Database::from_parts.
  • Write/Read data from the Database
  • Don't forget to run save periodically, or whenever it makes sense.
    • You can save in parallel to using the Database. However you will lock write acess while it is being written to storage.
# use std::collections::HashMap;
use rustbreak::{MemoryDatabase, deser::Ron};

let db = MemoryDatabase::<HashMap<String, String>, Ron>::memory(HashMap::new(), Ron);

println!("Writing to Database");
db.write(|db| {
    db.insert("hello".into(), String::from("world"));
    db.insert("foo".into(), String::from("bar"));
});

db.read(|db| {
    // db.insert("foo".into(), String::from("bar"));
    // The above line will not compile since we are only reading
    println!("Hello: {:?}", db.get("hello"));
});

Encodings

The following parts explain how to enable the respective features. You can also enable several at the same time.

Yaml

If you would like to use yaml you need to specify yaml_enc as a feature:

[dependencies.rustbreak]
version = "2"
features = ["yaml_enc"]

You can now use rustbreak::deser::Yaml as deserialization struct.

Ron

If you would like to use ron you need to specify ron_enc as a feature:

[dependencies.rustbreak]
version = "2"
features = ["ron_enc"]

You can now use rustbreak::deser::Ron as deserialization struct.

Bincode

If you would like to use bincode you need to specify bin_enc as a feature:

[dependencies.rustbreak]
version = "2"
features = ["bin_enc"]

You can now use rustbreak::deser::Bincode as deserialization struct.

rustbreak's People

Contributors

bennyyip avatar doumanash avatar matthiasbeyer avatar mglolenstine avatar niluxv avatar theneikos avatar vaartis avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rustbreak's Issues

Better Documentation

Before this hits 1.0 it needs better documentation, thinks that could be covered:

  • How should it be used with multiple threads?
  • Can this livelock?
  • What does Lock and Transaction do? How do they differ?

Basically just an overview and a possible usage example, while making sure to not bore the user with unnecessary details.

Different data formats for different db files

It would be useful to be able to specify different data formats for different db files.
I.e. not hardcoding one format at compilation via a feature but choosing at runtime when instantiating the db.

Push for v2

I am now in the finishing strides for version 2!

It comes with several enhancements to the concept that started Rustbreak: Having a single file where you can dump a data structure (usually a HashMap).

I have taken this idea and pushed it as far as I could.

As an overview v2 supports:

  • Strong guarantuees on usability, things should be great to use and fit into the Rust ecosystem
    • This is why Serde and failure have been adopted as interface mechanisms
  • You can switch backends quickly and simply
  • You can pick your serialization strategy how you see fit, and even bring your own! (Just some glue required)

However before releasing I'd like you all to give it a whirl, check out the files in the examples, and check out the preliminary documentation for any shortcomings you notice. I will do the same and push out v2 once I believe that it is polished enough.

Thank you 🥇

Transition to Rust edition 2018

The rustbreak source code is still rust edition 2015. Transitioning to edition 2018 might improve the maintainability since probably most (or at least an increasing number of) developers are used to this edition.

Transitioning is very easy with cargo fix.

Errors with newest stable Rust

Compiling manu v0.1.0 (file:///home/trsh/manu/backEnd2/manu)
error[E0659]: `deserialize` is ambiguous
   --> /home/trsh/.cargo/registry/src/github.com-1ecc6299db9ec823/rustbreak-1.4.0/src/lib.rs:156:13
    |
156 |         use enc::deserialize;
    |             ^^^^^^^^^^^^^^^^
    |
note: `deserialize` could refer to the name imported here
   --> /home/trsh/.cargo/registry/src/github.com-1ecc6299db9ec823/rustbreak-1.4.0/src/lib.rs:80:37
    |
80  |     #[cfg(feature = "bin")] pub use bincode_enc::*;
    |                                     ^^^^^^^^^^^^^^
note: `deserialize` could also refer to the name imported here
   --> /home/trsh/.cargo/registry/src/github.com-1ecc6299db9ec823/rustbreak-1.4.0/src/lib.rs:82:41
    |
82  |     #[cfg(feature = "ron_enc")] pub use ron_enc::*;
    |                                         ^^^^^^^^^^
    = note: consider adding an explicit import of `deserialize` to disambiguate

error[E0659]: `serialize` is ambiguous
   --> /home/trsh/.cargo/registry/src/github.com-1ecc6299db9ec823/rustbreak-1.4.0/src/lib.rs:183:13
    |
183 |         use enc::serialize;

and more

Why does contains_key need S: DeserializeOwned ?

In v1, contains_key has the following signature:

pub fn contains_key<S: DeserializeOwned, K: ?Sized>(&self, key: &K) -> Result<bool>
        where T: Borrow<K>, K: Hash + Eq

Why does it need the S parameter? I don't see it being used in its body or return type, and when I don't pass an explicit S, the compiler complains that a type annotation is needed.
Does it only return true if the key is in the db with the given S type?

In my use case I want to see if the key is present regardless of its type, so that I can detect if my struct has gone out of sync with the data in the db (if the key is present but deserialization fails, I want to panic so that it will force me to introduce a new struct instead and write a migration to upgrade the data from the old struct to the new struct).

Update RON

Rustbreak 2.0.0-rc3 still depends on RON 0.2.0, while the current version of RON is 0.5.1.

Rustbreak 2.0.0 would be a good moment to upgrade dependencies since a major version upgrade is allowed to break backwards compatibility.

Describing generic database

Hello,
I'm creating an application where I use FileDatabase in general case, but I would like to be able to use MemoryDatabase as a fallback option in some cases.

Currently I have no idea how to describe a database with an unspecified backend in Rust's type system, because there is no way of knowing the size of something that implements Backend trait at compile time.

I would try this:

fn create_database<T>() -> Result<T, RustbreakError>
    where T: Database<MyData, dyn Backend + 'static, Bincode> {

But Database is a struct, and I couldn't find any trait that would accurately describe it's behavior.

How can I work around this?

Thanks for the amazing library, btw

Yaml formatting could be better

With the yaml feature the db file looks like this:

---
Key: "---\nasd: []\nfoo: \n  bar: []\n  baz: 0"

This isn't really human readable/editable because of the newlines and whitespace sensitivity in the string.
Was it really intended to store it like that (value as escaped string)?

Btw, what do you think of RON as an option?

Create an iterator of entries in the database

I find myself needing to check each entry in the database for Uuid collisions, and this seemed like a common enough feature to request inclusion. The way i imagine this is adding a method entries to Database that produces an iterator over (&Key, &Value) in the database. I would be willing to work on this if you would accept a PR for it.

CBOR format support

When I deserialize my struct from binary with bincode after adding a field to the struct, it cant deserialize the struct, losing my old data (the other struct fields that stayed the same since the last session).
Apparently, this can be fixed by using a different format that includes field names.
I tried yaml but it takes a long time to serialize/deserialize and the file is very large (for my use case).
Out of all the formats that include field names, CBOR (which serde supports) is the one that should be fastest to (de)serialize and should have the lowest file size because it's binary.
Would it be possible to add CBOR support to rustbreak? :)

Btw, my use case is persistence of data of my application and the structs change between sessions, sometimes fields are added/removed and I want to always load as much data as possible from the previous session (using #[serde(default)] on the struct to use Default::default() on a field granularity level to initialize new fields that aren't in the persisted data).

Out of memory if file size > memory size

I really like the speed and simplicity of this crate. The reason I chose it was because it can process a large amount of data quickly.

Unfortunately if the size of my BTreeMap exceeds available memory, I will have an OS error. I think it was a memory protection fault but I could be wrong.

It was my assumption that a File backed Database used Mmap and that the OS would sort everything out to make a disk-based BTreeMap happen.

At any rate, I wanted to thank you for this handy crate. Perhaps you might like to post a warning in the docs that the file-size may not be able to exceed the available memory?

Problems with Ron and json Value integers

When using setup like

let db = FileDatabase::<HashMap<String, Value>, Ron>::from_path("test.ron", HashMap::new())

in the returned Value

let res = db.get(&self.uid.to_string());

all integers are converted to floats (ex 1 to 1.0)

Missing detailed information from the back-end

When using FileDatabase backend I'm getting

The backend has encountered an error

but I have no idea what kind of error has happened. It'd be great if the original error was preserved.

UB with yaml DeSer

Miri detects Undefined Behaviour when executing integration_tests::mem_yaml: (this used to ICE due to a bug in miri)

Miri log
error: Undefined Behavior: type validation failed: encountered uninitialized bytes at .<enum-tag>, but expected a valid enum tag
   --> ~/.cargo/registry/src/github.com-1ecc6299db9ec823/linked-hash-map-0.5.3/src/lib.rs:114:16
    |
114 |     let Node { key, value, .. } = *Box::from_raw(the_box);
    |                ^^^ type validation failed: encountered uninitialized bytes at .<enum-tag>, but expected a valid enum tag
    |
    = help: this indicates a bug in the program: it performed an invalid operation, and caused Undefined Behavior
    = help: see https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html for further information
            
    = note: inside `linked_hash_map::drop_empty_node::<yaml_rust::yaml::Yaml, yaml_rust::yaml::Yaml>` at ~/.cargo/registry/src/github.com-1ecc6299db9ec823/linked-hash-map-0.5.3/src/lib.rs:114:16
    = note: inside `<linked_hash_map::LinkedHashMap<yaml_rust::yaml::Yaml, yaml_rust::yaml::Yaml> as std::ops::Drop>::drop` at ~/.cargo/registry/src/github.com-1ecc6299db9ec823/linked-hash-map-0.5.3/src/lib.rs:809:17
    = note: inside `std::intrinsics::drop_in_place::<linked_hash_map::LinkedHashMap<yaml_rust::yaml::Yaml, yaml_rust::yaml::Yaml>> - shim(Some(linked_hash_map::LinkedHashMap<yaml_rust::yaml::Yaml, yaml_rust::yaml::Yaml>))` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184:1
    = note: inside `std::intrinsics::drop_in_place::<yaml_rust::yaml::Yaml> - shim(Some(yaml_rust::yaml::Yaml))` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184:1
    = note: inside `serde_yaml::ser::to_writer::<&mut std::vec::Vec<u8>, std::collections::HashMap<u64, std::string::String>>` at ~/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_yaml-0.8.13/src/ser.rs:400:1
    = note: inside `serde_yaml::ser::to_vec::<std::collections::HashMap<u64, std::string::String>>` at ~/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_yaml-0.8.13/src/ser.rs:411:5
    = note: inside `serde_yaml::ser::to_string::<std::collections::HashMap<u64, std::string::String>>` at ~/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_yaml-0.8.13/src/ser.rs:423:26
    = note: inside `<rustbreak::deser::Yaml as rustbreak::DeSerializer<std::collections::HashMap<u64, std::string::String>>>::serialize` at ~/Documents/my_sources/rust/git_project/rustbreak/src/deser.rs:123:16
    = note: inside `rustbreak::Database::<std::collections::HashMap<u64, std::string::String>, rustbreak::backend::MemoryBackend, rustbreak::deser::Yaml>::save_data_locked::<std::sync::RwLockReadGuard<std::collections::HashMap<u64, std::string::String>>>` at ~/Documents/my_sources/rust/git_project/rustbreak/src/lib.rs:548:19
    = note: inside `rustbreak::Database::<std::collections::HashMap<u64, std::string::String>, rustbreak::backend::MemoryBackend, rustbreak::deser::Yaml>::save` at ~/Documents/my_sources/rust/git_project/rustbreak/src/lib.rs:562:9
note: inside `test_basic_save_load::<rustbreak::backend::MemoryBackend, rustbreak::deser::Yaml>` at tests/integration_tests.rs:28:5
   --> tests/integration_tests.rs:28:5
    |
28  |     db.save().expect("error while saving");
    |     ^^^^^^^^^
note: inside `mem_yaml` at tests/integration_tests.rs:148:13
   --> tests/integration_tests.rs:148:13
    |
148 |             test_basic_save_load(&db);
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^
...
165 | test_basic_save_load!(mem_yaml, create_memdb(), Yaml, miri = true);
    | ------------------------------------------------------------------- in this macro invocation
note: inside closure at tests/integration_tests.rs:146:9
   --> tests/integration_tests.rs:146:9
    |
146 | /         fn $name() {
147 | |             let db: Database<Data, _, $enc> = $db;
148 | |             test_basic_save_load(&db);
149 | |             test_put_data(&db);
150 | |             test_multi_borrow(&db);
151 | |             test_convert_data(db);
152 | |         }
    | |_________^
...
165 |   test_basic_save_load!(mem_yaml, create_memdb(), Yaml, miri = true);
    |   ------------------------------------------------------------------- in this macro invocation
    = note: inside `<[closure@tests/integration_tests.rs:146:9: 152:10] as std::ops::FnOnce<()>>::call_once - shim` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:233:5
    = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:233:5
    = note: inside `test::__rust_begin_short_backtrace::<fn()>` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libtest/lib.rs:517:5
    = note: inside closure at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libtest/lib.rs:508:30
    = note: inside `<[closure@test::run_test::{{closure}}#2 0:fn()] as std::ops::FnOnce<()>>::call_once - shim(vtable)` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:233:5
    = note: inside `<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send> as std::ops::FnOnce<()>>::call_once` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/boxed.rs:1081:9
    = note: inside `<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>> as std::ops::FnOnce<()>>::call_once` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:318:9
    = note: inside `std::panicking::r#try::do_call::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:348:40
    = note: inside `std::panicking::r#try::<(), std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>>` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:325:15
    = note: inside `std::panic::catch_unwind::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394:14
    = note: inside `test::run_test_in_process` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libtest/lib.rs:541:18
    = note: inside closure at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libtest/lib.rs:450:39
    = note: inside `test::run_test::run_test_inner` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libtest/lib.rs:475:13
    = note: inside `test::run_test` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libtest/lib.rs:505:28
    = note: inside `test::run_tests::<[closure@test::run_tests_console::{{closure}}#2 0:&mut test::console::ConsoleTestState, 1:&mut std::boxed::Box<dyn test::formatters::OutputFormatter>]>` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libtest/lib.rs:284:13
    = note: inside `test::run_tests_console` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libtest/console.rs:280:5
    = note: inside `test::test_main` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libtest/lib.rs:120:15
    = note: inside `test::test_main_static` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libtest/lib.rs:139:5
    = note: inside `main`
    = note: inside closure at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67:34
    = note: inside closure at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:52:73
    = note: inside `std::sys_common::backtrace::__rust_begin_short_backtrace::<[closure@std::rt::lang_start_internal::{{closure}}#0::{{closure}}#0 0:&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/sys_common/backtrace.rs:130:5
    = note: inside closure at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:52:13
    = note: inside `std::panicking::r#try::do_call::<[closure@std::rt::lang_start_internal::{{closure}}#0 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:348:40
    = note: inside `std::panicking::r#try::<i32, [closure@std::rt::lang_start_internal::{{closure}}#0 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe]>` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:325:15
    = note: inside `std::panic::catch_unwind::<[closure@std::rt::lang_start_internal::{{closure}}#0 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394:14
    = note: inside `std::rt::lang_start_internal` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:51:25
    = note: inside `std::rt::lang_start::<()>` at ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67:5
    = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

test mem_yaml ... 
error: could not compile `rustbreak`.

To learn more, run the command again with --verbose.

It happens in linked-hash-map, a dependency of serde_yaml. There is not much we can do about it, but causing UB just by calling Database::save is very undesirable. It shoud at least be stated in the documentation until it is fixed.

Is call to load() necessary?

let db = FileDatabase::from_path(path, HashMap::new())?;
db.write();
db.save();

Seems to always overwrite the database with only one item, as if from_path created a fresh new database each time.

If I try to load the db:

let db = FileDatabase::from_path(path, HashMap::new())?;
db.load()?;
db.write();
db.save();

then it works only if there is one already, so it's a bit cumbersome to use. Additionally, load() fails if the file has 0 length. Unfortunately, FileDatabase::from_path does create the path with 0 length, so opening of the db and not saving breaks loading later.

I expected sqlite behavior: if there's no database already, create one. If there is one already, keep (load) the data.

Implement TransactionLock!

Right now we have Transaction and we have Lock but we have no way to do both at the same time!
It wouldn't make sense to start a transaction and then lock, so that can be ignored.

It would be acceptable to create a new Type instead of using Transaction, even if that would be preferred. If the implementation is very nice then it might even be acceptable to break compatibility a bit (we are still in 0.x) although it would need to be for good reason.

Remove unimplemented! calls

They are there in case a lock gets poisoned. Right now this is possible if a panic occurs with a Lock being held (or with a bug).

There are two possibilities:

  • Return with an error of it being poisoned
  • Try to restore it on your own

The reason why the second approach is sane is because one could assume that the bug does not originate from the library (this is a rather bold claim though) so if things did indeed panic it wouldn't leave the internal state in a buggy place.

If the first approach is selected then a way to restore the lock would make the most sense.

Documentation about mutliprocessing

Hi !

In documentation, we sees that rustbreak is thread safe but it is not specified if we may use it in a multi-process fashion sharing the same database file and what precaution users must adopt.

I see that open() uses try_lock_exclusive() so I suppose multiple processes can use a single file but have to release the whole database to let other access it.

Thanks

More checks in CI

A few checks to CI we could add:

  • Run cargo audit (required)
  • Run tests with TSAN (required):
    cargo clean && env CC=clang CXX=clang++ CFLAGS=-fsanitize=thread CXXFLAGS=-fsanitize=thread RUSTFLAGS=-Zsanitizer=thread RUSTDOCFLAGS=-Zsanitizer=thread cargo +nightly test -Z build-std --target x86_64-unknown-linux-gnu --all-features
  • Run appropriate tests in MIRI (optional): cargo miri test
    (tests using FFI, IO or other unsupported features can be ignored using #[cfg_atrr(miri, ignore)] on the test)

Database::try_read()

Hi,

it would be nice to have a variant of Database::read() that supports Result<T, E> as return value.

Right now one has to do

let val = db.read(|data| /* something that might fail *)??;

which ... works but is not that convenient. Not sure how easy it is to implement this.
Might be something for 2.1.0...

Bincode not loading in v2 rc3

static CACHE_FILE_REL_PATH: &'static str = "store/xx_cache";

let db = FileDatabase::<HashMap<String, Value>, Bincode>::from_path(r_path, HashMap::new())
.expect("xxx");

let ld = db.load();// "The backend has encountered an error"

Load performance problems

let now = Instant::now();
let ld = self.db.load();
println!("secs={}", now.elapsed().as_secs());

Gives me ~10 seconds.

The file is 600 000 lines of Yaml, but isn't it to much load time?

Atomic file saves?

The file backend seems to truncate and overwrite its File in place. If the program is interrupted during the write, the data will be corrupt/lost.

Could you make file saves atomic? i.e. write to a temp file with random filename, and then rename temp file over the destination? This way the database file will either be fully replaced or not replaced at all. This would give the database some durability.

confusing error message

Hi!
I'm a bit confused about the error message when doing:

let db = Database::open("store").unwrap();
// no further usage of db

error[E0282]: unable to infer enough type information about `T`
  --> src/main.rs:47:14
   |
47 |     let db = Database::open("store").unwrap();
   |              ^^^^^^^^^^^^^^ cannot infer type for `T`
   |
   = note: type annotations or generic parameter binding required

I assume the error means the type of data I want to store in that db, not the db itself.
-> is this assumption correct?
Maybe this behavior should be documented?

Loading database with empty File fails

        FileDatabase::from_path(path, Default::default())
            .and_then(|fdb| fdb.load().map(|_| fdb))
            .map_err(Error::from)
            .map(Database)

This fails with 2.0.0-rc3 when called on an empty file (I'm using the Yaml backend).

Add more examples

Right now we only have a single example that is suitable for a single threaded application, we might want a few other ones. (They could be extracted out of the tests)

  • Multithreaded
  • Lock
  • Transaction

They should all be meaningful (at least somewhat) and possibly fit into a single page (so about 60 lines).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.