Can we query by prefix?

    let result = collection.find_many(doc! {
        "key": {
            "$prefix": "ap", // match apple, apply...
        },
    })?;

Hello there, good day!
More context

It would be very cool to also implement an encryption for the database at a file level, I found this explanation about it from another embedded database of the C# world.

Compiling pypolodb fails on Apple Silicon (M1 Pro tested)

python3 setup.py build
running build
running build_ext
building 'polodb' extension
creating build/temp.macosx-12-arm64-3.9
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -Iinclude -I/opt/homebrew/include -I/opt/homebrew/opt/[email protected]/include -I/opt/homebrew/opt/sqlite/include -I/Users/fahad/Code/dopetrader-deta/venv/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c polodb_ext.c -o build/temp.macosx-12-arm64-3.9/polodb_ext.o
creating build/lib.macosx-12-arm64-3.9
clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk build/temp.macosx-12-arm64-3.9/polodb_ext.o /var/folders/0q/jc8d0l0n2jv0fydvx5b_zhzr0000gn/T/polodb_lib/0.10.4/libpolodb_clib.a -L/opt/homebrew/lib -L/opt/homebrew/opt/[email protected]/lib -L/opt/homebrew/opt/sqlite/lib -o build/lib.macosx-12-arm64-3.9/polodb.cpython-39-darwin.so
ld: warning: ignoring file /var/folders/0q/jc8d0l0n2jv0fydvx5b_zhzr0000gn/T/polodb_lib/0.10.4/libpolodb_clib.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64

Installing using pip or using the built version gives the same issue:

$ python3
Python 3.9.9 (main, Nov 21 2021, 03:16:13)
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import polodb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/fahad/Code/polodbtest/venv/lib/python3.9/site-packages/polodb-0.10.4-py3.9-macosx-12-arm64.egg/polodb.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_PLDB_arr_get'
>>>

This issue is tracking the implementation of aggregation:

Reference: MongoDB

• $match (#118)
• $group
• $count (#122)
• $sum
• $sort

Hi,

See: https://www.polodb.org/docs/advanced/aggression

Cheers,
Alex

无法更新一个string类型的字段，Exception: unexpected type for field 'source_branch', expected: number, actual: String
实际这个字段的键值都是string类型的

When doing:

match collection.find_one(&mk_document! {"_key": "test"}).unwrap() {
		Some(val) => {
			assert_eq!("test", val.get("_key").unwrap().to_string());
                },
                None => {...}
}

Result:

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `"test"`,
 right: `"\"test\""`', src/db.rs:189:13

The assert_eq says that they are not the same. What I want is for the val to get the key and to return it the same as it entered. I see there is some use of &Value, however I am not really experienced enough to understand its implementation without examples. Is there some special method needed to convert Value to string? Thanks.

While testing polodb, I have seen multiple corruptions after the program crashed or just closed with ctrl+c.

Traceback when restarting the program:

drop error: io error: failed to fill whole buffer, backtrace:    0: <polodb_core::errors::Error as core::convert::From<std::io::error::Error>>::from
             at /home/poelzi/.cargo/git/checkouts/polodb-8eb4821ede0a0701/4f109d0/src/polodb_core/errors.rs:317:24
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/result.rs:1961:27
   2: polodb_core::lsm::lsm_backend::lsm_file_backend::LsmFileBackendInner::read_page
             at /home/poelzi/.cargo/git/checkouts/polodb-8eb4821ede0a0701/4f109d0/src/polodb_core/lsm/lsm_backend/lsm_file_backend.rs:576:9
   3: polodb_core::lsm::lsm_backend::lsm_file_backend::LsmFileBackendInner::checkpoint_snapshot
             at /home/poelzi/.cargo/git/checkouts/polodb-8eb4821ede0a0701/4f109d0/src/polodb_core/lsm/lsm_backend/lsm_file_backend.rs:557:29
   4: <polodb_core::lsm::lsm_backend::lsm_file_backend::LsmFileBackend as polodb_core::lsm::lsm_backend::lsm_backend::LsmBackend>::checkpoint_snapshot
             at /home/poelzi/.cargo/git/checkouts/polodb-8eb4821ede0a0701/4f109d0/src/polodb_core/lsm/lsm_backend/lsm_file_backend.rs:131:9
   5: polodb_core::lsm::lsm_kv::LsmKvInner::force_sync_last_segment
             at /home/poelzi/.cargo/git/checkouts/polodb-8eb4821ede0a0701/4f109d0/src/polodb_core/lsm/lsm_kv.rs:532:13
   6: <polodb_core::lsm::lsm_kv::LsmKvInner as core::ops::drop::Drop>::drop
             at /home/poelzi/.cargo/git/checkouts/polodb-8eb4821ede0a0701/4f109d0/src/polodb_core/lsm/lsm_kv.rs:543:27
   7: core::ptr::drop_in_place<polodb_core::lsm::lsm_kv::LsmKvInner>
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497:1
   8: alloc::sync::Arc<T>::drop_slow
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/sync.rs:1263:18
   9: <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/sync.rs:1897:13
  10: core::ptr::drop_in_place<alloc::sync::Arc<polodb_core::lsm::lsm_kv::LsmKvInner>>
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497:1
  11: core::ptr::drop_in_place<polodb_core::lsm::lsm_kv::LsmKv>
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497:1
  12: core::ptr::drop_in_place<polodb_core::db::db_inner::DatabaseInner>
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497:1
  13: alloc::sync::Arc<T>::drop_slow
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/sync.rs:1263:18
  14: <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/sync.rs:1897:13
  15: core::ptr::drop_in_place<alloc::sync::Arc<polodb_core::db::db_inner::DatabaseInner>>
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497:1
  16: core::ptr::drop_in_place<polodb_core::db::db::Database>
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497:1
  17: errata::runtime::Runtime::new
             at ./errata-server/src/runtime.rs:141:5

Database file:

# ll db/errata*
-rw-r--r-- 1 poelzi users 4096 Oct 11 14:02 db/erratad.polodb

Dump:

± |main {1} U:3 ?:14 ✗| → hexdump -C db/erratad.polodb 
00000000  04 22 02 24 54 41 42 4c  45 5f 4d 45 54 41 00 02  |.".$TABLE_META..|
00000010  63 6c 69 65 6e 74 5f 72  65 67 69 73 74 72 61 74  |client_registrat|
00000020  69 6f 6e 00 80 e5 e5 00  00 00 02 5f 69 64 00 14  |ion........_id..|
00000030  00 00 00 63 6c 69 65 6e  74 5f 72 65 67 69 73 74  |...client_regist|
00000040  72 61 74 69 6f 6e 00 02  74 79 70 65 00 0b 00 00  |ration..type....|
00000050  00 63 6f 6c 6c 65 63 74  69 6f 6e 00 03 69 6e 66  |.collection..inf|
00000060  6f 00 32 00 00 00 05 75  75 69 64 00 10 00 00 00  |o.2....uuid.....|
00000070  04 5d ba 0e 0f 67 56 11  ee 82 5b f7 eb f3 70 e6  |.]...gV...[...p.|
00000080  0b 09 63 72 65 61 74 65  41 74 00 96 6d 18 19 8b  |..createAt..m...|
00000090  01 00 00 00 03 69 6e 64  65 78 65 73 00 6d 00 00  |.....indexes.m..|
000000a0  00 03 68 6f 73 74 6e 61  6d 65 5f 31 00 27 00 00  |..hostname_1.'..|
000000b0  00 03 6b 65 79 73 00 13  00 00 00 10 68 6f 73 74  |..keys......host|
000000c0  6e 61 6d 65 00 01 00 00  00 00 0a 6f 70 74 69 6f  |name.......optio|
000000d0  6e 73 00 00 03 63 6c 69  65 6e 74 5f 69 64 5f 31  |ns...client_id_1|
000000e0  00 28 00 00 00 03 6b 65  79 73 00 14 00 00 00 10  |.(....keys......|
000000f0  63 6c 69 65 6e 74 5f 69  64 00 01 00 00 00 00 0a  |client_id.......|
00000100  6f 70 74 69 6f 6e 73 00  00 00 00 04 16 02 24 54  |options.......$T|
00000110  41 42 4c 45 5f 4d 45 54  41 00 02 63 6f 75 6e 74  |ABLE_META..count|
00000120  65 72 00 80 9a 9a 00 00  00 02 5f 69 64 00 08 00  |er........_id...|
00000130  00 00 63 6f 75 6e 74 65  72 00 02 74 79 70 65 00  |..counter..type.|
00000140  0b 00 00 00 63 6f 6c 6c  65 63 74 69 6f 6e 00 03  |....collection..|
00000150  69 6e 66 6f 00 32 00 00  00 05 75 75 69 64 00 10  |info.2....uuid..|
00000160  00 00 00 04 5d b9 ff 36  67 56 11 ee 82 5a f7 eb  |....]..6gV...Z..|
00000170  f3 70 e6 0b 09 63 72 65  61 74 65 41 74 00 96 6d  |.p...createAt..m|
00000180  18 19 8b 01 00 00 00 03  69 6e 64 65 78 65 73 00  |........indexes.|
00000190  2e 00 00 00 03 6b 65 79  5f 31 00 22 00 00 00 03  |.....key_1."....|
000001a0  6b 65 79 73 00 0e 00 00  00 10 6b 65 79 00 01 00  |keys......key...|
000001b0  00 00 00 0a 6f 70 74 69  6f 6e 73 00 00 00 00 04  |....options.....|
000001c0  16 02 24 54 41 42 4c 45  5f 4d 45 54 41 00 02 72  |..$TABLE_META..r|
000001d0  65 70 6f 72 74 73 00 81  04 04 01 00 00 02 5f 69  |eports........_i|
000001e0  64 00 08 00 00 00 72 65  70 6f 72 74 73 00 02 74  |d.....reports..t|
000001f0  79 70 65 00 0b 00 00 00  63 6f 6c 6c 65 63 74 69  |ype.....collecti|
00000200  6f 6e 00 03 69 6e 66 6f  00 32 00 00 00 05 75 75  |on..info.2....uu|
00000210  69 64 00 10 00 00 00 04  5d b9 d7 72 67 56 11 ee  |id......]..rgV..|
00000220  82 59 f7 eb f3 70 e6 0b  09 63 72 65 61 74 65 41  |.Y...p...createA|
00000230  74 00 95 6d 18 19 8b 01  00 00 00 03 69 6e 64 65  |t..m........inde|
00000240  78 65 73 00 98 00 00 00  03 68 6f 73 74 6e 61 6d  |xes......hostnam|
00000250  65 5f 31 00 27 00 00 00  03 6b 65 79 73 00 13 00  |e_1.'....keys...|
00000260  00 00 10 68 6f 73 74 6e  61 6d 65 00 01 00 00 00  |...hostname.....|
00000270  00 0a 6f 70 74 69 6f 6e  73 00 00 03 72 65 70 6f  |..options...repo|
00000280  72 74 5f 6e 72 5f 31 00  28 00 00 00 03 6b 65 79  |rt_nr_1.(....key|
00000290  73 00 14 00 00 00 10 72  65 70 6f 72 74 5f 6e 72  |s......report_nr|
000002a0  00 01 00 00 00 00 0a 6f  70 74 69 6f 6e 73 00 00  |.......options..|
000002b0  03 64 61 74 65 5f 31 00  23 00 00 00 03 6b 65 79  |.date_1.#....key|
000002c0  73 00 0f 00 00 00 10 64  61 74 65 00 01 00 00 00  |s......date.....|
000002d0  00 0a 6f 70 74 69 6f 6e  73 00 00 00 00 00 00 00  |..options.......|
000002e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

An investigation is needed as we may require a Swift/Obj-C binding to implement this. The ultimate goal is to enable smooth usage of PoloDB for iOS users.

• Test is PoloDB can run run on arm arch.
• Swift/Obj-C binding.

Python 3.9.7 (both in virtual environment and global install), polodb version 0.10.0. Ubuntu 18.04.

Fresh database (created for testing, I haven't used polo before)

 ➜ python
Python 3.9.7 (default, Sep 10 2021, 00:03:59) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import polodb
>>> db = polodb.Database("testpolo")
>>> db.createCollection("testing")
Segmentation fault (core dumped)

Despite the segfault, creation is successful:

 ➜ python
>>> import polodb
>>> db = polodb.Database("testpolo")
>>> db.createCollection("testing")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Exception: collection name 'testing' already exists
>>> db.collection("testing")
Segmentation fault (core dumped)

but I cannot switch to the collection anyway because segfault again.

Edit: Same thing with same usage in lower version (0.8.1 from pypi, although polodb.version() says it's 0.8.0?)

Library

polodb

Description

Upgrade the library clap to the latest version.

The array is nested within another array when I try to grab it from the database. Am I just missing something obvious?

let active = mk_array!["item one", "item two"];
println!("{:?}", active);
// > Array([String("item one"), String("item two")])

let mut document = mk_document! {"_id": "active", "items": active};
match store.insert(document.as_mut()) {
    Err(e) => panic!("Failed to insert document: {}", e),
    Ok(b) => println!("Inserted {}", b),
}

let query = mk_document! {"_id": "active"};
let result = match store.find(Some(&query)) {
    Err(e) => panic!("Failed to find document: {}", e),
    Ok(d) => d,
};

println!("{:?}", result[0].get("items");
// There's an extra Array here
// > Some(Array(Array([String("item one"), String("item two")])))

Since the legacy binding is deprecated, needs a new binding

请问test_db.py出现以下问题可能原因是：

test_db.py:13 (test_create_collection)
def test_create_collection():

db = polodb.Database(DB_PATH)

E Exception: IOErr: 拒绝访问。 (os error 5)

find_many_with_session sometimes crash , when I update one database file repeatedly.
following are trace.

thanks.
'main' panicked at 'index 24 is greater than length 1', github.com-1ecc6299db9ec823\polodb_core-3.5.2\page\data_page_wrapper.rs:82:13
thread 'main' panicked at 'C:\Users\alcatel.cargo\registry\src\github.com-1ecc6299db9ec823\polodb_core-3.5.2\db\db.rs:177:43
stack backtrace:
0: 0x7ff73487a360 - std::backtrace_rs::backtrace::dbghelp::trace::hf349c34c3457c740
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src....\backtrace\src\backtrace/dbghelp.rs:98:5
1: 0x7ff73487a360 - std::backtrace_rs::backtrace::trace_unsynchronized::h2bd29529ecc3ce0e
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src....\backtrace\src\backtrace/mod.rs:66:5
2: 0x7ff73487a360 - std::sys_common::backtrace::_print_fmt::hb3b653f470c22f3c
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src\sys_common/backtrace.rs:65:5
3: 0x7ff73487a360 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::ha1b729084c6ddc62
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src\sys_common/backtrace.rs:44:22
4: 0x7ff7348d9f6b - core::fmt::write::he318666ecc6d0e50
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\core\src\fmt/mod.rs:1213:17
5: 0x7ff73486cf15 - std::io::Write::write_fmt::h16e260ff7842914c
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src\io/mod.rs:1682:15
6: 0x7ff73487a0b4 - std::sys_common::backtrace::_print::hfa2b093dead91421
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src\sys_common/backtrace.rs:47:5
7: 0x7ff73487a0b4 - std::sys_common::backtrace::print::ha0089f50e86f9a42
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src\sys_common/backtrace.rs:34:9
8: 0x7ff73487d3ff - std::panicking::default_hook::{{closure}}::h23cb21cc6d06499c
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src/panicking.rs:267:22
9: 0x7ff73487d0d3 - std::panicking::default_hook::h73a8e55e598e91dc
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src/panicking.rs:286:9
10: 0x7ff73487dd49 - std::panicking::rust_panic_with_hook::h38620170de56dbc1
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src/panicking.rs:688:13
11: 0x7ff73487dab6 - std::panicking::begin_panic_handler::{{closure}}::h19e96512213681c1
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src/panicking.rs:579:13
12: 0x7ff73487ae8f - std::sys_common::backtrace::__rust_end_short_backtrace::h0803ea066b746adc
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src\sys_common/backtrace.rs:137:18
13: 0x7ff73487d7c0 - rust_begin_unwind
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\std\src/panicking.rs:575:5
14: 0x7ff7348d6755 - core::panicking::panic_fmt::h6c5d7c67ee48b787
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\core\src/panicking.rs:64:14
15: 0x7ff7348d6d36 - core::result::unwrap_failed::hca540035cf99f0f4
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library\core\src/result.rs:1790:5
16: 0x7ff7345b0060 - <polodb_core::session::client_session::ClientSession as core::ops::drop::Drop>::drop::h68c99acfa97aab78
17: 0x7ff73456e6c9 - test_db::upsert_test_item::h79cc79b0b09dedfd

collection
    .update(
        Some(&mk_document! { "_key": key_name }),
        &mk_document! { "value": value },
    )
    .unwrap();

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: UnknownUpdateOperation("value")', src/db.rs:143:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I want to update the entry with the "_key" as key_name's value field, however, it displays the above error. any help?

When the collection.update runs in the below snippet (after debug:update), it fails to update the value in the database.

pub fn db_insert_config(
	db: &mut polodb_core::Database,
	key_name: &str,
	value: &str,
) -> Result<String, Error> {
	let mut document = mk_document! {
		"_key": key_name,
		"value": value
	};
	println!("{:?}", document);

	let inside = db_inside(db, document.as_mut());
	let mut collection = db.collection("config").unwrap();

	//checks if key already has a value
	if inside {
		println!("debug:update");
		//updates the database entry with new values
		collection
			.update(
				Some(&mk_document! { "_key": key_name }),
				&mk_document! {
				   "$set": mk_document! {
					  "value": value
				   }
				},
			)
			.unwrap();
	} else {
		println!("debug:insert");
		//inserts a new database entry
		collection.insert(&mut document).unwrap();
	}
	Ok(value.to_string())
}

pub fn db_inside(db: &mut polodb_core::Database, bruh: &mut polodb_bson::Document) -> bool {
	let mut collection = db.collection("config").unwrap();

	match collection.find_one(bruh).unwrap() {
		Some(_val) => true,
		None => false,
	}

referenced to update by using:

let bybit_defaults = || -> Result<(), AnyHowError> {
				let mut database = Database::open(database_location().as_str()).unwrap();

				println!("  1. Change default pair to current pair");
				println!("  2. Change default subaccount to current subaccount");

				loop {
					bl();
					let choice = ask("[Option Number]", Some("bybitdefaultsoptionnumber".to_string()))?;
					match choice.as_str() {
						"1" => {
							//let pair_to_check = ask("", None)?;
							println!("{}", db_insert_config(&mut database, "bybit_default_pair", pair)?);
							println!("  Changed default pair successfully");
						}
						"2" => {
							//let subaccount_to_check = ask("", None)?;
							db_insert_config(&mut database, "bybit_default_sub", sub_account)?;
							println!("  Changed default subaccount successfully");
						}
						_ => {
							println!("  {}", boldt("!! Not a choice !!"));
							continue;
						}
					}
					break;
				}
				Ok(())
			};

when collection.find_one("_key": "bybit_default_pair") is ran, it returns the first inserted value. like the update never happened.

Hi, any suggestion about this? Otherwise can't pass Database struct through state using axum.

Trying to write a go wrapper for PoloDB but met this issue while installing the python module:

Traceback (most recent call last):
  File "test_db.py", line 1, in <module>
    import polodb
ImportError: dlopen(/Users/*/anaconda3/lib/python3.7/site-packages/polodb-0.6.0-py3.7-macosx-10.9-x86_64.egg/polodb.cpython-37m-darwin.so, 2): Symbol not found: _PLDB_step
  Referenced from: /Users/*/anaconda3/lib/python3.7/site-packages/polodb-0.6.0-py3.7-macosx-10.9-x86_64.egg/polodb.cpython-37m-darwin.so
  Expected in: flat namespace
 in /Users/*/anaconda3/lib/python3.7/site-packages/polodb-0.6.0-py3.7-macosx-10.9-x86_64.egg/polodb.cpython-37m-darwin.so

System information:

macOS Big Sur 11.0.1
Apple clang version 12.0.0 (clang-1200.0.32.28)
Target: x86_64-apple-darwin20.1.0

The current JS API is sync, async API is needed.

Cannot use db.collection() to select a collection to insert into using the Python wrapper

Traceback (most recent call last):
  File "test_db.py", line 32, in <module>
    test_create_collection()
  File "test_db.py", line 16, in test_create_collection
    collection = db.collection('test')
SystemError: <method 'collection' of 'polodb.Database' objects> returned NULL without setting an error

code:
collection = db.collection('test')

Hello there, good Day!
More context

I have found this talk about concurrency in databases, it may be helpful.

I really liked the project and am planning to use this database in a production setting. Is there any way to get data in a paginated way and if not are there any plans to do so?

After upgrading to the new version, some places have given me the DatabaseOccupied error which is most likely why I had some corruption. In my db.rs file I have many functions that rely on each other that independently use the database. To fix opening the database over and over, I added code to pass the database from the top level function (get_db_info) all throughout the reliant functions. When inserting, it does not give me an error, but when I try to access the database using the same function again, it gives me an error. I have no idea what this error is but I hope I give a good idea of my problem and context within the code.

... is a placeholder for omitted information.

Here is the problematic code:

pub async fn get_db_info(...) -> ... {
	let mut db = Database::open(database_location().as_str()).unwrap();
        ftx_pub_key = get_dbinf_by_entry(&mut db, ...);
        ...
}

pub fn get_dbinf_by_entry(db: &mut polodb_core::Database, ...) -> ... {
        let mut collection = db.collection("config").unwrap();
        db_insert_config(db, ...)?
        ...
}

pub fn db_insert_config(db: &mut polodb_core::Database, ...) -> ... {
	let mut document = mk_document! {
		...
	};

	let inside = db_inside(db, document.as_mut());
	let mut collection = db.collection("config").unwrap();

	if inside {
		//updates the database entry with new values
		collection
			.update(
				Some(&mk_document! { "_key": ... }),
				&mk_document! {
				   "$set": mk_document! {
					  "value": ...
				   }
				},
			)
			.unwrap();
	} else {
		collection.insert(&mut document).unwrap();
	}
        ...
}

pub fn db_inside(db: &mut polodb_core::Database, bruh: &mut polodb_bson::Document) -> bool {
	let mut collection = db.collection("config").unwrap();

	match collection.find_one(bruh).unwrap() {
		Some(_val) => true,
		None => false,
	}
}

Is the version of a database specified in the database file itself?

If it does, that is cool. If it does not, it would be nice to have a feature where (at least) the past major version is automatically converted, meaning manual conversion would not have to be a thing.

Hey!

The Advanced condition is missing some examples for other conditions. I can spend some time adding those.
But I don't see any repository in the PoloDB organization corresponding to the documentation of https://www.polodb.org/docs .

Do you accept contribution to the documentation ?

Hello there, good day!
There is this very cool talk I found online about text searching in rust. However, I also found 2 implementations of search libraries that also implement indexes:

• https://docs.rs/tantivy/latest/tantivy/ <- this one is more mature and has more feature
• https://docs.rs/noise_search/latest/noise_search/

I'm not very familiar with the project, so I think it will be difficult for me to implement this feature right now, but as I saw that there is no implementation yet, so I figure I would research a little about the topic to see if I find something useful for you!

Also, there is another implementation of an embedded database for C# called LiteDB, and they have an explanation on their index implementation. Btw, their implementation is based on this video.

An investigation is needed as we may require a Swift/Obj-C binding to implement this. The ultimate goal is to enable smooth usage of PoloDB for Android users.

• Test is PoloDB can run run on arm arch.
• Java/Kotlin binding.

While it is possible to insert a document containing nested documents into a collection, it doesn't appear possible to use them in queries. Meaning it's impossible to filter documents by a nested value. I initially encountered this when trying to filter by an enum containing variants with fields (as Bson serializes them as nested documents).

Here I have written a minimal example of what I mean, using just a struct inside another struct:

#[derive(Debug, PartialEq, Serialize, Deserialize)]
struct Example {
    inner: Inner,
}

#[derive(Debug, PartialEq, Serialize, Deserialize)]
struct Inner {
    data: String,
}

let example = Example { inner: Inner { data: "foobar".into() } };

let db = Database::open_memory().unwrap();
let col: Collection<Example> = db.collection("test");

col.insert_one(&example).unwrap();

let found1 = col.find_one(None).unwrap().unwrap();
assert_eq!(example, found1);

// create query document using the data we just read from the database
let query = polodb_core::bson::to_document(&found1).unwrap();

let found2 = col.find_one(query).unwrap(); // panics, find_one() returns Err
assert_eq!(Some(found1), found2);

For reference, the Bson-serialized example data looks like this:

Document({
    "inner": Document({
        "data": String("foobar")
    })
})

The error returned from the last .find_one() operation:

InvalidField(InvalidFieldStruct { field_type: "query", field_name: "data", path: Some("/inner/data") })

MongoDB reference Query on Embedded/Nested Documents

Additionally, if I instead use mongodb's dot notation for the query:

let query = doc! {
    "inner.data": "foobar"
};
let found2 = col.find_one(query).unwrap();
assert_eq!(Some(found1), found2); // panics, found2 is None

This find_one() operation no longer returns an error, but it also doesn't return any data.

I am referring to this example.

I made a small modification to change the lookup key name

collection.find(doc! {
	    "name": { "regex": polodb_core::bson::Regex {
	        "pattern": "\w*Ring\w*",
	        "options": "i",
	    }}
})?;

I got this error from Cargo:

error: unknown character escape: `w`
"pattern": "\w*Ring\w*",
        ^ unknown character escape
help: if you meant to write a literal backslash (perhaps escaping in a regular expression), consider a raw string literal
        "pattern": r"\w*Ring\w*",

I did as suggested, i.e. changed it to "pattern": r"\w*Ring\w*", but then I got

missing fields optionsandpatternin initializer ofRegex``

Hi,

I saw you merged wasm feature into master, but if I run a quick test, using polodb_core = "4.4.0", I get the following compiler error:

error[E0599]: no function or associated item named open_file found for struct Database in the current scope
--> src/polo_repository/mod.rs:12:26
|
12 | return Database::open_file("resources.db").unwrap();
| ^^^^^^^^^ function or associated item not found in Database

the code snippets is just:

`use std::fs::File;
use std::io::BufReader;

use polodb_core::Database;
use serde::{Deserialize, Serialize};
use serde_json::Deserializer;

lazy_static! {
pub static ref POLO_DB: Database = {
println!("Initializing PoloDB");

    return Database::open_file("resources.db").unwrap();
};

}

fn insert_json_to_polodb<T: for<'de> Deserialize<'de> + Serialize>(
file_path: &str,
collection_name: &str,
) {
let collection = POLO_DB.collection(collection_name);
let stream =
Deserializer::from_reader(BufReader::new(File::open(file_path).unwrap())).into_iter::();

stream.for_each(|data| {
    collection.insert_one(data.unwrap()).unwrap();
});

}`

This issue is used to track the progress of IndexedDB backend

• POC
• Implements backend
• Tests
• JS API wrapper

I am using a axum server with a polodb_core backend.
axum = 0.7.1
polodb_core 4.4.0

The database keeps getting corrupted. About once a day, I get a crash. I plan to do more testing including stress testing, but to get things started, this is what I know now.

It started with
thread 'tokio-runtime-worker' panicked at /home/hydro/.cargo/registry/src/index.crates.io-6f17d22bba15001f/polodb_core-4.4.0/lsm/lsm_backend/lsm_file_backend.rs:180:9:
assertion failed: flag == format::LSM_INSERT || flag == format::LSM_POINT_DELETE

Code
let coll: Collection = db.collection("lthp");
let cnt = coll.count_documents().unwrap();
let sd = serde_json::to_string(data).expect("Bad Data");
//let t = serde_json::from_value(data);
//println!("{:?}", sd);
let utc: DateTime = Utc::now();
let ts = utc.timestamp();
let rslt = coll.insert_one(LTHP {
id: cnt,
ts: ts,
bme280: data["bme280"].clone(),
as7341: data["as7341"].clone(),
});

It is possible to provide wasm or native binaries for running in the Node.js.
Moreover, it may be able to use bindings consistent with the web browser runtime.

I am storing records and one of the fields is an Enum type

pub enum Cuisine {
    French,
    Chinese,
    Thai,
    Japanese,
    Korean,
    Indian,
    Italian,
    Mexian,
    Turkish,
    Other(String),
}

There is no problem with the insertion. However, a query

let restaurants = collection.find(doc! {
        "cuisine": cuisine,
 })?;

where cuisine is a variant of Cuisine type

produces the following compiler error:

error[E0277]: the trait bound `Bson: From<Cuisine>` is not satisfied
the trait `From<Cuisine>` is not implemented for `Bson`

Is there a way to query by an Enum variant at all, or this is currently not possible without implementing Display trait for it, then converting it to a String?

Currently, I wrote some macro for user to generate json structures friendly

let mut doc = mk_document! {
        "_id": 10,
        "value": "something",
};

But it's not friendly enough for nested structures:

let mut doc = mk_document! {
        "_id": 10,
        "value": mk_document! {
                "content": "something"
        },
        "arr": mk_array![],
};

I think this way would be more elegant:

let mut doc = mk_document! {
        "_id": 10,
        "value": {
                "content": "something"
        },
        "arr": [],
};

我在创建了 数据集之后 调用是没有问题的

db.createCollection('students');
const collection = db.collection('students');
console.log(collection)

但是同一段代码在下次运行的时候就会报错
collection name 'students' already exists

感觉改成 在 db.collection('students');的时候不存在的话自动创建数据集，存在的话直接返回已存在的数据集，这样会更灵活一下

clippy会提示一些比较好的写法

It would be slick to have a update_or_insert (upsert) collection method that would take a query to uniquely identify a document, and the new or updated document. If the query matches an existing document, the document is updated with the new values provided, else it is created.

In my particular application, new and existing records are not so easy to tease apart, and fetched data includes a mix of both. I would rather not have to query the database for existing records, split the list, and perform both an insert_many and a number of update_one operations in a loop. I appreciate that transactions make this pretty trivial in terms of performance and consistency, but it would be a super nice convenience for an already very convenient crate. Having the _many version of this would be even better!

// impl<T>  Collection<T>
pub fn update_or_insert(&self, query: Document, upsert: Document) -> Result<UpdateOrInsertResult>;

// App
let mut session = db.start_session().unwrap();
session.start_transaction(None).unwrap();
 
let collection = db.collection::<Item>("items");
let new_and_updated: Vec<Item> = refresh_items().await?;

for item in new_and_updated {
  collection.update_or_insert(
    doc! { "guid": item.guid },
    bson::to_document(item).unwrap(),
  )?;
}

Hello, I'm having a rather frustrating issue that I'm positive is not working as intended. I am using polodb_core version 4.4.0.

I can create a db file with Database::open_file(), I can create a collection, I can insert documents to it, I can search them and I can update them. All of these operations seem to be written to disk properly, as in, the next run of the program does have the created/updated documents.

However! This is not the case with deletions! When I call either delete_one or delete_many, the selected documents are deleted from memory as expected —I can verify they are no longer there by calling find afterwards, so delete_one being called with the wrong filters is not a possibility—. But in the next run of the program, the deleted documents are back, suggesting that the delete operation has not persisted:

use polodb_core::{Database, Document, bson::doc};

let db = Database::open_file("tasks.db").unwrap();
let col = db.collection::<Document>("tasks");

// Run #1
col.insert_one(doc! { "name": "t1" })?;
col.find(None)?; // The document { "name": "t1" } is returned
// Run #2
col.delete_one(doc! { "name": "t1" })?;
col.find(None)?; // No documents returned
// Run #3
col.find(None)?; // The document { "name": "t1" } is returned, but none should be returned instead

In fact, toying around with doing inserts and deletes I found out that they also kinda not work together as expected, but I'm not following the logic behind what persists and what doesn't:

// Run #4
col.delete_one(doc! {"name": "t1"})?;
col.insert_one(doc! {"name": "t2"})?; // Will inserting a different document force the deletion to commit?
col.find(None)?; // Only the document {"name": "t2"} is returned
// Run #5
col.find(None)?; // Both documents {"name": "t1"} and {"name": "t2"} are returned, only the latter should exist

This issue makes PoloDB largely unreliable for my purposes, since I don't really know the extent of this issue and wether there are more cases that don't persist. I have been researching all causes I could come accross. I ensured that my filesystem is not interfering. I think this is probably a bug.

I could not find any document explaining how I should proceed about reporting bugs, so if I can provide additional information or report this somewhere else, please let me know.

I wrote a Swift adapter of PoloDB in XCode, but it's not good for distributing, a Cocopod is needed.

I use a model like this:

/// Structure for storing client registration data
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ClientSettings {
    pub _id: Option<polodb_core::bson::oid::ObjectId>,
    /// Generic client name
    pub name: String,
    /// Hostname of the client
    pub hostname: String,
    /// UUID of the client
    pub client_id: String,
    /// Secret for authentication
    pub client_secret: String,
    /// Type of system
    pub typ: ClientType,
    /// Date of last modification
    pub last_modified: DateTime,
    /// Date of creation
    pub created: DateTime,
    /// Collection of facts about the client
    pub facts: HashMap<String, String>,
}

When the _id value is None and collection.insert_one(&client) is called, the returned
inserted_id is Null.
When the document is serialized to bson, the _id is set to Null.
This is then used as an id, instead of generating a real one.

If I remove the _id field, the insert operation returns a proper _id.

Since bson crate have all functionalities that you implemented in polodb_bson crate. And it have additional features such as serde , chrono. Do you have a plan to use bson crate in this project? If you like, then I can work on it. Because I am working on a project that need to use mongodb and polodb in same time.

Hey !

I just started using PoloDB and it works like a charm ! Great software and it's very intuitive 😄 .

I have a quick question. The documentation specifies which search operator is implemented from the MongoDB engine (I guess it's the MongoDB engine). Does it includes the $regex operator ?

I want to search for a specific pattern inside the DB with a simple pattern like

{
"$regex": [{
  "name":  r".*name.*"g
}]

Thanks!

It would be useful to have some examples for this module, such as in an examples/ folder.

还没有实际测试，不过非常支持
readme里面小小错字

PoloDB is a embedded JSON-based database.
'a' 可以改成'an'

很多不需要高并行独立进程的简单场景这种数据库就非常实用了，本人很多个人项目用的就是sqlite.
本人并非程序员,只会一些C++/python/js，完全不会搞rust，但是等项目成熟一些可以帮着一起贡献文档啦,可视化啦之类的
PS:感觉可以加入一些CI/CD之类的自动化流程，以后项目更复杂了的话会提高效率