objectbox / objectbox-dart Goto Github PK
View Code? Open in Web Editor NEWFlutter database for super-fast Dart object persistence
Home Page: https://docs.objectbox.io/getting-started
License: Apache License 2.0
Flutter database for super-fast Dart object persistence
Home Page: https://docs.objectbox.io/getting-started
License: Apache License 2.0
Looking at some other dart code I've got an impression the API should be asynchronous (returning Futures). Thoughts?
The way we currently marshal and unmarshal flatbuffers into objects isn't optimal. We should get rid of the intermediary propVals map<string, dynamic>
. Ideally creating objects (or flatbuffers in the other direction) directly in the generated code.
Goal: minimal solution including a query builder and a reusable query object. For reference, please check the Java query API.
"flatbuffers_id": ${(prop['id'] as int) - 1},
is always property.Id - 1
it can be removed completely (as in other language bindings)dartfmt -l 120
(and add it to contribution guide?)Currently, generator has a single "integration" test - verifying the code is generated properly.
It could use more tests - especially all supported types and annotations should be covered by tests.
Currently, binding code must be generated manually using pub run build_runner build
before running/building an application
Is there a way this could be streamlined?
Why don't use an Effective Dart for this project?
Like here for constant names definitions you break it.
Thanks!
Add to Box:
Dart v2.6 is going to break compatibility in FFI (it was in "preview" after all, so no surprise there). From what I've seen with the current dev version (.8), it was rather significant so I have a feeling we'll have to drop Dart v2.5 compatibility to be able to switch to v2.6
See the changes that have already landed in the dev release and the FFI project board for what's coming.
Currently, writing strings to and reading strings from memory is done character by character. Try to find a more efficient method to do that.
Is mostly related to CString (#7), but also memory chunks in general.
The typical ObjectBox example is a simple note taking app. So let's do that using Flutter...
Destination: this repo inside a new examples/flutter/ dir.
For reference:
https://github.com/objectbox/objectbox-examples
If I'm not wrong, the check(cond)
does not result in helpful messages.
To be more friendly to users, we should tell a) what's wrong and b) how to fix it.
Currently, user code sees all symbols defined in lib/src/binding & lib/src/modelinfo.
I'm rather concerned about binding as it fairly pollutes the symbol-space (all the C typedefs, enums, helper functions, etc..)
We should limit the visibility so that those functions are not propagated to user code.
Native ObjectBox uses UTF-8, while Dart's String uses UTF-16.
Check our CString wrapper Dart class, which should do proper conversion.
PS.: Is there a better way to process chars than iterating over each character? E.g. is there a byte buffer, we could interpret as UTF-8? If FFI does not offer that yet, please check if they already have an issue for that (or create one).
Verify (= have a test case) the generator supports generating a single database model for entities spread around multiple files.
The code seems a little fragile, with the bitshifts and so on so it should be covered by a unit test.
If you run the branch #51 and #58, and compare the count
and find
result on the PropertyQuery
, you'll find that the find
function sometimes returns a partially correct result set, with certain random bytes (every time you run it), that are the same as the random bytes as count
consistently returns.
This leads me to think that these obx functions return via ffi before the actual correct values are populated on the array alloc. This reminds me of the async await
the Future
discussion #49 .
We should track the development of this issue.
It might be available on the bleeding edge dart.
We shouldn't require to specify Type.Long
for ID properties. There's no alternative to it, so we should verify the Dart type as int
and set the ObjectBox type to Long.
Update the readme.
The passed object should be updated to also contain ID after a successful insert. This applies to box.put()
and box.putMany()
Example for put()
:
final object = TestEntity.initText("Hello");
int putId = box.put(object);
expect(putId, 1);
expect(object.id, 1);
Using obx_opt_*
functions, properties like the store path, database filename etc. can be set. See here for the list of possible options.
Use a Store
constructor with named arguments to easily specify any non-default options.
Depends on dart-archive/ffi#6
Currently, the generator omits (doesn't store) an entity field with an unknown annotation.
From generator.dart:
// skip unknown annotations
print(
"warning: skipping field '${f.name}' in entity '${element.name}', as it has the unknown annotation type '$annotType'");
continue;
}
Instead, we should process those fields normally and ignore unknown annotations.
Support write transactions by Store::runInTransaction
See Go for reference: https://github.com/objectbox/objectbox-go/blob/master/objectbox/objectbox.go#L107
We're using GitHub Actions to pub run test
- execute tests in the test
folder - see .github/workflows/`
Another workflow should be added to build the flutter demo/example.
package:test seems to be the standard for Dart unit testing?
I got this, by moving the lib one level lower (project root):
...
Invalid argument(s): Failed to load dynamic library (dlopen(libobjectbox.dylib, 1): no suitable image found. Did find:
libobjectbox.dylib: code signature in (libobjectbox.dylib) not valid for use in process using Library Validation: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.)
dart:ffi new DynamicLibrary.open
...
It might be a new security thing. I'm not the only one. According to this thread, we either need objectbox's or dart's signing key, or remove the signing hash from the compiling dart binary. I fixed it by doing the last bit.
From bash or zsh wherever:
codesign --remove-signature `which dart`
Originally posted by @Buggaboo in https://github.com/objectbox/objectbox-dart/timeline
Because every value can be null
in Dart, correct serialization of it must be tested.
It was brought up on reddit that there might at least one compatibility issue with Flutter:
I noticed in the box class you're using dart:mirrors. As far as I know that doesn't work in flutter, did that recently get changed? Here is some discussion about it from a while ago.
So, ensure everything run on Flutter.
Bonus task: can we unit test using Flutter?
Double-check that we at least support the basic Dart types for entity properties and as query conditions: int, double, String, bool.
If we don't support byte arrays yet, create a separate ticket; as far as I can tell there is Uint8List and potentially ByteData to be supported.
Queries currently use some handy operator overloads, e.g. box.query(Note_.text == "text")
in addition to the standard box.query(Note_.text.equals("text"))
.
Equality seems to be a special case, causing a linter issue. Make sure it's really OK to use it the way we do, otherwise, we'll have to drop it.
Linter reports:
error: 'QueryStringProperty.==' ('Condition Function(String)') isn't a valid override of 'Object.==' ('=='). (invalid_override at [objectbox] lib/src/query/query.dart:86)
info: Equality operator `==` invocation with references of unrelated types. (unrelated_type_equality_checks at [objectbox] test/query_test.dart:52)
Also, we clearly break the requirements given here: https://api.dartlang.org/stable/2.5.2/dart-core/Object/operator_equals.html
Currently, to use objectbox-dart
with flutter on Android, a user must manually specify a dependency on objectbox-android
, which is a gradle artifact published by the objectbox-java
project. Ideally, we should specify this dependency here in the dart/flutter lib, so the consumers wouldn't have to do it.
Follow up from #2
See my question here: flutter/flutter#33227 (comment)
A branch where I tried something (it didn't work though): https://github.com/objectbox/objectbox-dart/tree/flutter-plugin
The code generator is tested using "golden/expected" files - verifying the generated code matches the expected code exactly. While this is a great way to make sure the code doesn't change unexpectedly and there are no regressions, it poses a challenge when actually changing the generator code (as a developer).
To achieve a good developer experience, the test-cases expected code should be updatable by calling a single command. For example, in Go, tests are normally called as go test tests/generator
while there's an option to update the .expected
files calling go test tests/generator -update
.
After the files are updated, it's up to the developer to verify the changes are correct (by manually reviewing them), before committing the code to git.
After the flutter mobile example is prepared, it can be used to try if we can get the library to work on iOS. Seems like a .framework
would be necessary. Hopefully, we can extract it from an objectbox-swift release
Create equivalents of the @Unique
and @Index
Java annotations for Dart.
Seems like we can detect leaks using valgrind:
dart2native ./test/box_test.dart -o box-test
valgrind --show-mismatched-frees=no ./box-test
==13549== LEAK SUMMARY:
==13549== definitely lost: 0 bytes in 0 blocks
==13549== indirectly lost: 0 bytes in 0 blocks
==13549== possibly lost: 0 bytes in 0 blocks
==13549== still reachable: 91,012 bytes in 188 blocks
==13549== suppressed: 0 bytes in 0 blocks
And If I manually comment out a free()
call on an ffi.Pointer
(e.g. in box.count()
), it shows a leak:
==13580== LEAK SUMMARY:
==13580== definitely lost: 128 bytes in 16 blocks
==13580== indirectly lost: 0 bytes in 0 blocks
==13580== possibly lost: 0 bytes in 0 blocks
==13580== still reachable: 91,012 bytes in 188 blocks
==13580== suppressed: 0 bytes in 0 blocks
Similarly, when I comment out a call to obx_query_close()
and run valgrind on the query-test, it catches a leak of the core resources as well:
==15074== LEAK SUMMARY:
==15074== definitely lost: 6,200 bytes in 181 blocks
==15074== indirectly lost: 218,547 bytes in 3,199 blocks
==15074== possibly lost: 0 bytes in 0 blocks
==15074== still reachable: 91,012 bytes in 188 blocks
==15074== suppressed: 0 bytes in 0 blocks
Therefore, it is clear it can help detect leaks that could be potentially introduced in the future code changes. There is also a load of Warning: set address range perms: large range
and mismatched free() / delete / delete []
reports but I don't think those are related to our code, at least for the latter. That would require further investigation though.
Create a script for project developers and CI. The script:
/test/*_test.dart
files using dart2native
valgrind
The script should ideally return a non-zero code in case of an error, so it can easily be used in the CI.
Prepare docs, resolve code quality issues (#31 #28) and publish, ideally using CI, from the master branch?
See https://dart.dev/guides/libraries/create-library-packages#distributing-a-library
Public APIs should have Doc
comments so they would show up on pub.dev docs page
// TODO (perf) find a way to get access to the underlying memory of Uint8List to avoid a copy
// In that case, don't forget to change the caller (FlatbuffersManager) which expect to get a copy
// if (data.length == bytes.length) {
// bytes._dataPtr = data.some-way-to-get-the-underlying-memory-pointer
// return ptr;
// }
I saw this bit of code and did some research, then came to the conclusion, that dart doesn't have a List
to UnsafePointer
type (yet).
To avoid copying twice (i.e. once for collecting bytes in the buffer, to flatten the entity, twice for copying the buffer to the Pointer
alloc), you might as well not use the (flat buffer) builder, and directly allocate a Pointer<Uint8>
(?) with the the proper payload. The new builder should allocate, align and copy at the last moment, when finish
is called.
I'm not completely sure if the changes should be made on google's builder, or this project's.
@Entity() class MyEntity { ... }
should suffice@Id()
) need to be annotatedobjectbox_model_generator
, together with these rules on how to resolve changes and conflicts_MyEntity_OBXModel
object) needs to be parsed beforehand in order to detect changesObjectBox usually depends on build time tooling, e.g. parse source code with annotation in an AST so that we can generate some code that is compiled along with the user's code (entity (type) binding). Also, ObjectBox tracks the data model and IDs via a JSON file. The entity binding and the data model JSON itself should go in new tickets once we know better what we have.
What build time tooling is available in Dart? What can it do?
Links:
Instead of loading model from JSON, include all information in the generated code. Some of the information is already there, in the generated Entity_
class's static members so it may be useful to reuse that but it's not mandatory.
Another chunk of code coming this way, stay tuned.
Currently, when one runs pub run build_runner build
in the library root, it also generates the code for examples, considering all files the same project, effectively combining their databases. How can we prevent that?
It's necessary to test whether minimum and maximum integer values for all integer types currently supported by the generator are stored and re-fetched correctly.
Goal: make ObjectBox simple to use when deployed to Android
First, we need to have a binary distribution of the dynamic lib for Android (ObjectBox Java for Android comes with JNI API; not the C API...).
Resources:
Open questions:
Once done: create a ticket to do the same for iOS
Operating system: Ubuntu 16.04 [to be confirmed for other OSs]
Dart version: 2.4.1
When trying to call ObjectBox functions with e.g. invalid parameters, normally the error should be reported on the command line as well as being reflected as an error code retrieved from obx_last_error_code
. As an example, try creating an entity with ID -1 (@Entity(id: -1, uid: 1)
), which is illegal in ObjectBox. Instead of seeing a meaningful message, the following is output, in other cases with the OBX_ERROR_NO_ERROR_INFO
error code:
001-10:36:16.1248 [ERROR] Cannot handle thrown exception, no ptr
After some investigation, it became obvious that this issue is due to this problem in the Dart FFI, i.e. something rather fundamental in C++.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.