Handle decoding null values in the core library. Adapt the C FFI if needed.

Support enum-based error handling in datajoint-core.

The ConnectionSettings URI builder should check for required settings during string construction and return the proper errors to the caller.

The ConnectionSettings struct needs to be used for creating an instance of the Connection struct.

Set up the initial file structure for the datajoint-core repository. This repository should contain two Rust library crates (packages) and should follow standard conventions for organizing both. The following two packages should be contained in the repository:

• datajoint-core - A Rust library for shared code across DataJoint clients. Will currently facilitate connections to SQL databases and raw querying.
• datajoint-core-ffi-c - A Rust library for calling datajoint-core from other languages using the C FFI. This crate defines all C-interface functions (written in Rust) that expose the functionality of datajoint-core to other languages.

The datajoint-core package should currently contain only one submodule: the connection submodule. This submodule should somehow be accessible to the outside world (for exmaple: datajoint_core::connection). In the future, datajoint-core will support more submodules as more functionality is ported over, such as query, schema, and more.

Both Rust packages should be usable as a library crate that can be used by other Rust programs.

Here is an example structure that puts these two packages in one repository:

/packages
    /datajoint-core
        /src
            /connection
        Cargo.toml
    /datajoint-core-ffi-c
        /src
        Cargo.toml
LICENSE
README.md

Will allow better integration with datajoint-python.

Don't need to test everything, but do need to test some of the C FFI-exclusive code.

A database connection should have a public interface for running raw SQL queries (in string form). Queries are assumed to be correct for the database with no placeholder arguments. Query results and potential errors should be safely communicated to the caller.

Wrap C FFI for Connection class in a Python class.

Wrap C FFI for TableRow class in a Python class.

Allow a database connection to be initialized and established using a settings object (as defined by #3). Initialization and connection should be (if possible) two separate actions to reflect what currently happens in the client libraries.

https://keepachangelog.com/en/1.0.0/

May need more clarification from @guzman-raphael on what exactly this means from an API standpoint.

Wrap C FFI for TableRowVector class in a Python class.

C FFI should be able to retrieve the error message associated with the last error code received.

https://docs.github.com/en/actions/publishing-packages/publishing-docker-images

Design how placeholder arguments can be generically sent over the C FFI to the Rust library.

The execute method should be exposed in the Rust library for insertions, deletions, modifications.

After returning a C string to the caller, the caller needs to call datajoint_core_cstring_free to free the string from memory

All C FFI code should have Rustdoc comments, properly formatted, explaining how the function works, side effects, output parameters, and more.

Support disconnecting synchronously from a connection.

Support decoding and encoding the SQL bigint type (64-bit integers) in the core library.

Define and implement a unified set of settings (along with their defaults) for connecting to MySQL and Postgres databases. datajoint-core should currently only use default settings, but setting these settings programmatically and reading them from a file should also be supported. All settings should be appropriately mapped to SQLx for creating a database connection.

Set up SQLx, a crate for connecting to and querying various types of SQL databases. This crate should only be used by datajoint-core.

https://docs.github.com/en/actions/publishing-packages/publishing-docker-images

Methods that exist currently in datajoint-core should be exposed over a C FFI in datajoint-core-ffi-c.

The getters and setters are a bit clunky, especially when it comes to setting the database type. This class should extremely simple to update connections on the fly.

Create tests that will use multiple units together to maximize code coverage

datajoint-core should support binding placeholder arguments for queries sent through SQLx. Rust library only!

Add the DataJoint Binary type to the Rust library, with proper decoding.

Depends on #22. Functionality should be exposed over C FFI.

The C FFI should have generic decoding using one method as opposed to needing to call a method for every possible data type. The Rust library will need to support generic decoding as well.

pub enum DecodeResult {
    Int8(i8),
    UInt8(u8),
    Int16(i16),
    UInt16(u16),
    Int32(i32),
    UInt32(u32),
    String(String),
    Float32(f32),
    Float64(f64),
    Bytes(Vec<u8>),
}

pub fn TableColumnRef::type_name(&self) -> DataJointType;
pub fn TableRow::decode(&self, column: TableColumnRef) -> DecodeResult;
pub fn TableRow::try_decode(&self, column: TableColumnRef) -> Result<DecodeResult, Error>;

// Getters based on native type.
int table_row_get_int8(TableRow* this, TableColumnRef* column, int8_t* out);
int table_row_get_by_name_int8(TableRow* this, const char* column_name, int8_t* out);
int table_row_get_by_ordinal_int8(TableRow* this, size_t column_ordinal, int8_t* out);

int table_row_get_uint8(TableRow* this, TableColumnRef* column, uint8_t* out);
int table_row_get_by_name_uint8(TableRow* this, const char* column_name, uint8_t* out);
int table_row_get_by_ordinal_uint8(TableRow* this, size_t column_ordinal, uint8_t* out);

// And more...

// Decodes the value to the buffer allocated by the caller.
int table_row_decode_to_buffer(TableRow* this, TableColumnRef* column, void* buffer, size_t buffer_size, size_t* output_size, NativeDecodedType* type);

// Wraps void* data, size_t size, and NativeDecodedType type.
struct AllocatedDecodedValue;

// Decodes the value, making its own allocation in the process.
int table_row_decode_to_allocation(TableRow* this, TableColumnRef* column, AllocatedDecodedValue* output);

// Allocates memory for the wrapper struct.
AllocatedDecodedValue* allocated_decoded_value_new();
// Returns allocated memory.
void* allocated_decoded_value_data(AllocatedDecodedValue* this);
// Returns size of memory allocation.
size_t allocated_decoded_value_size(AllocatedDecodedValue* this);
// Returns type of allocated memory for reading.
NativeDecodedType allocated_decoded_value_type(AllocatedDecodedValue* this);
// Calls the proper deallocator.
void allocated_decoded_value_free(AllocatedDecodedValue* this);

The result of cursor::fetch_all() in datajoint-core should be exposed over the C FFI for reading multiple rows at a time and indexing into a vector of rows.

Finish #58, adapting it to the new memory management model in the C FFI

The existing dj implementations in MATLAB and python are both object-oriented, but C-ffi's are necessarily not object-oriented. This will ultimately result in a lot of code duplication as interfaces are built in the target language to bundle up the exposed C functions.

Both MATLAB and python support C++ FFI's, which would reduce this overhead. In python, using e.g. SWIG, one can define a class interface in a C++ header and interface with that class as a regular python class.

namespace dj {
  class Table {
    void insert(*Entries);
    ...
  }
}

class Animal(dj.Table):
  ...

The MATLAB side of things is less ideal, since it does not (currently) support direct inheritance of C++ classes and, even worse, doesn't permit factory constructors. However, one can otherwise interface with C++ objects through the built-in clibgen package. So while building the table interface may require some redundant code, other interfaces like query objects that don't involve user class definitions wouldn't.

Doing this amounts to wrapping the C interface in an additional C++ header, linking to the existing dll, and building with SWIG/clibgen, which is fairly straightforward. An added benefit is that SWIG offers bindings to other target languages including javascript, which you essentially get for free.

var Animal = {...};
Object.setPrototypeOf(Animal, dj.Table);

Wrap C FFI for Cursor class in a Python class

Structs

• ConnectionSettings
• Connection
• Executor
• Cursor
• TableRow
• TableRowVector

Functions

ConnectionSettings

• ConnectionSettings* connection_settings_new();
• void connection_settings_free(ConnectionSettings* self);
• void connection_settings_set_*(ConnectionSettings* self, T value);
• T connection_settings_get_*(ConnectionSettings* self);

Connection

• Connection* connection_new(ConnectionSettings* settings);
• void connection_free(Connection* self);
• int connection_connect(Connection* self);
• int connection_disconnect(Connection* self);
• int connection_reconnect(Connection* self);
• ConnectionSettings* connection_get_settings(Connection* self);
• int connection_executor(Connection* self, Executor* out);
• int connection_execute_query(Connection* self, const char* query, unsigned long long* out_size);
• int connection_fetch_query(Connection* self, const char* query, Cursor* out_cursor);

Executor

• Executor *executor_new();
• void executor_free(Executor* self);
• int executor_execute(Executor* self, const char* query, unsigned long long* out_size);
• int executor_fetch_one(Executor* self, const char* query, TableRow* out);
• int executor_fetch_all(Executor* self, const char* query, TableRowVector* out);
• Cursor* executor_cursor(Executor* self, Cursor* out);

Cursor

• Cursor *cursor_new();
• void cursor_free(Cursor* self);
• int cursor_next(Cursor* self, TableRow* out);
• int cursor_rest(Cursor* self, TableRowVector* out);

TableColumnRef

• TableColumnRef* table_column_ref_new();
• void table_column_ref_free(TableColumnRef* self);
• size_t table_column_ref_ordinal(TableColumnRef* self):
• const char* table_column_ref_name(TableColumnRef* self);
• DataJointType table_column_ref_type(TableColumnRef* self);

TableRow

• TableRow* table_row_new();
• void table_row_free(TableRow* self);
• int table_row_is_empty(TableRow* self);
• int table_row_columns(TableRow* self, TableColumnRef* out_columns, size_t columns_size);
• size_t table_row_column_count(TableRow* self);
• int table_row_get_column_with_name(TableRow* self, const char* column_name, TableColumnRef* out);
• int table_row_get_column_with_ordinal(TableRow* self, size_t ordinal, TableColumnRef* out);
• int table_row_get_*(TableRow* self, TableColumnRef* column, T* out);
• int table_row_get_by_name_*(TableRow* self, const char* column_name, T* out);
• int table_row_get_by_ordinal_*(TableRow* self, size_t ordinal, T* out);

TableRowVector

• TableRowVector* table_row_vector_new();
• void table_row_vector_free(TableRowVector* self);
• size_t table_row_vector_size(TableRowVector* self);
• TableRow* table_row_vector_get(TableRowVector* self, size_t index);

Notes

• T* T_new() functions for structs without default constructors should call malloc to allocate memory.
• NULL checks should be implemented for all self parameters and associated with a common error code.
• All functions that return int return an error code. 0 for success, non-zero for error. No panics allowed. All try_* versions of methods should be called under the hood.

Getters are not properly deallocating strings returned from the C FFI.