This should be an improvement of this:
http://istc-bigdata.org/tiledb/tutorials/index.html#begin-mechanics

So that tiles that are decompressed (and in the future decrypted) can be kept in main memory for subsequent accesses.

There should be the option of encrypting all TileDB data at-rest.

This is a large topic. We will start discussions soon.

This will make our design more recursive (everything that TileDB stores is expressed as arrays, thus inheriting all the functionality).

We will replace this by a better error logging functionality.

Set array (and metadata) schema members via functions. Also create a tiledb_array_schema_check function that checks if the array schema is correct. Get rid of array_schema_c.h and metadata_schema_c.h.

By default, consolidation should happen in an LSM-tree-like way by checking the fragment sizes. The user should be able to set a "step" value in the array schema, which is essentially the fanout of the LSM-tree bottom-up merges.

Some compressors allow for an input buffer size of type int, effectively reducing the size of a buffer to be compressed to 2GB. A step-wise compression in this case is needed. This will affect bookkeeping.

Right now, most of the examples must be run after a certain set of others.

Currently when using the C-API there is no way to tell what the error was (class) without parsing the error string.

It would be nice to print out the currently running test. Also Google Test had a break down of which test passed and which failed. Can we do something similar with Catch?

This will make our design more recursive (everything that TileDB stores is expressed as arrays, thus inheriting all the functionality).

We need to implement a Clang format style guide. I spend way too much time messing around with TileDB's particular style format.

See:https://github.com/kiyo-masui/bitshuffle

Some functions (like the ones related to compression or directory management) should be moved to appropriate places/classes in the code that handle these specific aspects (e.g., a compressor class, or a file system class).

When I try to create metadata, TileDB is giving me the following error (when in debug mode): [TileDB::ArraySchema] Error: Cannot set domain; Lower domain bound larger than its corresponding upper.
It's not actually returning an error code, it's just printing the error. And it seems to be working correctly otherwise.

Since the metadata schema doesn't have a domain that is set (by the user at least), is this something that's crossing over from the data schema?

Make sure it looks good on Doxygen. Check putting all parties in a single copyright statement, versus one statement per party.

currently the tests create a local workspace which is not cleaned up upon failure. This triggers an assertion and causes subsequent tests to fail if the workspace is not removed manually.

This should be an improvement of the Wiki build instructions, the examples we provide under folder examples in this repo and this tutorial:
http://istc-bigdata.org/tiledb/tutorials/index.html#begin-capi

There should be the option of verifying the integrity of the TileDB data at-rest.

Given the following workspace setup:

workspace/test1
workspace/test2/test3

you cannot move from test2/test3 -> test1/test3 as it fails with the error
[TileDB::StorageManager] Error: Move failed; Invalid source directory

The storage manager should not maintain master_catalog metadata for the workspaces.

This will be the default metadata attached to an array, which we will support by adding a simple API.

In the storage manager, the only functionality OpenSSL provides is the MD5 hash function. It would be better to include a standalone version of the md5 hash implementation to simplify the build process.

The current metadata system uses MD5 as it's hash function. As a cryptographic hash function, it has the nice property that collisions are exceedingly rare. This is important as currently there is no key collision handling.

However, there are much faster non-cryptographic functions than MD5 like Murmur hash (and many others). Maybe there is a performance win for meta-data operations to switch to a faster hash function. The metadata schema could also tag the hash function used to derive the keys so that different hash functions can be substituted based on workload.

I don't know what the performance hit is for converting data/metadata to a non-native byteorder but I think we should use little-endian as a standard for all internal data.

This is the array with the simplest possible schema. It constitutes the base case for the recursive array architecture we are building.

The #define's related to the C-api should be moved to tiledb.h. Constants related to the c++ storage manager should be moved there.

In general we need a better separation between the C and C++ layers. The c++ storage manager should be self contained and not be entangled with the C api layer that uses it.

We may need to handle those through API functions.

Right now there is only a single configuration object which is passed as input to tiledb_ctx_init. This should be removed from there and make an API function to attach a configuration object (also handled via API functions) to the context. Similar configuration objects should be attached to arrays, metadata, etc.

Writing to a global buffer is problematic when using multiple threads.

Array Mode, metadata mode, IO method, etc should be enums instead of preprocessor defines. It makes debugging a bit nicer to see the enum name and not its value.

They kind of seem redundant, since their functionality are the same as groups.

The configuration should be pushed at the array level, and implemented as a BasicArray.

I should be thread-/process-safe.

Upon exercising the error path in TileDB::Fragment, subsequent tests trigger an assertion.

Error:

======================================================================
ERROR: test_workspace_list (test_libtiledb.LibTileDBTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jacobbolewski/anaconda3/envs/tiledb/TileDB-Py/tiledb/tests/test_libtiledb.py", line 191, in test_workspace_list
    libtiledb.workspace_create(ctx, wrk2)
  File "tiledb/libtiledb.pyx", line 468, in tiledb.libtiledb.workspace_create (tiledb/libtiledb.cpp:7020)
    raise TileDBError()
tiledb.libtiledb.TileDBError: [TileDB::Fragment] Error: Cannot rename fragment directory; Directory not empty

----------------------------------------------------------------------

Assertion after error:

(tiledb) tiledb/TileDB-Py [jcb/cytiledb●] » python -c "import tiledb; tiledb.test()"
test_delete (test_libtiledb.LibTileDBTest) ... Assertion failed: (starts_with(stripped_fragment_name, "__")), function sort_fragment_names, file /Users/jacobbolewski/TileDB/TileDB-dev/core/src/storage_manager/storage_manager.cc, line 2394.
[1]    4100 abort      python -c "import tiledb; tiledb.test()"

(tiledb) tiledb/TileDB-Py [jcb/cytiledb●] » python -c "import tiledb; tiledb.test()"
test_delete (test_libtiledb.LibTileDBTest) ... Assertion failed: (starts_with(stripped_fragment_name, "__")), function sort_fragment_names, file /Users/jacobbolewski/TileDB/TileDB-dev/core/src/storage_manager/storage_manager.cc, line 2394.
[1]    4114 abort      python -c "import tiledb; tiledb.test()"

Clearing the ~/.tiledb directory causes the error to go away, but some bit of state is not being cleaned up upon hitting the original error path.

Currently, each array spawns and manages an AIO thread. We must have a single (or one per disk) thread at the storage manager for handling all AIO.