olange / data-pipes Goto Github PK

Next to the description of the schema of data literals, we would need to describe transformations of theirselves. The ability to express data transformations in the data plane promotes:

• reusability of data;
• reduced network access.
• immutability, until data leaves the pipeline (as long as the transforms provide such guarantee).

Figure out a way to express transformations in data-pipes.

Challenges

Higher-level data-structures challenges: we would like the ability to use/handle/transport metadata and schema — although transforms might leave metadata and schema in terminals only — and may be, provide immutable data-structures as a result (however, how to handle streaming data, and how-to traverse the worker context?), or may be just frozen objects.

Directions

Start with the Streams API, as it provides transformation of binary data and chunked data (aka streaming). As well as service workers, web workers, transferable objects and indexed DB.

Motivation

Usually, when client-side data transforms reach a certain degree of sophistication, developers move them to the middle- or back-end.

In the case of complex back-end APIs, moving transforms to these tiers might be either not feasible, or unpractical, requiring extensive testing. For instance, Cloud Firestore offers a rich set of near real-time synchronization features of mobile and web apps with the datastore, traversing CDNs, yet its querying API is limited — though already powerful.

Hence the need to transform them in the front-end — however we lack a universal API for such transforms, including immutable data-structures where possible, as Clojure would provide.

Ability to transform data in Javascript is scattered around:

• JS native data-structures and APIs (Map/Set/upcoming Record);
• Immutable data-structures and API (ImmutableJS);
• JS abilities (Transferable Objects, Structured data cloning);
• JS chunked binary data processing (Streams API);
• Web Structured-data Storage API (IndexedDB);
• Libraries, such as underscore, lodash, DataScript, …

And have I forgot to name XSLT and XPath? XSLT expresses data transforms and is available to browsers — however this technology geared towards transforming XML documents.

Find a way to attach metadata to our data-structures.

Directions

Attach a metadata Symbol to data-structures. Use only native objects (Boolean, Number, String, …) to represent terminal values, instead of built-in primitive values – see first comment hereafter; good idea, but they do not pass in/out workers.

Motivation

• Ability to describe from where an object originates;
• Ability to describe the version of an object, and its creation time;
• Ability to describe how it was factored – which logic rules applied to automatically define that an accounting statement might be linked to a certain account to its debit; or that the account was instead provided by the user, overriding an automatically set value.

Description of the schema of returned data serves following purposes:

• automatically format data values, based on types and locale;
• automatically generate grids for recordsets;
• generate fields based on type and formatting hints.

Schema description should circulate in following forms:

• as an simple yet expressive DSL, allowing front-end developpers to provide the schema description explicitely;
• as an immutable data structure, within a front-end, that can be easily navigated and used by code, and that matches the shape of data.

Ideally, it should be possible to produce such a schema description automatically:

• either by deriving it from data providers that can deliver a schema, namely GraphQL servers;
• or by inferring it from the data itself.

The DSL should also support optional annotations, that would allow to:

• refine formatting of values (for instance, limit length of a String type to 25 chars);
• override schema description automatically inferred.

Prefer schema description languages that mimic type systems that provide support for typing of data literals, such as GraphQL or Flow.