rlay-project / rlay-utils Goto Github PK

The .remoteCid flag, introduced pre-v0.2.0, is a way to signal if the entity exists remotely or not. However, when casting between entities, especially higher-level entities as exposed by rlay-ontology packages, the default absence of the .remoteCid flag causes ugly code. E.g.

newEntity.remoteCid = oldEntity.remoteCid

As @hobofan suggested, the presence of such a flag may just be cosmetically and not necessary for any of the methods (e.g. assert) to work properly. As a first instance we will therefore try to remove the .remoteCid flag to see if any behavior changes occur.

As a second step, the entity could hold state about which assertions it pass through to the client to optimize some of its behaviors

@MichaelHirn Can we remove the esformatter dependency? It seems that it's not really used and it has an open security vulnerability (millermedeiros/esformatter#497).

The current CI build seems to be broken due to the webocket dependency introduced through web3. Luckily by now the 1.0-beta release of web3 have been released as the 1.2 branch, where this issue should be fixed.

See: https://github.com/ethereum/web3.js/releases/tag/v1.2.1

Currently rlay-client-lib is written in ES6 which is weakly typed. For the sake of maintainability and extensibility we should switch to TypeScript which is strongly typed and reduces burdens on tests and makes hard to debug errors less likely.

This should happen before any other v0.3 related work.

At the moment there are no tests that guarantee that multiple assertions of the same Data- or ObjectProperty but with different values are supported. My guess is, it doesn't work yet. But e.g. for rlay-transform it is important that it works.

Create/Assert

this is not tested but educated guess is, that it should work:

await person.assert({
  favoriteAnimals: 'horse',
  favoriteAnimals: 'deer'
});

but add support for this as well

await person.assert({
  favoriteAnimals: ['horse', 'deer']
});

Resolve

Assume we created/asserted an individual like this:

await person.assert({
  favoriteAnimals: 'horse',
  favoriteAnimals: 'deer'
});

then await person.resolve() most likely won't give the desired output and will just be

assert.equal(person.favoriteAnimals, 'deer');

but should be

assert.deepEqual(person.favoriteAnimals, ['horse', 'deer']);

Especially Individuals and standardize a bit better so that implementing higher-level entity factories becomes easier and requires less overhead.

EntityFactory currently has a prepareRlayFormat method that is applied to the payload or payload part that comes in as input for the .create and .from methods.

The purpose of that is that higher-level entities can be created, e.g. a DataPropertyAssertion that enforces a property cid or a subject cid like 0x00.

The purpose of the prepareRlayFormat and extended attributes and methods is three-fold and a bit tangled up atm: validate schema, default values, enforce values.

But in its current form it introduces a lot of code that is hard to test, not fully tested, and introduces a big amount of complexity.

Would be great to think more about this and refactor it in a future release.

With the arrival of v0.2.0 it becomes easier (recommended) to separate application ontologies into their own npm package which exposes (if it should be reused in other packages) their auto-generated client. It is also custom behavior that these rlay-ontology packages expose their own classes which provide higher-level interfaces to their ontology. Example rlay-ontology package here: https://github.com/MichaelHirn/rlay-ontology-airtable

However, to consume these rlay-ontology packages i.e. require them in rlay-ontology packages and applications and use their ontology and high-level interfaces, is non trivial and requires manually changing the auto-generated client of the application.

To prevent that, we need to change two things:

• integrating the schema of rlay-ontology packages
• integrating the higher-level interfaces of rlay-ontology packages

Integrating the schema of rlay-ontology packages

The proposed solution is to update rlay-seed to produce an index of entityKey => entityPayload instead of entityKey => entityCID as it is currently implemented; an update of rlay-generate to only require the output of rlay-seed (and no path to any ontology file) or even work on the input of rlay-seed. Then rlay-ontology packages have to expose their src/ontology.js file which module.exports a rlay-seed JSON object. In the application that ontology JSON object can then be required and dropped into the imports attribute of its own rlay-seed JSON object. rlay-seed needs to accept payloads instead of only cids for its imports attribute.

If you want your rlay-ontology package to be required by other packages, you export the output of rlay-seed (aka. the ontology file's rlay-seed JSON object), otherwise don't.

Integrating the higher-level interfaces of rlay-ontology packages

A bit more unclear but orientate ourselves on plugin systems like those from babel, eslint, etc.

cc @hobofan

Example:

Assume CID 0x123 responds to payload { type: 'DataProperty', ...}.

The proposed behavior for .find('0x123') is:

client.Entity.find('0x123'); // does not throw, returns `entity instanceof Rlay_Entity`
client.DataProperty.find('0x123'); // does not throw, returns `entity instanceof Rlay_DataProperty`
client.Individual.find('0x123'); // does throw; error message: expected payload to be of type 'Individual' but got 'DataProperty'. Call client.DataProperty.find.
client.{X != DataProperty || Entity}.find('0x123'); // does throw; same as client.Individual.find

The point of this is, that you can assert a specific Entity type when using find.

If you want to fetch it without knowing the type of the entity one should call client.Entity.find

Some naive assumption of an existing key 🙄:

I passed in ['0x...'] by accident instead of '0x...' and the resulting error is misleading, check before if input is a string at least.

As of right now, rlay (server) and the rlay-ontology does support any functionality to restrict the schema of an ontology. For example: Only http:Connection can have ObjectPropertyAssertion of type http:requests. W3C provides SHACL for this use-case and GraphQL has it's own schema definition language.

But, rlay-client-lib with it's generated client supports some very basic way of setting fields and defaults, which are used when .assert and .create are called. It seems that this could be a good start to get schema validation sorted, at least for the client. The functionality could well be provided by a third-party library (research and dd has not been untertaken so far) and could then be implemented very quickly.

However, so far support for schema validation has not been a blocker in any way and it seems like a nice-to-have further down the road. So this, issue is just for documenting with lowest priority.

Simple Summary

rlay-generate is a CLI command, that auto-generates a convenient js client to interact with rlay and the underlying database based on the schema of the application.

Example

Schema

HTTPConnection Class
HTTPRequest Class

payload DataProperty

requests ObjectProperty

Specifications

Create Entity

createX (body): where X is the name of a Class and body is an Object where its keys are the names of the DataProperties or ObjectProperties.

const rlay = require('./generated/rlay-client')

// Create a new HTTP connection
async function main() {  
  const newUser = await rlay.createHTTPConnection({ 
   $inherent: { id: 'abc' },
   payload: JSON.stringify(data)
  })
}

Query Single Entity

X (body): where X is the name of a Class and body is an Object where its keys are the names of the DataProperties or ObjectProperties.

const rlay = require('./generated/rlay-client')

// Create a new HTTP connection
async function main() {  
  const newUser = await rlay.HTTPConnection({ cid: cid })
}

Query Multiple Entities

Xs (body): where X is the name of a Class and body is an Object where its keys where and the value is a QUERY instance.

const rlay = require('./generated/rlay-client')

// Create a new HTTP connection
async function main() {  
  const newUser = await rlay.HTTPConnections({ where: QUERY })
}

Update Entity/Entities

updateX (body): where X is the name of a Class and body is an Object where its valid keys are where and the value is a QUERY instance and data and the value is an Object where its keys are the names of the DataProperties or ObjectProperties .

const rlay = require('./generated/rlay-client')

// Create a new HTTP connection
async function main() {  
  const newUser = await rlay.updateHTTPConnection({
    where: QUERY,
    data: { payload: JSON.stringify(data) }
  })
}

Delete Entity/Entities

deleteX (body): where X is the name of a Class and body is an Object where its valid keys are where and the value is a QUERY instance and data and the value is an Object where its keys are the names of the DataProperties or ObjectProperties .

const rlay = require('./generated/rlay-client')

// Create a new HTTP connection
async function main() {  
  const newUser = await rlay.updateHTTPConnection({
    where: QUERY,
    data: { payload: JSON.stringify(data) }
  })
}

Open Questions:

• What is a QUERY instance?
• How do you handle more complex QUERYs, e.g. IS NOT EMPTY, >=, INCLUDES
• How do you handle pagination? (E.g. when QUERY result is large)

Implementation

Reference: #1

Follow up to #1 .

Repo to reproduce: https://github.com/hobofan/rlay-utils-repro/tree/fe349c4ca80ec79c22b2b2200ab9ac1c96a63d8f

Follow up to #1: #1 (comment)

When passing a seemingly unknown property, it just ignores that silently and creates an empty Individual

Summarizing the discussion @hobofan and I have had regarding namespaces in ontologies.

In RDF we have something like rdf:string and schema:string, where different ontologies - rdf and schema in this example - can have the same ontological entity (schema) name - in this case string. Important to note is, that although they share the same entity name they may differ semantically (and rdf:string can mean something else than a schema:string).

The namespace allows developers and users of these ontologies to use multiple ontologies in their application, without fearing any name-clashes, esentially giving guarantees of resolvability via a human-understandable ID and not only via the CID.

Presently, rlay does not support any namespace capabilities and it is up to the application developer/ontology user to ensure that different ontologies (or their own ontology) does not clash. The way this can be ensured right now, is by taking the namespace into the name of the ontological entity, e.g. StringRDF or StringSchema.

However, this makes sharing ontologies difficult, because of the implied naming convention. Also, this does not make nameclashes impossible either.

Proposed solutions are referring to npm, Annotations, and more or less rlay custom annotations to introduce namespaces. Namespaces are closely related to sharing and importing ontology packages, so it might make sense to combine multiple strategies to achieve this.

However, no clear winner emerged in the last discussion, which is why we created this issue, to collect thoughts and allow to refresh our memories on the subject.

Abstract

This is a proposal for a standardized mechanism for annotating non-assertive entities such as Class, Annotation, DataProperty, ObjectProperty, and more to have a bi-directional way from a unique human-understandable ID - a Rlay Schema Name - to its CID without requiring a lookup table of some sorts.

Proposal

This document proposes the following four rlay-built-in AnnotationProperties: RlayNamespace, RlayName, RlayType, RlayFormat. For every non-assertive entity such as Class, Annotation, DataProperty, etc. four Annotations, one for each AnnotationProperty, must be created and passed along as inherent annotation properties.

We can encode a RSN in the following format:

[rlayNamespace]:[rlayName]:[rlayType]:[rlayFormat]

For example:

http:Request:ObjectProperty:puppeteerRequest

The corresponding CID of the ObjectProperty can now be computed without requiring a roundtrip or lookup table.

Defaults

As the full RSN can be cumbersome to type, especially if there is only one RlayType or RlayFormat for any given RlayNamespace + RlayName, it should be able to omit them.

Only RlayNamespace and RlayName are required.

RlayNamespace

#3 covers an initial conversation around namespacing ontologies (schemas), without reaching a conclusion. However, this proposal could tie the two routes of annotation and npm together, where the RlayNamespace comes from the npm package name + version.

.resolve, introduced with v0.2.0, fetches class, data, and object assertions (property and assert assertions) for an individual, decodes them, and attaches them to the individual so that they can be accessed via entity.customNameOfTheSchemaAssertion.

Many methods, especially those from higher-level factories like those introduced by rlay-ontology packages, require that individual was resolved first. However, there is no single mechanism to check if an entity is resolved or not, which leads at its best to ugly and non-obvious code. E.g.

if (entity.properties instanceof Object) {
  await entity.resolve()
}

There are also several related issues, such as mental overhead when coding to keep in mind if an entity is resolved at that instance, or in another route run-time overhead when a resolved entity is resolved again, etc.

Solutions

smart resolve

A simple solution would be to add a entity.isResolved() method which is just a nice way to basically do what the non-obvious code from above does. Then update entity.resolve() to only call out to rlay-client if entity.isResolved() is false. This makes calling entity.resolve() cheap reduces boilerplate to a minimum. However, .resolve would need a forceFlag so that a resolve can be triggered when for example the entity has received more assertions in the meantime. This can be limited to only needing to call entity.resolve({force: true}) when an application elsewhere might have asserted something about the entity, by integrating .resolve into .assert. However, this might needs more thought.

ResolvedEntity class

A potentially more elegant solution where any Individual can be cast into a ResolvedIndividualEntity, which must not come with the core rlay-client-lib. Might have similar logic to the first smart resolve proposal. I think smart resolve might be the better way to go here, at least for now, as .resolve is so often used and core.

Calling .resolve on an entity (Individual) leads to all (property and assert) assertions being fetched, decoded, and attached to the entity. However, if there is only assertion that has the same assertion property it will be attached like this entity.customKeyAssertionName instanceof Object || instanceof {String|Boolean|Numeric|...} and if there are two or more assertions that share the same assertion property it will be attached like this entity.customkeyAssertionName instanceof Array.

This causes ugly boilerplate when trying to use the value and to fix that we need to make it always return an iterable object e.g. an array. That object may have some helper methods like .getFirst to make it easier to access when there is only a single assertions

calling individual.resolve() results in individual.properties including non-properties if that individual was used as target in an ObjectPropertyAssertion.

Problems seems to be in the neo4j query

May work already, but is not yet tested.

Is a prerequisite for aggregating over assertions that are contradicting each other.

Not 100% sure, but I think the rlay-client-lib devDependency should be a path dependency to the directory, which would help with keeping it in sync.

See:

Related: #70