Giter Club home page Giter Club logo

typedbclient.jl's Introduction

TypeDB Client for Julia

Stable Dev Build Status BDD codecov

This is a community approach to translate a given client interface to the knowledge graph database TypeDB, made by Vaticle, into Julia Language.

Starting out as an educational project and to benefit the Julia ecosystem.

Please review the user guide or if you'd like to help building this check the contribution guidlines in the docs.

What is TypeDB about

Read up here: https://vaticle.com/

TypeDB is the knowledge graph engine to organise complex networks of data and making it queryable, by performing knowledge engineering. Rooted in Knowledge Representation and Automated Reasoning, TypeDB provides the knowledge foundation for cognitive and intelligent (e.g. AI) systems, by providing an intelligent language for modelling, transactions and analytics. Being a distributed database, TypeDB is designed to scale over a network of computers through partitioning and replication.

Announcements

TypeDB 2.0 has been released, these are the highlights of the new version:

Replaced Cassandra with RocksDB
New Graph Storage Engine: replacing JanusGraph
New TypeDB Type System: our Knowledge Representation
New Graql: even simpler and more powerful
New Traversal Engine: replacing TinkerPop/Gremlin
New Query Planner: an Integer Linear Program
New Reasoning Engine: based on Event Loop + Actor Model
New Query Engine: an Asynchronous Producer-Consumer
New Client-Server Protocol: a Reactive Stream
New TypeDB Cluster: a Raft based distributed TypeDB
New TypeDB Console: powered by PicoCLI + JLine
New Benchmarking System: an Agent-based Simulation
New Grabl CI/CD: replacing CircleCI

Read about it here in the forum.

How to start

Check the docs for install instructions and examples (WIP)

For further questions about how to start, ideally find us on Discord.

The TypeDB community also has a Discord server here.

Please find open issues that look for help in order to contribute.

Project status

The current project status can be seen on the GitHub Project Kanban.

Contributions are welcome!

typedbclient.jl's People

Contributors

azzaare avatar frankurbach avatar github-actions[bot] avatar jakewilliami avatar mkschulze avatar scottpjones avatar tk3369 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

typedbclient.jl's Issues

gRPC implementation

The Grakn core server communicates via gRPC with it's clients.

Currently there is no working gRPC library in Julia. There are gRPC.jl or GRPC.jl, but these need to be checked and updated to a working state.

See issue #19

The long term goal would require a native Julia implementation of gRPC.
There has been several attempts to find people doing it here: #grpc/grpc#24786
These let to an offer to get help facilitate the work, clarify requirements, do API reviews, set up interop tests as mentioned in the issue.

And here: https://discourse.julialang.org/t/grpc-api-community-request/51029
which led to no avail yet.

I also wrote a mail to @tanmaykm, he replied and might give it a shot to get existing stuff working back again (no promise), but may not be able to spend a lot of time on it. He would be happy if anyone else is willing to contribute.

gRPC is supposed to be a non trivial protocol, so I think it doesn't seem feasible to tackle this first and wait for completion until we can start with the GraknClient.
Bildschirmfoto 2021-01-17 um 00 25 06

Instead we should find an interim way by either calling into the Python gRPC library with Python or using Julias native C calling functionality to call into the shared gRPC core library, which is used by all client implementations. I guess this would require additional work, as the C lib seems to be more low level code.

Quickest solution to get started with developing the GraknClient.jl seems to be to call into Python and then see if somebody wants to take a leap on implementing/revamping a Julia native gRPC lib in the long run to eventualy avoid the Python call.

Project status

Hi :)

I am currently working on the Wikipedia page of graph based databases.

TypeDB has/had currently only Java declared as compatible language.

I attend to change this, and I am currently uncertain, if this project here is alive and well maintained.

Can you clearify?

BDD implementation for testing

Graknlabs follows the Behaviour driven development (BDD) principle. It would be great to also have this imlemented at some point.

bdd

There is the Grakn Behaviour Specification and there is a native Julia package called ExecutableSpecifications.jl and some talk in this issue #erikedin/Behavior.jl#38

Some things that the Grakn features and steps require, would need some work on this ExecutableSpecifications.jl to be able to use that in our client.

This could maybe end up in a small stipple.app to show the BDD results of the tests like in the picture below:
Bildschirmfoto 2021-01-17 um 01 32 21

Or we might even be able to use Grabl at some point.

In this Grakn 2.0 announcement 13). post it has been mentioned, that it would be available to 3rd partys as well some day.

BDD development mega issue

This is a summary issue that keeps track of the work involved for implementing the BDD test suite for GraknClient.jl (for background, see issue #22). Note that ExecutableSpecifications.jl needs to be enhanced to support this project, and that's being tracked separately.

We should be able to largely model the work from Grakn's python-client. The links below point to that repo for reference.

HTTP/2 protocol in Julia

GOAL
In order to be able to eventually have a grpc library available in Julia, either via writing a new one or revamping gRPC.jl or GRPC.jl, there would need to be a way to make use of the HTTP/2 protocol.

Current situation and possible solutions
There are several solutions that have been started, but are either not completed or not maintained anymore.

Solution 1:
HTTP2.jl
This library is not maintained anymore.

It would require the HPack.jl 0.2.0 (Julia v1.0 compatibility), which itself is not maintained anymore. It looks like it already has been done maybe here in this tree, but has never been released.

The author @samoconnor is not active on GitHub anymore, his last contribution is from March 2019.
Solution: HTTP2.jl could be forked and setup to be working again.

Solution 2:
There is HTTP.jl
@samoconnor started to integrate the HPack.jl into HTTP.jl here

And there is an open PR here: #JuliaWeb/HTTP.jl#331
I asked @quinnj on Slack about the state and he wants to look into it and estimate the required workload and if it would need sponsoring or not.
Solution: We could wait for his response and decide, wether it makes sense to finish that PR or not or look for sponsors or smth.

Solution 3:
Calling into C
Somebody mentioned this MIT licensed pack to be a solution in this issue: #JuliaWeb/HTTP.jl#448
It is about this technology: litespeed-quic-http3-open-source

This is the pack, not sure if HTTP/3 or QUIC actually offer the HTTP/2 parts
https://github.com/litespeedtech/lsquic

But there is the C compression library here
Found on their official site here

Solution: This could be called within Julia.

Solution 4:
Calling grpcio from Python via PyCall and don't care for a Julia implementation for now.
Grakn themselves use the grpcio for their Python Client.

BDD example implementation

It would be a great help for me to have a few reference examples to work with.

The feature file I reference is here: https://github.com/Humans-of-Julia/GraknClient.jl/blob/BDD--steps/test/behaviour/features/concept/thing/attribute.feature


Connection example

  1. Gherkin
Feature: Concept Attribute

  Background:
    Given connection does not have any database
  1. Python
@step("connection does not have any database")
def step_impl(context: Context):
    assert len(context.client.databases().all()) == 0
  1. Java
 @Given("connection does not have any database")
    public void connection_does_not_have_any_database() {
        super.connection_does_not_have_any_database();
    }
  1. Julia # this should be correct perhaps
@given("connection does not have any database") do context
    @expect length(get_all(databases(context[:client]))) == 0
end

Put attribute example

  1. Gherkin
  Background:
    # Write schema for the test scenarios
    Given put attribute type: is-alive, with value type: boolean
  1. Java
  @When("{var} = attribute\\( ?{type_label} ?) as\\( ?boolean ?) put: {bool}")
    public void attribute_type_as_boolean_put(String var, String typeLabel, boolean value) {
        put(var, tx().concepts().getAttributeType(typeLabel).asBoolean().asRemote(tx()).put(value));
    }
  1. Julia
@when("{var:Var} = attribute({type_label}) as(boolean) put: {value:Bool}") do context
    context[Symbol(var)] = [:tx(), :concepts(), get_attribute_type(:type_label::String), :as_remote(context[:tx()]), :as_boolean(), :put(value::bool)]
end

especially this is pretty surely wrong:
[:tx(), :concepts(), get_attribute_type(:type_label::String), :as_remote(context[:tx()]), :as_boolean(), :put(value::bool)]


entity Set owns attribute example

  1. Gherkin
Feature: Concept Attribute

  Background:
    Given put entity type: person
    Given entity(person) set owns attribute type: is-alive
  1. Java
    @When("{root_label}\\( ?{type_label} ?) set owns attribute type: {type_label}")
    public void thing_type_set_has_attribute_type(RootLabel rootLabel, String typeLabel, String attributeLabel) {
        AttributeType attributeType = tx().concepts().getAttributeType(attributeLabel);
        get_thing_type(rootLabel, typeLabel).asRemote(tx()).setOwns(attributeType);
    }
  1. Julia suggestion from suggestmissingsteps
@given("entity(person) set owns attribute type: is-alive") do context
    @fail "Implement me"
end
  1. Julia
@given("{root_label}({type_label}) set owns attribute type: {type_label}") do context
    @expect "Implement me"
end

Script prepare the translation from the java client to Julia

The attached script runs not without preparing the right paths individually for your machine. Today, the command of preparing the generated file from proto-files didn't work. This should be done by hand. Above and below this every line can be executed after preparing the right paths. Because the issues don't allow attaching Julia files I will place the script file in the translate_java directory in the project GraknClient.jl.
This issue can be closed after the final translation is done.

Stable connection to server for longer times during work with session

In Julia versions > 1.7.2 the every 4 seconds needed ping to the database server isn't stable after a few minutes of running a session. Above the mentioned Julia version something was changed in the Thread mechanism where the ping lives in the lifespan of the session. As it seems to be the grpc client isn't thread safe in conjunction with the thread mechanism of Julia above 1.7.2

protobuf files

As can be seen in the picture below and can be read up in the docs here, we would need special .proto files to make the client be able to communicate with the grakn-core server.

Bildschirmfoto 2021-01-17 um 00 25 06

@jakewilliami has started to create these files here by using the Protobuf.jl but the automatic generation failed like this:

┌─[usr@vs-214-01-l: generated] [main ?] └──╼ $ protoc -I=proto --julia_out=. proto/session.proto protobuf/options.proto: File not found. session.proto:23:1: Import "protobuf/options.proto" was not found or had errors. session.proto:45:13: "Options" is not defined.

But options.proto should be found:

`┌─[usr@vs-214-01-l: generated] [main ?]
└──╼ $ tree
.
├── concept_pb.jl
├── database_pb.jl
├── grakn.jl
├── options_pb.jl
└── proto
├── Build
├── answer.proto
├── concept.proto
├── database.proto
├── options.proto
├── query.proto
├── session.proto
└── transaction.proto

1 directory, 12 files`

In the proto directory are all of the original protoc files here

These would also need to be updated to the latest versions now, in case we want to generate the files in pure Julia.

However, there is a grakn-protocol repo here
that is also on PiPy and compatible with the 2.0 alpha here.

Decision has to be made, on how we go on with this as well.

AttributeType constructor should run with Label

To build a Attribute the building block for the constructor is a Label. Until now the constructor requires a string that isn't given by the function calling the constructor. Excuse the ending txt in the pinned file. But GitHub didn't support .jl files.
AttributeType.txt

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.