Giter Club home page Giter Club logo

terminuscms's Introduction

TerminusDB Logo

CI Issues

TerminusDB is a distributed database with a collaboration model.

It is designed to be like git, but for data. If you are reading this, give this repo a star.

The building blocks of the model are -

  • Revision Control: commits for every update
  • Diff: differences between commits can be interpreted as patches between states
  • Push/Pull/Clone: communicate diffs between nodes using push / pull / clone
  • Query: You can query any state of the database at any commit.

TerminusDB allows you to link JSON documents in a knowledge graph through a document API. TerminusDB is available as a standalone server, or you can use our headless content and knowledge management system TerminusCMS.

TerminusDB Version 11

TerminusDB 11 features a new Rustified storage backend that reduces storage overhead and latency, improves search performance, and simplifies interchange. TerminusDB 11 also comes with some exciting features to make building easier and faster -

  • GraphQL - Use GraphQL as a proper graph query language with deep link discover and path queries
  • Added @unfoldable document flag to frames - Making data curation easier by unfolding subdocuments within a frame to add all relevant data in one place
  • Add @metadata to frames - Include additional metadata to document frames including data formatted as Markdown.

Installation Guide

Get it from the Snap Store

The easiest way to install TerminusDB as a developer is by using the Snap. It does not provide a daemon and is mainly intended for developers that want to try TerminusDB.

For deployments, or if you want to use the local dashboard:

  1. Add the following to a .env file in the source directory:
OPENAI_KEY=YOUR_OPENAI_KEY_HERE
# And optionally specify number of pages for the vector database
# for instance
BUFFER_AMOUNT=120000
  1. docker compose up

You should be able to view TerminusDB running by default at localhost:6363

If you're installing TerminusDB on Windows with Docker, our friends at DFRNT wrote this comprehensive guide.

You can also install TerminusDB from the Source Code.

TerminusDB CLI

A simple example creating a person with friends can be created as follows:

terminusdb db create admin/example1
terminusdb doc insert --graph_type=schema admin/example1 <<EOF
{ "@id" : "Person",
  "@type" : "Class",
  "name" : "xsd:string",
  "occupation" : "xsd:string",
  "friends" : { "@type" : "Set",
                "@class" : "Person" }}
EOF
terminusdb doc insert admin/example1 --message='adding Gavin' <<EOF
{ "@type" : "Person","name" : "Gavin", "occupation" : "Coder"}
EOF

You can attach this to a database on TerminusCMS in order to distribute it by setting your remote. First you will need to create a resource from TerminusCMS. This is analogous to creating a repository on GitHub.

Log into TerminusCMS, create a new data product, and make sure you have an access token to the team in which we created that data product. Then copy the URL to clone from the data product info page.

If you make a data product called example in the team Terminators. We could then issue the following command using the TerminusDB CLI. You should get an API token from your profile page and replace 'XYZ' with this token.

terminusdb remote add admin/example1 origin 'https://cloud-dev.terminusdb.com/Terminators/example' --token='XYZ'
terminusdb fetch admin/example1 --token='XYZ'
terminusdb push admin/example1 --token='XYZ'

Once completed, you'll have a remote copy of this database.

More on the CLI here

Community

Come visit us on Discord or our community Subreddit. On Twitter, we're @TerminusDB.

Documentation

Check out our documentation site for more information: TerminusDB documentation website.

We are working hard to improve our docs - if you see an issue, please open an issue in the documentation repo.

White paper on our delta-encoding approach to data management.

Check the Python Client ๐Ÿ. Or the JavaScript Client ๐ŸŒ.

WOQL is a powerful query language that allows you to express complex patterns over arbitrary data structures concisely. What makes it so expressive and easy to use is the radical simplicity of the core underlying concepts.

Changes in this Version

Release Notes

Copyright

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

terminuscms's People

Contributors

oj423 avatar spl avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

terminuscms's Issues

Squash should have a warning

This breaks the history, and makes it so that a force fetch / pull option will be required. We currently do not have a force pull option.

Make it possible to change multiplity of properties when allowable

Is your feature request related to a problem? Please describe.
Currently it is possible to switch multiplicity in-between {optional, mandatory, set}, but not between them and {list}, provided no instances violate the multiplicity constraints of each. It would be helpful for schema editors to allow casting any of {optional, mandatory, set} to a list and vice versa, provided the list elements can be successfully casted into a set.

The distinction between a set and a list is not always clear or known to users ahead of time, and may also change over time as use evolves. It would be helpful to allow movement between them as a nice-to-have enhancement.

Describe the solution you'd like
Allow casting properties from any of {optional, mandatory, set} into a list and vice versa

Describe alternatives you've considered
Export, deletion of data and importing into the new format would be the workaround to not having this feature.

Schema checking before subdocument deletion fails dependency checks (resulting in inconsistency)

Describe the bug
Schema checking before subdocument deletion fails dependency checking, to detect documents that include the subdocuments in their schema.

To Reproduce
Steps to reproduce the behavior:

  1. Go to TerminusX
  2. Create a subdocument and put it as an optional linked property of a document
  3. Try deleting the subdocument in TerminusX (not possible as the trashcan icon is correctly greyed out)
  4. Delete the subdocument through the API
  5. Refresh TerminusX browser client
  6. TerminusX is inconsistent between what is in the JSON (reference to subdocument is still there), and what is shown in the UI (reference to subdocument is gone)

Expected behavior
Schema reference checking should fail with the subdocument not deleted and an error 400 returned.

Screenshots
Property not shown after manual deletion through the API
image

But, the property is still there in the JSON
image

Info (please complete the following information):

  • OS: macOS Edge Chromium and manual API connection
  • Connected to TerminusX 2021-12-31

Additional context
N/A

Machine tokens for web services and automated processes

Currently, all users in TerminusX are presumed to be actual people, with their own e-mail adress, or some account at another service. However, a major use case for a database is to be able to access it in some automated fashion, such as from a web service, or from some automated process.

To support this, a user should be able to create a machine token that a script or service can use to authenticate with. This token should give access to a scope configured by the user, which could be equivalent to the scope of things the user is able to access, but more likely is to be reduced to a more limited scope, such as a single database, or even a limited set of permissions on a single database.

TerminusX dashboard - Product Explorer: Query reformatted and query edits lost

Steps:

  1. Copy/paste WOQL from Start with a Client:
WOQL.triple(
      "v:Player",
      "position",
      WOQL.string("Full Back")
    ).triple("v:Player", "position", "v:position");
  1. Switch to JSON-LD view then switch back to JS view. Query changes to:
WOQL.and(
    
    WOQL.triple("v:Player", "position", {"@type":"xsd:string","@value":"Full Back"}),
    
    WOQL.triple("v:Player", "position", "v:position")
)
  1. Add a comment to the query:
WOQL.and(
    
    // This is a test comment
  
    WOQL.triple("v:Player", "position", {"@type":"xsd:string","@value":"Full Back"}),
    
    WOQL.triple("v:Player", "position", "v:position")
)
  1. Repeat step 2. The Comment is removed:
WOQL.and(
      
    WOQL.triple("v:Player", "position", {"@type":"xsd:string","@value":"Full Back"}),
    
    WOQL.triple("v:Player", "position", "v:position")
)
  1. Similarly, if you comment out parts of the query then repeat step 2, those parts are also lost, an example below:
WOQL.and(
      
    // WOQL.triple("v:Player", "position", {"@type":"xsd:string","@value":"Full Back"}),
    
    WOQL.triple("v:Player", "position", "v:position")
)

Resulting query:

WOQL.triple("v:Player", "position", "v:position")

Unable to send feedback

Describe the bug
I get an error message when trying to send feedback from TerminusX

To Reproduce
In the TerminusX gui, click on the envelope in the bottom left corner. Fill in subject and message. Click Send, error message appears. Browser console says: Unchecked lastError value: Error: Could not establish connection. Receiving end does not exist. 2 background.js:2455

Expected behavior
Message to send through with confirmation.

Screenshots
image

Info (please complete the following information):

  • OS: Pop!_OS 21.04
  • Browser: Firefox 94.0

TerminusX dashboard - JSON elements created in different order to schema elements

When switching to the JSON view, the sequence in which the JSON elements are created differs from the schema elements. This applies to enums too. An example below. The Schema is visually shown in the correct, logical order - enums grouped to the left, documents and sub-documents on the right in correct hierarchical order. However, the JSON view, shown further below, has no logical order:

  • Enum definitions are separated by several document definitions.
  • Sub-classes are defined before parent classes.
  • The parent class is defined somewhere between.

Visual schema

image

JSON definition

image

TerminusX WOQL Editor missing

Describe the bug
In TerminusX, the 4th tile on the left that previously hosted the WOQL editor pane is gone.

To Reproduce
Steps to reproduce the behavior:

  1. Go to dashboard.terminusdb.com
  2. Login
  3. On the left, the 4th tile with the WOQL editor is missing

Expected behavior
I should be able to test WOQL in the dashboard.

Screenshots

image

Info (please complete the following information):

  • OS: Macos. Edge Chromium browser
  • How did you run terminus-server TerminusX

Additional context
Add any other context about the problem here.

Squash description is not optional but says it is

When squashing a branch, an option is given to give a description for the squash commit. The box says that this description is optional, but when trying to squash without this description, the squash fails with a vague error message.

TerminusX: Empty WOQL query gave WoqlErrorResponse with WOQLSyntaxError

On loading a data product on TerminusX and watching the console, I saw that a request returned a 400 status.

The request:

POST https://cloud.terminusdb.com/Community_1/api/woql/Community_1/;lasdf asdlkfj/local/branch/main
{"query":{}}

The response:

{
  "@type":"api:WoqlErrorResponse",
  "api:error": {"@type":"vio:WOQLSyntaxError", "vio:path": [], "vio:query": {}},
  "api:message":"Not well formed WOQL JSON-LD",
  "api:status":"api:failure"
}

The error doesn't seem to cause any problem in the UI as far as I can tell.

TerminusX dashboard - property definitions lost

Property names are overridden with "title" and property definitions wiped out in some cases on "Schema check failure." Also, a more helpful error message will be of great value, especially to beginners.

Screen-print:

image

Report actual schema error

It's not easy to determine the cause of failure if all that is reported is that there is an error. Even just giving a formatted version of the witness may be better.

Screenshot from 2022-06-08 11-21-00

TerminusX GUI - Context sensitive-help

Add buttons or links to documentation, tutorials, or blogs to provide context-sensitive help on the functions available in the GUI. Example functions:

  • Create data products.
  • Documents/sub-documents/parent/child relationships.
  • Key types - lexical, random, etc.
  • Property types - set, list.
  • WOQL - see also #17
  • Etc.

TerminusX dashboard - Product Model Graph and JSON view visual enhancements

Low priority enhancements to the Product Model views. If possible, implement options to provide the following:

GRAPH VIEW

Add link property lines to documents

Example:

[ Organization ] ---------------------- [ team ]

Add relationship/link names
Display these on the lines between documents.

Example:

[ Organization ] ------[ consists-of ] -------- [ team ]

Properties in document boxes
Display these in the document boxes or in tooltips on hover.

Differentiate properties
Highlight or color-code link properties on the Properties tab and/or change the link icon to represent a link.

Visually link/add relationships between documents
Visually link documents by selecting documents using something like Shift+Cick.

Display a background grid
Implement an option to display a grid in the background. - assists with aligning model components for presentation and documentation purposes.

Screen-print of the model
Implement an option to take a screen-print of the model with a transparent background - assists with documentation, especially tutorials.

Color configuration
Implement an option to change model component colors - enables users to implement schemes more consistent with their company and documentation branding.

Legend
Add a legend describing all the different model components.

JSON VIEW

  • Add syntax coloring.
  • Increase the height of the view box to make more of the definition visible.
  • Enable edits to the definition (automatically update Graph view.)
  • Create graph view by importing a JSON definition (not sure if this is already in place.)
  • Make the view a bit more intelligent. For example, when hovering over the '"@type"' value , display/list all possible values. Also consider enabling the user to override the existing value by selecting an alternative from that list.

What's New Popup

To have a What's New button or option so people are aware of new features to try or changes that might impact their work.

Dashboard: deep linking & browser history

Is your feature request related to a problem? Please describe.

  • I sometimes want to quickly inspect/edit an entity in the dashboard, but finding a specific entity in the dashboard can be quite cumbersome
  • When browsing the Document Explorer, I can't use the browser navigation to go back

Describe the solution you'd like
Both issues could be resolved by moving parts of the application state into the address bar:
https://dashboard.terminusdb.com/document_explorer/[team]/[data product]/[branch]/[type]/[id]

It would be nice if the js and python clients would also offer a method for retrieving the dashboard url of a given document

Additional context
I'm working on a custom interface which does not yet support editing primitive values. While this is not frequently needed, it would still be nice to quickly access the document in the TerminusX dashboard if it is.

TerminusX dashboard - undo removes all changes

When clicking the undo button, all changes are undone up to the last save-point. A user will typically expect the last action to be undone only. This has the potential to undo numerous changes and wipe out several hours of work (without warning) in an instant.

Would recommend fixing this as a priority as it can be immensely counterproductive and a potential deal-breaker, especially for new users.

See also #26 (comment)

TerminusX dashboard testing - more helpful error messages

In some cases, error messages generated by the dashboard are unclear - an example below. A possible rewrite of this error message:

Please associate a property as the key field for the document:

  1. Select a property from the Select Property dropdown on the Documents tab or
  2. Create a property on the Properties tab, then implement step 1.

image

TerminusX subdocument property editor for description field can't be used

Describe the bug
Adding text to the description of a property (String or Numeric) in TerminusX fails. Error 400 is returned from the server (reported in browser console).

To Reproduce
Steps to reproduce the behavior:

  1. Go to Data Product Model view
  2. Select a subdocument type
  3. Go to properties tab
  4. Add property or edit existing property of type string or numeric
  5. Add id name, and random string to description field (example: "test")
  6. Click on floppy icon/save
  7. See error "Failed to update schema graph"

Expected behavior
The added description to the property of the subdocument should be saved into the schema graph for the subdocument.

Screenshots
image

Info (please complete the following information):

  • OS: macOS/Microsoft Edge (Chromium-based) browser
  • TerminusX console

Additional context
N/A

TerminusX dashboard - duplicate properties

  • Duplicate link properties created (shouldn't be allowed?)
  • The screen goes blank when attempting to delete a duplicate.
  • All properties of any type are deleted when trying to delete a duplicate.

Screen-prints:

Duplicates created:

image

Issues when trying to delete a duplicate:

image

User message to help them understand their access privileges

Describe the bug
I tested the dev environment and it works like a charm for me. Really great stuff. There's one slight issue I noticed. This was my process:

I created a data product, schema and added some data, then invited my colleague to join as a collaborator.

My colleague joined no problem.

I then change my colleague to an info reader user.

Here there is a slight issue - When trying to view the document detail - it rightly forbids me, but doesn't tell me why. It just hangs on the screen saying it is fetching data (shown below).

Expected behavior
It could do with a message like: "Your permissions allow you schema level access. To view documents, please contact the team admin"

Screenshots
image

Info (please complete the following information):

  • OS: Windows 10, Chrome
  • TerminusX - Dev

TerminusX dashboard - some potentially simple productivity enhancements

Please consider the following productivity enhancements on the model view:

  • Ctrl+Z and Ctrl+Y to undo/redo the last change.
  • Del key to delete an object (warning or confirmation message isn't necessary if undo/redo functionality is available.)
  • Save model elements in the order they were created or subsequently manually rearranged. For example, a user may intuitively create related documents sequentially or later drag/group related documents to create a more intuitive or logical visualization. At present, each save operation resets any reordering the user performs.

property accordion should open unexpanded

It is very difficult to get an overview of properties in the schema UI for a given class since the property accordion comes expanded, making the actual existing properties obscure.

TerminusX Dashboard: Data Product Model Will not Display -

Describe the bug
After a schema update, Data Product Model on TerminusX Dashboard shows a black screen.

To Reproduce
Steps to reproduce the behavior:

  1. Open TerminusX Dashboard
  2. Select database
  3. Click on 'Data Product Model'
  4. See error

Expected behavior
Should display a graphical representation of the DB schema

Screenshots
Screen is black with no navbar or sidebar.
Screenshot 2021-09-30 at 14-39-45 TerminusX

Info (please complete the following information):

  • OS: Windows 10 Pro 20H2
  • Browser: FireFox 92.0.1 (64-bit)
  • How did you run terminus-server": TerminusX Dashboard
  • SWI Prolog version if you're using the manual installation: N/A

Additional context
Console logs two errors:

 TypeError: s is undefined  tdb-dashboard.min.js: 39486
    m tdb-dashboard.min.js:100711
    m tdb-dashboard.min.js:100690
    m tdb-dashboard.min.js:100686
    Nb tdb-dashboard.min.js:100725
    Nb tdb-dashboard.min.js:100730
    Nb tdb-dashboard.min.js:100731
    Yb tdb-dashboard.min.js:101232
    as tdb-dashboard.min.js:39544
    wl tdb-dashboard.min.js:40565
    unstable_runWithPriority tdb-dashboard.min.js:48243
    Vi tdb-dashboard.min.js:37667
    bl tdb-dashboard.min.js:40548
    rl tdb-dashboard.min.js:40102
    Gi tdb-dashboard.min.js:37697
    unstable_runWithPriority tdb-dashboard.min.js:48243
    Vi tdb-dashboard.min.js:37667
    Gi tdb-dashboard.min.js:37693
    qi tdb-dashboard.min.js:37685
    Xs tdb-dashboard.min.js:39961
    ga tdb-dashboard.min.js:38578
    t tdb-dashboard.min.js:99798
    c tdb-dashboard.min.js:47325
    _invoke tdb-dashboard.min.js:47307
    k tdb-dashboard.min.js:47363
    zv tdb-dashboard.min.js:99689
    a tdb-dashboard.min.js:99703
Uncaught (in promise) TypeError: s is undefined
    m tdb-dashboard.min.js:100711
    m tdb-dashboard.min.js:100690
    m tdb-dashboard.min.js:100686
    Nb tdb-dashboard.min.js:100725
    Nb tdb-dashboard.min.js:100730
    Nb tdb-dashboard.min.js:100731
    Yb tdb-dashboard.min.js:101232
    as tdb-dashboard.min.js:39544
    wl tdb-dashboard.min.js:40565
    unstable_runWithPriority tdb-dashboard.min.js:48243
    Vi tdb-dashboard.min.js:37667
    bl tdb-dashboard.min.js:40548
    rl tdb-dashboard.min.js:40102
    Gi tdb-dashboard.min.js:37697
    unstable_runWithPriority tdb-dashboard.min.js:48243
    Vi tdb-dashboard.min.js:37667
    Gi tdb-dashboard.min.js:37693
    qi tdb-dashboard.min.js:37685
    Xs tdb-dashboard.min.js:39961
    ga tdb-dashboard.min.js:38578
    t tdb-dashboard.min.js:99798
    c tdb-dashboard.min.js:47325
    _invoke tdb-dashboard.min.js:47307
    k tdb-dashboard.min.js:47363
    zv tdb-dashboard.min.js:99689
    a tdb-dashboard.min.js:997

UI Feedback

Is your feature request related to a problem? Please describe.
When i create a data product in the UI I don't get good feedback if I don't fill in one of the necessary fields.

Describe the solution you'd like
I would like the fields to either have perma * or to have a red outline when they are not complete but need to be completed.

Describe alternatives you've considered
I have considered continuing and that is what I did, but I still think it could be better

Additional context
This is where i am talking about:
image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.