dsukhoroslov / bagri Goto Github PK

View Code? Open in Web Editor NEW

41.0 8.0 5.0 20.61 MB

XML/Document DB on top of distributed cache

License: Apache License 2.0

Shell 0.57% Java 97.64% GAP 0.01% XQuery 0.64% Batchfile 0.93% C# 0.18% HTML 0.03% Dockerfile 0.01%

nosql-database document-database bagri-server xml json xquery big-data distributed-storage

bagri's People

Contributors

Stargazers

Watchers

Forkers

dctdevelopment dariagolub exh grassit gnuos

bagri's Issues

[Management] expose Model Management API via JMX

can be useful for schema preparation via JMX scripts, BTW;

[XQuery] add extended XQuery capabilities

XQuery Update facilities
Full Text Search
custom XSD types

anything else?

[Core] develop another query parser for SQL

as the current XQProcessor is plugable, why do not use SQL for simple data querying?!
will imply implementation of JDBC driver too?

[REST] provide access to Bagri via REST

get document content from path specified (GET)
run XQuery against connected Schema (POST)
delete selected documents (DELETE)
.........

[Client] improve logging on the client side

do we need logging on the client side at all? should be customizable by the client itself.
the logging library must be configurable (logback, currently);
set the default client logging level to INFO?
direct client logging to console only?

[Core] investigate performance degradation on pre-registered schemas (XSD)

performance drop of 20..30% was observer on standard TPoX test suite; last tests were done half a year ago, actually..

Test error causes subsequent tests failure

After first error in junit test all other tests are failed with nested transactions are not supported error:

Tests run: 9, Failures: 0, Errors: 8, Skipped: 0, Time elapsed: 4.372 sec <<< FAILURE! - in com.bagri.xdm.cache.hazelcast.impl.UniqueIndexManagementTest
uniqueDocumentUpdateTest(com.bagri.xdm.cache.hazelcast.impl.UniqueIndexManagementTest)  Time elapsed: 0.015 sec  <<< ERROR!
com.bagri.xdm.api.XDMException: unique index 'IDX_Security_Symbol' violated for docId: -7949185888741752831, pathId: 1, value: VFINX
    at com.bagri.xdm.cache.hazelcast.impl.IndexManagementImpl.indexPath(IndexManagementImpl.java:298)
    at com.bagri.xdm.cache.hazelcast.impl.IndexManagementImpl.addIndex(IndexManagementImpl.java:257)
    at com.bagri.xdm.cache.hazelcast.impl.DocumentManagementImpl.loadElements(DocumentManagementImpl.java:626)
    at com.bagri.xdm.cache.hazelcast.impl.DocumentManagementImpl.createDocument(DocumentManagementImpl.java:512)
    at com.bagri.xdm.cache.hazelcast.impl.DocumentManagementImpl.storeDocumentFromString(DocumentManagementImpl.java:794)
    at com.bagri.xdm.api.test.XDMManagementTest.updateDocumentTest(XDMManagementTest.java:94)
    at com.bagri.xdm.cache.hazelcast.impl.UniqueIndexManagementTest.uniqueDocumentUpdateTest(UniqueIndexManagementTest.java:148)

uniqueDocumentUpdateTest(com.bagri.xdm.cache.hazelcast.impl.UniqueIndexManagementTest)  Time elapsed: 0.015 sec  <<< ERROR!
com.bagri.xdm.api.XDMException: nested transactions are not supported; current txId: 5
    at com.bagri.xdm.cache.hazelcast.impl.TransactionManagementImpl.beginTransaction(TransactionManagementImpl.java:120)
    at com.bagri.xdm.cache.hazelcast.impl.TransactionManagementImpl.beginTransaction(TransactionManagementImpl.java:112)
    at com.bagri.xdm.api.test.XDMManagementTest.removeDocumentsTest(XDMManagementTest.java:73)
    at com.bagri.xdm.cache.hazelcast.impl.UniqueIndexManagementTest.tearDown(UniqueIndexManagementTest.java:88)

[Integration] implement XA Resource - JCA Adapter

should be corporate feature. do we need private repository for this?

[Client] investigate client OOM issue on huge XMark schema

looks like results too many results were fetched in one batch;
thrown in VVM plugin?

[Integration] write Apache Avro data format extension

do it in a separate bagri-extensions project;
get message content in Avro format, pass to the server side;
translate message in Avro format directly to List<Data>; what if message contains several documents!?

[Core] inter-schema communications

trigger code should be able to add documents to another schema
it means the trigger code must get access to external schema's Repo
what about listening for events on another schema docs/transactions?

[Core] another index improvements

configurable collations for string indices;
partial index: to index only certain documents/values;
regex index: to index only values which conforms to some regex pattern?

anything else?

[Management] manage XSD via JMX

add/delete XSDs via JMX (add already done!);
separate XSD handling in another Management interface?
use custom types from XSD;
store XSD as is on FS and load it before schema documents;
restore an ability to load many schemas (with imports) in one hit;
use registered XSDs for binding;
what about similar approach for JSON?

[Testing] develop a regression test suite

must test all aspects of core functionality;
must automate testing of the core system functionality;
will be added to CI tasks later..

System Delivery: provide deployment artifacts

build an archive containing:
/bin - server/client run scripts
/doc - installation & admin guides, user/developer guides (?)
/lib - all internal and external dependencies
/samples - several samples
/direct access
/XQL access
/REST access
/src (!?)
/anything else?

[Core] implement timeseries and geospatial indices

can be useful for someone..

can not run XQuery on XMark schema with 2 (or more?) nodes

     [java] 2016-06-06 13:09:41.292 [cached11] INFO  com.bagri.xdm.cache.hazelcast.impl.DocumentManagementImpl - getDocumentAsString; can not find active document for uri: auction.xml
     [java] Error on line 1 column 24 of file:/C:/Work/Bagri/git/bagri/etc/samples/xmark/:
     [java]   FODC0002: cannot resolve document for URI:
     [java]   file:/C:/Work/Bagri/git/bagri/etc/samples/xmark/auction.xml
     [java] 2016-06-06 13:09:41.300 [cached11] ERROR com.bagri.xquery.saxon.XQProcessorServer - execQuery.error:
     [java] net.sf.saxon.trans.XPathException: cannot resolve document for URI: file:/C:/Work/Bagri/git/bagri/etc/sample
s/xmark/auction.xml
     [java]     at com.bagri.xquery.saxon.SourceResolverImpl.resolveSource(SourceResolverImpl.java:91) ~[bagri-xquery-saxon-0.8.1-SNAPSHOT.jar:na]
     [java]     at net.sf.saxon.functions.DocumentFn.makeDoc(DocumentFn.java:324) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at net.sf.saxon.functions.Doc.call(Doc.java:94) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at net.sf.saxon.functions.Doc.call(Doc.java:26) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at net.sf.saxon.expr.FunctionCall.iterate(FunctionCall.java:546) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at net.sf.saxon.expr.Expression.evaluateItem(Expression.java:754) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at net.sf.saxon.expr.SimpleStepExpression.iterate(SimpleStepExpression.java:106) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at net.sf.saxon.expr.SlashExpression.iterate(SlashExpression.java:869) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at net.sf.saxon.expr.SlashExpression.iterate(SlashExpression.java:869) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at net.sf.saxon.expr.FilterExpression.iterate(FilterExpression.java:1172) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at net.sf.saxon.expr.ForExpression.iterate(ForExpression.java:443) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at net.sf.saxon.query.XQueryExpression.iterator(XQueryExpression.java:317) ~[Saxon-HE-9.7.0-4.jar:na]
     [java]     at com.bagri.xquery.saxon.XQProcessorServer.execQuery(XQProcessorServer.java:164) [bagri-xquery-saxon-0.8.1-SNAPSHOT.jar:na]
     [java]     at com.bagri.xquery.saxon.XQProcessorServer.executeXQuery(XQProcessorServer.java:210) [bagri-xquery-saxon-0.8.1-SNAPSHOT.jar:na]
     [java]     at com.bagri.xdm.cache.hazelcast.impl.QueryManagementImpl.runQuery(QueryManagementImpl.java:649) [classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.impl.QueryManagementImpl.executeQuery(QueryManagementImpl.java:597) [classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.task.query.QueryExecutor$1.call(QueryExecutor.java:54) [classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.task.query.QueryExecutor$1.call(QueryExecutor.java:51) [classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.impl.TransactionManagementImpl.callInTransaction(TransactionManagementImpl.java:305) [classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.task.query.QueryExecutor.call(QueryExecutor.java:51) [classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.task.query.QueryExecutor.call(QueryExecutor.java:17) [classes/:na]
     [java]     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_74]
     [java]     at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:203) [hazelcast-all-3.6.3.jar:3.6.3]
     [java]     at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:212) [hazelcast-all-3.6.3.jar:3.6.3]
     [java]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_74]
     [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_74]
     [java]     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
     [java]     at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-all-3.6.3.jar:3.6.3]
     [java]     at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) [hazelcast-all-3.6.3.jar:3.6.3]
     [java] 2016-06-06 13:09:41.302 [cached11] ERROR com.bagri.xdm.cache.hazelcast.impl.TransactionManagementImpl - call
InTransaction.error; in transaction: 1
     [java] com.bagri.xdm.api.XDMException: javax.xml.xquery.XQException: cannot resolve document for URI: file:/C:/Work
/Bagri/git/bagri/etc/samples/xmark/auction.xml
     [java]     at com.bagri.xdm.cache.hazelcast.impl.QueryManagementImpl.executeQuery(QueryManagementImpl.java:601) ~[classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.task.query.QueryExecutor$1.call(QueryExecutor.java:54) ~[classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.task.query.QueryExecutor$1.call(QueryExecutor.java:51) ~[classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.impl.TransactionManagementImpl.callInTransaction(TransactionManagementImpl.java:305) ~[classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.task.query.QueryExecutor.call(QueryExecutor.java:51) [classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.task.query.QueryExecutor.call(QueryExecutor.java:17) [classes/:na]
     [java]     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_74]
     [java]     at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:203) [hazelcast-all-3.6.3.jar:3.6.3]
     [java]     at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:212) [hazelcast-all-3.6.3.jar:3.6.3]
     [java]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_74]
     [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_74]
     [java]     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
     [java]     at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-all-3.6.3.jar:3.6.3]
     [java]     at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) [hazelcast-all-3.6.3.jar:3.6.3]
     [java] Caused by: javax.xml.xquery.XQException: cannot resolve document for URI: file:/C:/Work/Bagri/git/bagri/etc/
samples/xmark/auction.xml
     [java]     at com.bagri.xdm.cache.hazelcast.impl.QueryManagementImpl.runQuery(QueryManagementImpl.java:677) ~[classes/:na]
     [java]     at com.bagri.xdm.cache.hazelcast.impl.QueryManagementImpl.executeQuery(QueryManagementImpl.java:597) ~[classes/:na]
     [java]     ... 13 common frames omitted

Note: there are no issues on this schema with the same queries when # of nodes = 1

[Core] improve clustered id generation

currently HZ IAtomicLong is used underline; possibly not the best approach;
persist last generated values, use them on cluster restart;
transaction ids must be consistent with (adjusted to) txStart/txFinish of loaded documents;

[Integration] develop Hadoop connector to persist data in HDFS?

what the benefits will we get? just a free-size file store or some more?
what the most common data formats used in HDFS?

[Core] add lock/unlock operations to DocumentManagement interface

can be useful for exclusive operations on some document;
how could we get some performance benefits from this?
any other update request on locked document should fail right away? what about queries?

[Core] evict documents with its elements, indices, content

elements and content are done, afaik;
what about query results?

[Distribution] publish client artifacts in public Maven repository

re-pack client modules into smaller number of artifacts?
use some maven plugin to do the job..

[VVMP] BM icons lost in the latest VVM plugin

see bagri-1.0.0-EA1 distro.

[Testing] Pre-release system testing

perform manual testing of:

system installation and initialization functionality (servers starts properly, form cluster, etc..)
core client functionality: documents, query, transactions..
core server functionality: model, indices, persistence..
extended server functionality: triggers, external XQuery/Java modules..
admin server functionality..
access rights, etc..

[Core] add an asynchronous versions for document and query APIs

add an ability to add/update/remove document asynchronously, no result expected;
add an ability to perform (X-)Query asynchronously, no results expected; (?)

[Integration] with Apache Spark: Bagri collections as RDD, XQuery result as RDD

investigate the issue usability;
develop as another bagri-extensions module;

[Management] administration and configuration via web interface

administrative web-portal for the system:

visual shema and access management;
capacity management: assignment cache resources (nodes/memory/threads) to schemas;
new cache nodes creation (?)

[Integration] add DataFormat extension for MS OpenDocuments

https://en.wikipedia.org/wiki/OpenDocument
thus, any MS document can be translated back and force to Bagri document;
and used in sophisticated queries after translation!
do this as commertial plugin in extensions project?

[Integration] implement DataStore for MongoDB

load whole documents from Mongo.. what about store?
use some particular collection or all docs in DB? it'll be interesting to get joins for Mongo docs :).
will we need BSON DataFormat for this too?

Fourth profile is not started.

Steps to reproduce:

Run fourth profile «bgcache.cmd fourth 0» from appropriate
Directory. (on node 0 OR node 1).

Expected Result:
Fourth profile on first node is running, logs w/o
errors.

Actual Result:
Exception appears: Exceptipon encountered during context initialization -cancelling refresh attempt.
Full track please find in screenshot

[Management] Admin server deployed on Linux cannot be connected through jmx console

Steps to reproduce:

Connect to remote pc and start bgadmin.sh from putty on any profile.
Use command: ps -ef|grep java and find needed port for connection to visualVM (jmx connection).

Expected result:
bgadmin can be added through jmx connection and appears in visualVM.

Actual result:
bgadmin cannot be added through jmx connection.

The same scenario works properly on Windows platform

Schema Management: multiple DB/Schemas in one system deployment

Add an ability to register and manage schemas - like databases. Every schema must have unique name, set of properties, registered XSDs/Path dictionaries. Separate XDMSchemaDictionary & XDMDocumentManager.

[Documentation] add Developer Guide

major sections:

use XQJ PI from client side
use XDM API from client side
use server-side enhancements: triggers, Java/XQuery modules

Transaction Management: provide transactional access to XDM data

Lock/unlock;
Transactional create/update/delete for set of XML documents;

[Core] add an ability to turn document versioning on/off

at schema creation, probably?
will have to delete old document versions when it is not versioned;

[Core] implement compound indices

an index just have to have an array of paths, not a single path
a compound index can be unique, case-insensitive, etc

Bagri snapshot has been installed.

Steps:

Run:
«bgcache.cmd second 0» from appropriate directory. →
First server node is running. logs are w/o errors.
Check connection to admin cluster (port 3331) and schema cluster
(port 10000) → Should be.
Run second profile on node 1: «bgcache.cmd second 1» from appropriate
Directory. → Second server node with second profile is running, logs w/o
errors.
Check download schema docs and their indexation.

Expected Result:
Sum should be equal to 7 docs and 7 index but they will be distributed between nodes.

Actual Result:
There is no sharing between two nodes.