buildersoftio / andyxstorage Goto Github PK

Buildersoft Andy X Data Storage. This project will be deprecated by the end of 2022

License: Apache License 2.0

C# 99.49% Dockerfile 0.51%

andy data-storage distributed-systems data-services

andyxstorage's Introduction

What is Andy X Storage?

Andy X Storage is an open-source standalone service that is used to store messages for Andy X. The Data Storage service is offers support for Multitenancy storage. X Data Storage hosts all messages and makes sure that all of them are readable for the client.

Get Started

Follow the Getting Started instructions how to run Andy X.

For local development and testing, you can run Andy X within a Docker container, for more info click here

How to Engage, Contribute, and Give Feedback

Some of the best ways to contribute are to try things out, file issues, join in design conversations, and make pull-requests.

Reporting security issues and bugs

Security issues and bugs should be reported privately, via email, [email protected]. You should receive a response within 24 hours.

Related projects

These are some other repos for related projects:

Andy X - Andy X Node, consume and produce messages
Andy X Cli - Manage all resources of Andy X

Deploying Andy X Data Storage with docker-compose

Andy X can be easily deployed on a docker container using docker-compose, for more info click here

Code of conduct

This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community. For more information, see the .NET Foundation Code of Conduct.

Support

Let's do it together! You can support us by clicking on the link below!

andyxstorage's People

Contributors

Stargazers

Watchers

andyxstorage's Issues

Cache Messages to ConsumerPointer when consumer is connected for the first time

Consumers when are connected for the first time with InitialPoistion Earliest, Before sending all messages, store pointers as unacked to Pointers db of consumer

Messages are not caching in the consumers when messages are stored

There is the bug when the number of uncked messages is less than the batchSize message ids are not being flushed into consumer

Add SkipCertificate into XNode Configuration for Production

As a DevOps I would like to skip certificate checking for Production Environments.

Create a service that will flush and close messages when there is no processing

Threads stops before storing messages

Threads are stopping before MessageIOService can store messages to local db.

Implement TenantUpdate settings

As a Developer I want to have the tenant settings updated in the storage for cluster configuration.

Add Backgroud Service for Consumer Acked Messages

Create Background Service that will move acked messages from active pointer database to archive.

Change Payload and Headers to JSON type

Payload and Headers should be moved to JSON type, to prepare the infrastructure for andyx-sql engine

Store Authorization Tokens for Tenants and Components

As a user of Andy X I want to store all token information into x data storage

Split Unacknowledged from acknowledged messages batches

To enable better performance and faster acknowledgment of messages
As a developer I want acknowledged messages to store after message processing

Redesign logging for Andy X Storage

Move Message IO Storing Service from on sync thread to ThreadingPool

Stop unacknowledged message Process when Consumer is disconnected

As an Andy X User, When Consumer is disconnected, the unacknowledged message process should stop.

SentDate is not sent to Client

SendDate of Message is not sent to the Client from the Storage

Sending of unacked messages are not being transmmited for all partitiondates

Messages when consumer is connected, and there are unacked messages in the queue; the storage is only transmitting unacked messages of just a partition file

Message Partition by day YYYYMMDD

As a developer I would like to save messages within SQLite and splitted in different files for different partition.

Implement StorageCalculatorService

As user of Andy X, Storages connected to a node/s, connected as shard should have the same ~ storage size.

A service that will check the size of storage and will will sent the size to the node. Node will continue sending messages to nodes that have the shortest size.

On Consumer Disconenct logging for subscription type is wrong for Shared and Failover

When Consumers with subscription type Shared and Failover are connected and consuming, when they are disconnected in storage-logs it says the subscription type 'exclusive'.

Node should sent the subscription_type when consumer is disconnected.

Store Messages as Batch in Storage

Implement Retention Period for the messages

As a Devops Engineer I want to configure the lifespan of the acked messages on my system. Also, I want to configure how much to check during the day.

Add Header to Records/Message

As a developer a want to send some KeyValuePair as Header for each Record/Message.

Enable tenants.json configuration file

As a devops I would like to config tenants, products, components and topic from configuration file

Change default environment variable names

As a developer when I run the Andy X Storage on docker, I would like to configure with ANDYX-ENVIRONMENT

Synchronize Tokens between Nodes in Andy X Cluster

Memory is not being released

Release memory when Storage is in IDLE,
Send Unacked messages as Batch to x node

Change Threads to Tasks in ThreadingPool

Send messages to consumer after partition is cached

If the consumer is already connected start sending messages to the consumer and don't wait till messages are cached. If consumer is not active and caching is not completed. continue with caching but add condition to transmit messages if consumer is active.

Create SyncService Consumers and Producers connected between Nodes in a Cluster

As a user I want to have the same configuration between Nodes in a cluster.
As a user I would like to see on all Nodes if a consumer or producer is connected to the cluster.