orleanscontrib / orleanscouchbaseprovider Goto Github PK

View Code? Open in Web Editor NEW

6.0 14.0 7.0 99 KB

License: MIT License

C# 100.00%

orleans orleans-storage-provider couchbase

orleanscouchbaseprovider's Introduction

OrleansCouchBaseProviders

This repository to contains a set of providers for running Microsoft Orleans using Couchbase as the storage layer.

Currently supports:

Storage Provider
Membership Provider
Document expiry per grain type
Reminders

How to use

The storage provider can be registered like this:

config.Globals.RegisterStorageProvider<Orleans.Storage.OrleansCouchBaseStorage>("default", new Dictionary<string, string>
{
    { "Server", "http://localhost:8091" },
    { "UserName", "" },
    { "Password", "" },
    { "BucketName", "default" }
});

Password can be left blank if the bucket is not password protected. For using multiple buckets register multiple ones with different names and then use them with the [StorageProvider(ProviderName = "provider name")] attribute on top of grains with state.

The membership provider can be used like this:

config.Globals.DeploymentId = "";
config.Globals.LivenessType = GlobalConfiguration.LivenessProviderType.Custom;
config.Globals.MembershipTableAssembly = "CouchBaseProviders";

config.Globals.RegisterStorageProvider<Orleans.Storage.OrleansCouchBaseStorage>("default", new Dictionary<string, string>
{
    { "Server", "http://localhost:8091" },
    { "UserName", "" },
    { "Password", "" },
    { "BucketName", "default" }
});
config.Globals.RegisterStorageProvider<Orleans.Storage.OrleansCouchBaseStorage>("PubSubStore", new Dictionary<string, string>
{
    { "Server", "http://localhost:8091" },
    { "UserName", "" },
    { "Password", "" },
    { "BucketName", "default" }
});
config.Globals.DataConnectionString = "http://localhost:8091";
config.PrimaryNode = null;
config.Globals.SeedNodes.Clear();

NOTE: The membership provider requires a bucket called membership.

Document expiry per grain type

By default documents written to Couchbase will not have an expiry value set.

Support has now been added to allow expiry values to be configured per grain type.

To use this feature you need to update your app.config or web.config file;

Add config section declaration

Add the following under the element:

<section name="orleans" type="CouchBaseProviders.Configuration.CouchbaseOrleansDocumentExpiry.CouchbaseOrleansConfigurationSection, CouchbaseProviders" />

Add the config section with per grain expiry values:

<orleans>
  <grainExpiry>
    <add grainType="grainX" expiresIn="0:0:1:0"></add>
  </grainExpiry>
</orleans>

The expiresIn value must be a valid TimeSpan format. Examples include:

10 seconds: 00:00:10
10 minutes: 00:10:00
10 hours: 10:00:00
10 days: 10:00:00:00

Refer to the app.confg provided in the CouchBaseStorageTests project for more information.

How to help

Take a look at the current issues and report any issues you find. The providers have been tested with CouchBase Community 4.1.

License

The MIT license.

orleanscouchbaseprovider's People

Contributors

Stargazers

Watchers

Forkers

mikegoldsmith marjum richorama mrd1234 kexplo dad045 zeroshi

orleanscouchbaseprovider's Issues

Check stream pubsub support

Currently it is not clear to me if the provider supports serializing and desterializing stream handles in the PubSub store.
If somebody who is using it can check it, it would be good.

Add NuGet publishing

We should add the package to NuGet as well. I've never done this before but would like to do after resolving #1 #2 and #3

Add logging to the library

The library doesn't provide any logging for the moment.
Logs with the appropriate level should be added

fix the dispose related issues

There is an issue regarding the way that library is being disposed which cases not all tests to run with each other and after first silo shuts down, the library is not usable in the same application. the issue doesn't exist in a real application since after a silo shutsdown/crashes , the library will be reinitialized in a new application.

Add binary storage option

Orleans storage providers should be able store grain state directly in a binary format and not in JSON.

The StorageProvider attribute has a property which indicates if the serialization should be to JSON or not and Orleans has a binary serializer SerializationManager which can serialize data to binary in a none-backward compatible manner. That is one option and there are many others as well.

Reading grain state with references to other grains fails after silos restart

Assuming a grain's state includes references to other grains, the state's persistence succeeds to the configured Couchbase store, but reading the state back from the store fails with:

Newtonsoft.Json.JsonSerializationException: Could not create an instance of type TestGrains.IStoredReferenceGrain. Type is an interface or abstract class and cannot be instantiated. Path 'StoredReferenceGrain.GrainId'

This occurs only after that the silos have been restarted, forcing grain deactivation, and prompting the Orleans runtime to read state upon grain reactivation.

Target .NET Core 2.0

Add support for multi-node Couchbase cluster connectivity storage configuration

The storage provider's configuration currently allows specification of a single Couchbase node URI, like so:

<Provider Type="Orleans.Storage.OrleansCouchBaseStorage" Name="CouchbaseStorage" Server="http://192.168.99.100:8091" UserName="" Password="" BucketName="orleans_storage" />

It does not seem possible however to specify multiple Couchbase cluster nodes into the same configuration. Looking into the code base, the provider's Init method adds the configured Server property as a single entry into the client configuration's Servers list:

var server = config.Properties["Server"];
.
.
clientConfig.Servers.Add(new Uri(server));

Granted that as long as the specified node is up when the client attempts to connect, it receives server configuration of the cluster's current state, including the current node list. However client connectivity is expected to fail if the configured node were to be down, or if it were to be outside of the cluster at the time.

Perhaps as an alternative, the provider configuration could allow Couchbase client configuration to be specified into app.config, enabling full configuration in addition to the required nodes list. (e.g, bucket options, connection pooling)

Fix MyGet deployment permissions

The deployment permissions for pushing to MyGet are invalid, this is because the OrleansContrib account on AppVeyor will hash the password differently to my personal account.

@richorama will likely need your help to setup the AppVeyor secure hashes.

Contributor permissions

Please can I get contributor permission to this repository?
@richorama

Add ability to set TTL per grain type or document

We have a large number of grain state documents in couchbase storage, most of which are no longer required once the processing they are used for has completed (which could take minutes to weeks depending on the type of grain and other factors).

Being able to set an expiry/ttl per grain type, or per document, and letting couchbase internally manage removal of expired documents would help us prevent disk space issues.

Worst case, being able to set a default expiry/ttl for the bucket would be better than not having the documents expire at all.

Please consider adding this feature.

Note: we are using version 4.6.2-3905 Enterprise Edition (build-3905).

Clean up build scripts

Generic Grain Types cause a NullReference exception in CouchBaseDataManager

When using a generic grain types, I get this exception :

Exc level 0: System.NullReferenceException: Object reference not set to an instance of an object.
   at Orleans.Storage.CouchBaseDataManager.<Read>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Storage.BaseJSONStorageProvider.<ReadStateAsync>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClass1_0.<<QueueTask>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.Catalog.<SetupActivationState>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.Catalog.<InitActivation>d__59.MoveNext()	28.17s

I notice the following line in BaseJSONStorageProvider
var grainTypeName = grainType.Split('.').Last();

So if there a generic argument to the type, it would be something like this:
Foo.Foo[Foo.Bar, Foo, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]

Splitting on the . would not work in this case.

Also because the generic arguments makes the key too long, I can see a possibility to easily exceed 250 characters as a key for Couchbase.

For now I am avoiding any generic grain types, but not sure if there would be a solution in couchbase provider for this problem

Add support for other serialization formats

Add support for storing data as binary as well.

Don't waste a whole bucket just for 'membership'

Couchbase buckets are a precious and rare resource.
Wasting a whole bucket just for a dozen membership records seems wrong.

Couldn't membership be implemented using couchbase datastructures?
https://blog.couchbase.com/net-core-list-queue-and-dictionary-data-structures-backed-by-couchbase/

Add the possiblity to store short type names

I am converting a project's data to use CouchBase since the other database i was using had issues and this plugin is now mature enough thanks to contribution of others. I'll be able to hopefully contribute again.

As a first step i need to be able to store types without their namespace in storage provider. @MikeGoldsmith Is the best way to do it , storing it as a config option in the dictionary? Is it a good idea to add such a feature to the main branch? Actually it has advantages in cases of long namespace and type names. What do you think?

When we support Orleans 2.0 then we'll have a ASP.NET core like config and things will be different but i need this as a first step for my return. I'm a reborn couchbase-ian if that's a thing (not true actually since i always knew couchbase is a good database and it was my first choice but due to a bug in this provider i wrote and need for a more complete viewer and lack of time at the situation back then i had to switch ) :)

Add support for reminders

Reminders are not supported yet. We should add support for them.