shunwang / n-orm Goto Github PK

What is impossible/inefficient to do so far ?

Rows (elements) quite often stored issue quite a large number of requests to 
the database. One simple mean to limit the number of requests would be to 
retain and aggregate write requests (store/deletes) up to a certain time before 
really sending them to the data store.

So far, its possible to search persisting elements only from key values.
It should be possible to specify indexes in the schema like:
{{{
@Persisting class Foo {
    @Key public String k;
    @Index(name="i1") public String prop1;
    @Index(name="i2", order=2) public String prop1;
}
}}}
so that one can perform the following search:
{{{
StorageManagement.getElements().ofClass(Foo.class).withIndex("i1").withKey("prop
1").setTo("bar").iterate()
}}}

Persisting elements have an order (PersistingElement implements Comparable) 
computed from keys, depending on their order.

For instance, for class
{{{
@Persisting public class Foo{
    @Key public int k1;
    @Key(order=2) public int k2;
}

Foo f11 = new Foo(); f11.k1=1 ; f11.k2=1;
Foo f12 = new Foo(); f12.k1=1 ; f12.k2=2;
Foo f21 = new Foo(); f21.k1=2 ; f21.k2=1;
Foo f22 = new Foo(); f22.k1=2 ; f22.k2=2;
}}}

f11 < f12 < f21 < f22

In some cases, one would like f12 < f11 < f22 < f21. This is especially true 
for Date keys, when latest object should the smallest, so that one could write:
{{{
StorageManagement.findElement().ofClass(Foo.class).any()
}}}
to get the element with the latest date.

Foo should become: 
{{{
@Persisting public class Foo{
    @Key public int k1;
    @Key(order=2, naturalOrder=false) public int k2;
}
}}}

We should try using asynchbase for requests, at least scans, puts, gets, and 
deletes.

We need to add in the doc what are and how to use write retention

It should be possible to declare a class to be pre-splitted across different 
regions of an HBase cluster.
Declaration could be made as follows:
{{{
@Persisting
@HBaseSchema(minRegions=5, separatedRegions=true)
public class Element {...}
}}}

When creating (or checking) for table Element, HBase driver should also check 
that 5 regions are available on the cluster, and that those regions are on 
different servers (separatedRegions parameter).  The number of regions should 
be limited to the number of servers. In case that there are less servers than 
desired regions, regularly (each hour? day? configurable ?), n-orm should check 
if another server is available and split largest region to that server.

Obviously, all of this should work in a multi-process environment...

Multi-store would be a store using different stores.
Different policies on how to use the related stores could make possible to:
* migrate to a different platform (write somewhere, read everywhere)
* lazily migrate data from one store to another (write somewhere, read 
everywhere and write somewhere if not from somewhere)
* use the most reactive store and test performance for different stores or 
different store parameters (write everywhere asynchronously and wait first ACK, 
read everywhere in parallel and wait for first answer)
* use one store as cache (e.g. targeting an in-memory store) in front of 
another (write everywhere, read somewhere and everywhere if no answer)
* ...

Current cache mechanism uses a per-thread cache to avoid synchronization 
problems between tasks. It should be possible to cache some persisting elements 
such that they are available across different threads:

myPerssitingElement.cache();

myPerssitingElement.cacheDuringSeconds(60);

Purpose of code changes on this branch:
Delays write requests on a store ; requests are updated with new values when  
another request on same table with same id is pending

When reviewing my code changes, please focus on:
Concurrency

After the review, I'll merge this branch into:
default

We need to add in the doc what are and how to use federated tables.

So far, the used stored are declared as a property file in the package of class 
file. This was fine for a simple store.

Now appeared write retention and read cache, and tomorrow multi-stores. This 
will become hard to declare assemblies of (delegating) stores. We need 
something like Spring XML files or Guice to set up a store assembly.

HTable is not thread safe ; HTable instances should be reserved to a specific 
thread.
We might use the per-thread cache.

Try making a class with

Set<Object> aCF;

put integers, Boolean and Strings inside

store

Find a persiting element greater that another element:
{{{
StorageManagement.findElement().ofClass(Foo.class).greaterThan(myFoo).any();
}}}

n-orm features an internal per-thread cache.

Still, there is a need for caching either per JVM, centralized or distributed 
cache in order to improve activateIfNotAlready or merely key-to-object mappings.

It should be possible to state which cache to use (e.g. redis, memcached, 
EHCache...) as it is already possible to state which data store to use.

Caches are different from data stores in two points:
- there is no need for ordering items
- there is a need for TTL (on a per-class or even per-object basis)

New stores deserve to be investigated:

 * REDIS/Jedis: consider using SortedSets to store column families
 * lily: would be great for complex search for object (but is there a mean to increment efficiently ?)
 * Google BigTable
 * Amazon SimpleDB
 * MongoDB: would be restricted to 3 levels of hierarchy
 * Cassandra
 * JDBC: it is possible to make n-orm work on any SQL database ; tables would correspond to column families

Make it possible to specify store-specific properties in a query.

For instance:
findElements().ofClass(Book.class).withSpecificProperty("minTimestamp").setTo(13
73384115).withAtMost(100).elements().iterate()

It is often the case that a business model is subject to change.
Some of them are naturally handled, such as adding properties.
Some other might be more difficult to handle, e.g. removing or renaming a 
property, or changing type for a property or a column family.

A mean to specify how to handle old data as they are read should be provided. 
It could take the shape of a method intended to handle problematic activation 
cases.

@Persisting public class BO {
@Key public String aKey;
public int aProperty; //was initially defined as boolean

public void handleActivationProblems(Iterator<ActivationException> problems) {
    while (problems.hasNext()) {
        ActivationException p = problems.next();
        if (p instanceof FieldNotFoundActivationException)
            problems.remove();
        else if (p instanceof ValueNotAcceptedActivationException) {
            ValueNotAcceptedActivationException vna = (ValueNotAcceptedActivationException)p;
            if (p.getField().getName().equals("aProperty")) {
                aProperty = ((Boolean)p.getRefusedValue()).booleanValue() ? 0 : 1;
                problems.remove();
            }
        }
    }
    //If "problems" is not cleared after this method is invoked, an exception will be thrown
}
}