morphiaorg / morphia Goto Github PK

When filtering on fields you can only provide includes, or excludes, but not 
both.

In Morphia class change

  public <T> T fromDBObject(Class<T> entityClass, BasicDBObject
dbObject);


to 

  public <T> T fromDBObject(Class<T> entityClass, DBObject
dbObject);

Create a release package which includes source, jar(s), javadocs and a html 
version of the wiki docs.

I have made some significant changes and I'd like to work on a branch (in the project) so that my 
changes will be visible, and we can discuss them. In addition I would like to have a morphia-user 
group created so we have a place to discuss these changes. I'm happy to do all the work if you make 
me a commit'r :)

Many of the changes were influenced by my code coming from [http://code.google.com/p/objectify-
appengine Objectify] on [http://code.google.com/appengine AppEngine].

All of the unit tests work; I have even added some new ones to test the old code. This is still a 
work in progress, but I have kept compatibility while adding these new features and data-access 
patterns. If I end up using this I have a lot of other ideas I'd like to implement (I'd like it to 
look more like Objectify in some ways: more usable Query interface, maybe even mapping over the 
concept of a Key, ancestors, indexed fields, lifecycle annotations, and others). For now I just 
wanted to contribute back to the project with my changes; and to start a discussion.

For the moment I have coded (and written test cases) for the following:
 * Supporting natural ids (something other than an ObjectId) - which can be set before saving
 * Removed the restriction that you use @MongoDocument (unless you are using package-class scanning)
 * Removed the restriction of a @MongoCollectionName field;
   * Added ability to use Class.getSimpleName() as default collection name
   * Added ability to set the default collection name on the @MongoDocument annotation
 * Added a new generic Datastore interface to use instead of creating DAOs/object-type: 
{{{
public interface Datastore {
    <T> T get(Object clazzOrEntity, long id);
    <T> T get(Object clazzOrEntity, String id);
    <T> Iterator<T> get(Object clazzOrEntity, long[] ids);
    <T> Iterator<T> get(Object clazzOrEntity, String[] ids);

    <T> void save(T entity);
    <T> void save(Iterable<T> entities);

    <T> void delete(T entity);
    <T> void delete(Object clazzOrEntity, long id);
    <T> void delete(Object clazzOrEntity, String id);
    <T> void delete(Object clazzOrEntity, long[] ids);
    <T> void delete(Object clazzOrEntity, String[] ids);

    <T> Iterator<T> find(Object clazzOrEntity);
    <T> Iterator<T> find(Object clazzOrEntity, DBObject query);
    <T> Iterator<T> find(Object clazzOrEntity, DBObject query, DBObject fields, int offset, int 
size);

    <T> long getCount(Object clazzOrEntity);
    <T> long getCount(Object clazzOrEntity, DBObject query);
}
}}}
 * Datastore.save doesn't get the object after saving it. Instead the Id/CollectionName fields are 
populated on the saved object (=faster).
 * Datastore.delete of object doesn't send the full object, just the key (=faster).
 * Datastore.find takes a query, and skip/limit optionally
 * Some bug and performance fixes
   * Reflection is now done *once* when the Class is mapped (moved into new MappedClass; which also 
does validation)
   * Cleaned up paging on AbstractMongoDAO.Find to work on the server (instead of walking through the 
skipCount with the Cursor, from the client)
   * Fixed all compiler warnings (missing serialVersionUID, unused fields, unchecked, etc.)
   * Added warnings for types that will not be persisted during mapping
   * Add more code comments (some)

At the moment Morphia create DBObjects directly using object fields. This is a simple 
solution and works similar to Java serialization. But sometimes a more sophisticated 
functionality is needed. Using properties could be a lot more powerful. For instance one 
field could be mapped to a different type of property(-ies):

class SomeMagicDate {
  Calendar date;
  public int getDay() {
    return date.get(Calendar.DAY_OF_MONTH);
  }
  public int getMonth() {
    return date.get(Calendar.MONTH);
  }
  public void setDay(int day) {
    this.date.set(Calendar.DAY_OF_MONTH, day);
  }
  public void setMonth(int month) {
    this.date.set(Calendar.MONTH, month);
  }

}

Normally object of this class will be serialized to DBObject with one field called date. 
But in this situation the better choice will be to serialize properties "month" and "day".

Add methods from Constraints to Query interface.

It would be nice to be able to get to the original dbobject that created the 
object.

This could allow for a subclassing strategy that could provide extensibility 
in a model.  Then a project could take advantage of the schema-less 
nature of mongo.

Example code could look like this:

@MongoDocument
public class Hotel {

    @MongoID
    private String id;
    @MongoCollectionName
    private String collectionName;

    private String name;
    private int stars;

    @MongoEmbedded
    private Address address;

    // ... getters and setters
}

public class BrandedHotel extends Hotel {

    public static BrandedHotel fromHotel(Morphia morphia, Hotel hotel) {
        return morphia.fromDBObject(BrandedHotel.class, 
hotel.getDBObject());
    }

    private String brandName;

    // ... getters and setters
}

You can easily do it with adding next plugin to your pom.xml

             <plugin>
                <inherited>true</inherited>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-source-plugin</artifactId>
                <executions>
                    <execution>
                        <id>attach-sources</id>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

Provide functionality of setting @Reference(lazy=true) to defer the
necessary fetching of objects, thus cutting the object graph pulled from
mongodb.

as discussed on the list i provided a small prototype that just deals with
Single-Entity references, does not handle ignoreMissing properly yet and
should only serve as a basis for discussion.

please see TestLazySingleReference on the attached patch.

Add support for storing (and searching on) the whole super-class tree and the interfaces implemented 
on any of them. Most likely this will be stored as an array of strings called super-classes 
(including the current class-name).

Something like this should be supported:

@Polymorphic
public class Shape {
}

public class Circle extends Shape {
}

public class Square extends Shape {
}

Datastore ds = ...;
List<Shape> shapes = ds.find(Shape.class).asList()
for(Shape shape : shapes)
  print(shape.getClass().getName());

> Circle
> Square
> Shape
> Circle
> Circle

I'm using morphia and really love it. Currently a found that Date fileds are 
stored as text values e.g. "modified_at" : "Wed Feb 24 2010 13:26:01 
GMT+0100" and not as a number as expected for MongoDB. Should I specify 
something to make it work correctly?

David

Add @EntityListener support; works just like lifecycle methods but for 
another class.

@Entity
@EntityListeners(com.acme.AlertMonitor.class)
public class Account {
@Id long accountId;
}

public class AlertMonitor {
@PostPersist
public void newAccountAlert(Account acct) {
Alerts.sendMarketingInfo(acct.getAccountId(), acct.getBal-
ance());
}
}

com.google.code.morphia.MongoMappingException: Could not map from DBObject

I'm trying to map and object from the mongo database, but I keep getting this 
error. I map to the database using morphia, but only when I try to 
reconstruct it does it fail. I attach the class I'm trying to map.

<b>What steps will reproduce the problem?</b>
1. apply the attached patch
2. run Tests in com.google.code.morphia.suggestions

<b>What is the expected output? What do you see instead?</b>

a) PostPersist does not seem to be propagated as i expected it to be (i
could be wrong, of course)
b) very,very minor corner-case: An 'empty' Entity becomes null after reload
(somewhat understandable, but could be handled or rejected?)

<b>What version of the product are you using? On what operating system?</b>
SVN Head after 0.93

please bear with my skills using junit4, i am still used to 3.8.

While using MongoDB and Morphia, there are 3 object I need to pass around:

Mongo, DB and Morphia.

My thinking is that it does not make sense for the Morphia object to be
able to be used for different databases. I can't think of when its needed
to map my objects to 2 databases in the exact same way. Even then I could
make use of 2 different Morphia instances.

If you agree, would it be reasonable to bind the Mongo as well as the DB
object into Morphia itself?

Morphia morphia = new Morphia(mongo, mydb);

One side effect are much leaner DAOs:

public class MyDAO extends DAO<String,My> {
      MyDAO(Morphia morphia) {
         super(morphia);
      }
}


What do you think?

While building an integration module for tapestry 5, I need to know the
mapped class type in order to build a DataSource.

Right now the call look something like this:
GridDataSource ds = new MorphiaDataSource(customerDAO, Customer.class);

It's not too bad, but its redundant. Would be great if I could get the
class type from the DAO instead of using another parameter.

In cases where there is a type mismatch/error output the full class name in 
the exception/log; remove Class.getSimpleName() calls where appropriate.

I cant get  morphia.mapPackage("models"); to working. Explicit mapping is 
working properly via e.g. morphia.map(User.class).map(Role.class);

When I use mapPackage then morphia claims that there was not maping on 
particular class.

David

When embedding an Entity, that itself has a reference to the root Entity,
Morphia (reasonably) produces a StackOverflow.

Don´t know if this easy to detect, so i´d call this a minor issue.

<b>What steps will reproduce the problem?</b>
1. run attached TestCase

<b>What version of the product are you using? On what operating system?</b>
0.93-SNAPSHOT

java.lang.StackOverflowError
    at java.util.HashMap$Entry.<init>(Unknown Source)
    at java.util.LinkedHashMap$Entry.<init>(Unknown Source)
    at java.util.LinkedHashMap.createEntry(Unknown Source)
    at java.util.LinkedHashMap.addEntry(Unknown Source)
    at java.util.HashMap.put(Unknown Source)
    at com.mongodb.BasicDBObject.put(BasicDBObject.java:153)
    at com.google.code.morphia.mapping.Mapper.mapValuesToDBObject(Mapper.java:467)
    at com.google.code.morphia.mapping.Mapper.toDBObject(Mapper.java:282)
    at
com.google.code.morphia.mapping.Mapper.mapEmbeddedToDBObject(Mapper.java:375)
    at com.google.code.morphia.mapping.Mapper.toDBObject(Mapper.java:280)

...

Add validation rules to check for a no-arg constructor.

Hi,

I found no other way of contacting the project, either by mail or by a 
group, so I fear I have to use the issue tracking for this :)

I've started using morphia a couple of days, and it is working fine.

The only thing which really feels a bit clumsy is the distinction between 
the actual document and the DAO object, and I wondered if it was possible 
to integrate the two into one class, or maybe create a new 
AbstractMongoDAOEntity, which inherits from AbstractMongoEntity and 
implements the DAO functions.

I really would like to see something like this, which is possible with 
other ORM layers:

Hotel h = new Hotel();
h.setName("Hilton");
h.save();

I'm not sure if it is a good idea, or if it is possible at all, but I got 
thinking about it, and really would like to see it.

I'm just getting re-acquainted with Java, otherwise I would have tackled 
the problem myself and just submitted a patch, and I don't feel confident 
enough with the changes in Java since the last time I really used it to 
create a solution which is up to the standards.

But I would be willing to work on it, maybe with a little help and 
pointers.

Cheers,
Jens

Create a MorphiaOptions object to control the behavior of the driver, and our 
interactions.

1.) WriteConcerns (important for sharding/replica-sets)
2.) Safe mode (with exceptions on errors from the mapper)
3.) Retry/Timeout for driver
4.) Thread Pool (threadsPerHost)
5.) ??? map mapping behavior like embedded/reference/serialized, null/empty 
handling, List/Set/Map class defaults

Please see attached (failing) Testcase.

Hi,

First of all I would like to thank you for this project. Honestly I was just about starting on a 
project just like this one myself. You saved me all the work.

I would like to make an request. Would it be possible to have all the fields mapped by default. 
Eg. in JPA you don't have to state @Basic on the fields. Then I guess you need something like 
@Transient (maybe better with @MongoTransient) and @MongoObject (just like @Entity in JPA) . 
The first to make sure that you may leave fields unmapped and the later to instruct the mapper 
to map the object.

Keep up the good work.

ResultsImpl has his constructor without public declaration..defaulting to 
package-private, so it's not accessible outside his package...

attached a single word patch 

ciao
luca

In many cases developers need to have complex queries, but AbstractMongoDAO
class doesn't provide any methods to solve this feature.

In my project I use attached class as parent for all DAO classes.

PS: Sorry, my english is not good.

Sometimes users need to have a custom serialization and deserialization of 
objects. They can use MongoDB driver itself to create DBObject manually. But 
sometimes they want to mix custom serialization with automatic mapping. Main 
objects (@MongoDocuments) could be mapped automatically by Morphia but some 
embedded properties using a custom serialization. For example such techniques 
can be used to improve the performance of serialization.

I have an idea to add some new attibute to @MongoEmbedded. For example 
attribute called "customSerialization". If this flag is set to true then 
Morphia delegate serialization and deserialization to specific methods of 
objects. For instance serialization can be done in the method called 
"DBObject toDBObject()" and deserialization can be done in constructor - 
public SomeObject(DBObject).

Add a write-through datastore cache. This will cache entities loaded/saved by 
the instance of the datastore, via a instance hashmap. Adding this enables 
the datastore to effectively cache simple get(MyEntity.class, id) calls and 
get(MyEntity.class, ids). All writes will pass through the cache and update 
it once completed.

This will be similar to the JPA (lst level) session cache.

This can be written as CachingDatastoreImpl which can be substituted in the 
DatastoreService.getDatastore(true), or something like that.

Hi,

I attached a patch files to fix the two following errors that I found
during the development of our internal CMS.

1. If a field is set to null and his type is a Set an exception is thrown
because if a multiple values field is null by default this field is set
with an empty ArrayList

mf.field.set(entity, propAnnotation != null ?
propAnnotation.listClass().newInstance() : new ArrayList());

The solution is to check the bSet flag to initialize a List or a Set.

2. Within the updateKeyInfo method in the DatastoreImpl.java class, an
exception has been thrown when the type of the value "Object value = 
mc.idField.get(entity);" is a String.

Object dbId = dbObj.get(Mapper.ID_KEY);

So the "if (value != null && !value.equals(dbId))" is always equals to
false because the following expression is false:
  - String.equals(objectId)

I switched the if and it works.

----------------------------------------

Also I would like to know how I can participate to this project.

Thanks.

Jérôme.

What's the best way of making a document ID from a string? Is there support 
for getting mongo to add IDs? It complains when I remove the annotation.

see attached TestCase

<b>What is the expected output? What do you see instead?</b>
Top-Level Entities seem to get PreSave called twice per save operation.

0.93-SNAP

please see attached testcase

<b>What steps will reproduce the problem?</b>
1. Add this to your pom.xml:
<dependency>
    <groupId>com.google.code</groupId>
    <artifactId>morphia</artifactId>
    <version>0.92</version>
</dependency>
2. mvn compile

<b>What is the expected output? What do you see instead?</b>
Should see something along the lines of &quot;SUCCESSFUL&quot;

Instead the following shows up:
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

Missing:
----------
1) com.google.code:morphia:jar:0.92

  Try downloading the file manually from the project website.

<b>What version of the product are you using? On what operating system?</b>
Apache Maven 2.2.1 (r801777; 2009-08-06 14:16:01-0500)
Java version: 1.6.0_19
Java home: /opt/java/jre
Default locale: en_US, platform encoding: UTF-8
OS name: &quot;linux&quot; version: &quot;2.6.33-arch&quot; arch: &quot;i386&quot; Family: &quot;unix&quot;

Change the current Mapper.objectToValue() and Mapper.valueToObject() methods 
to use the type conversion system. See discussions on the list for more 
details.

There should be a default collection of type converters (attached to the 
mapper) that  can convert to/from Date/Locale/primitives/Key/Enum.

In the case where a float is stored in mongodb and the java type is an int 
there may be data lose. We should check for data conversion loss (checking if 
the int value == the float value), and/or give a warning/error. 

When loading this can be done, but when saving the object will be overridden 
and there may be data lost.

We need support for Update and FindAndModify (but let's call it modify or something). This should be 
supported in the Datastore/DAO; It would also be nice if this worked directly on the entity instance where 
applicable. It would suck if the system didn't let you keep the database and entity instances consistent 
without jumping through lots of hoops.

//runs the update against the query results (on the server)
<T> void Datastore.update(Query<T> q, Update<T> u)
<T> void Datastore.update(Query<T> q, Update<T> u, boolean firstOnly)
<T> void Datastore.update(Query<T> q, Update<T> u, boolean createIfMissing)

//runs the update against the first query result (on the server) returning orig instance
<T> T Datastore.modify(Query<T> q, Update<T> u)
//runs the update against the first query result (on the server) returning the new instance
<T> T Datastore.modifyReturningNew(Query<T> q, Update<T> u)


//runs the update against entity (on the server) returning the orig representation
<T> T Datastore.modify(T entity, Update<T> u)

We could use something like beanutils (jakarta commons) or another reflection based pojo expression language 
under the covers. In the examples above, after the db update occurs, we can alter the entity in the same way; 
there will be difference because collection operations may produce different orders depending on the comparer 
used in java. 

The Update interface would look something like this:

interface Update<T>{

Update set(String fieldExpr, Object value)
Update unset(String fieldExpr)

Update add(String fieldExpr, Object value)
Update add(String fieldExpr, Object value, boolean addDup)
Update add(String fieldExpr, List values)
Update remove(String fieldExpr)
Update remove(String fieldExpr, Object value)
Update removeAll(String fieldExpr)

Update decr(String fieldExpr)
Update incr(String fieldExpr)

}


@Enity(&quot;employees&quot;)
class Employee{

@Id String id;
double salary;
String title;
@Embedded List<Review> reviews;
Date startDate;

}

Employee scott=...;
Datastore ds=...;

ds.modify(scott, (new UpdateStatement()).set(&quot;salary&quot;, 1.1 * scott.getSalary()).add(&quot;reviews&quot;, new 
Review(&quot;Scott is doing well; we are bumping his salary 10%&quot;, new Date(), ApprovedBy.HR)));

//salary cut (by 2K) across the boards.
ds.update(ds.find(Employee.class), (new UpdateStatement()).decr(&quot;salary&quot;, 2000));
//salary cut (by 500) for new employees.
ds.update(ds.find(Employee.class, &quot;startDate >&quot;, minDate), (new UpdateStatement()).decr(&quot;salary&quot;, 500));

Add a page about search, and the Compass Annotations with examples...

We need to figure out a strategy for removed all the className fields from 
the db objects. With lots of small documents (or embedded ones) this becomes 
a bit of overhead. It is unsightly :(

We should be able to get rid of them when there is a concrete class (not an 
interface), or if a concrete class is specified in the annotation decorating 
the field.

Note: We also need to keep in mind a strategy for polymorphism. We may be 
adding back support for storing (and searching on) the whole super-class tree 
and the interfaces implemented on each one. Most likely this will be stored 
as an array of strings called super-classes (including the current 
classname).

Will there be support for float mapping in the near future?

Hello
due to the mongodb lack of relational integrity, it is possible to have in 
a collection a reference to an object in another collection that has been 
deleted.

as of 0.93 this breaks toList cursor iteration.

i've tracked this down to mapReferencesFromDBObject where :
BasicDBObject refDbObject = (BasicDBObject) dbRef.fetch(); 
can return a null refDbObject.

subsequent call throws inevitable exceptions..

this urge the need to have a consistent way to handle such situations, 
whitout the complexity of a relational database (if we need it we'll use 
it..). 

in the attached patch (to  Mapper r195) i added "if not exists anymore 
ignore it" behaviour.

thanks for your work.
Luca

It's great that morphia maps Locale's and Enum's correctly.

DAOs however do not recognize them in queries. It would be great to be able
to do something like:

Locale locale = new Locale("de");
Map<String, Object> query = new HashMap<String,Object>();
query.put(&quot;field&quot;, locale);

Right now, mongo raises a failure:
Caused by: java.lang.IllegalArgumentException: can't serialize class
java.util.Locale
    at com.mongodb.ByteEncoder._putObjectField(ByteEncoder.java:284)

Add support for complex objects as @Id values. This may be an @Embedded, 
@Reference, or a serializable object.

@Embedded
class OrderId {
String customerId;
Date date;
}

@Entity
class Order {
@Id OrderId id;

List<OrderItem> items;
Double taxPercent;
String notes;
}

Add a @NotSaved to specify which values to ignore when saving an entity. 
Optionally a filter class can be used to save some values but not others. 
Fields annotated with @NotSaved are still loaded, but not saved. 

{{{
class User {
@Id String id;
@NotSaved String fullname;
@NotSaved({IfEmpty.class, IfNull.class}) String firstName;
@NotSaved(IfDefault.class) String lastName;

//migrate data from fullname field to first/lastName fields.
@PostLoad void postLoad(){
   if ( fullname != null && !fullname.isEmpty()){
      firstName = fullname.split(" ")[0];
      lastName = fullname.split(" ")[1];
   }
}
}
}}}

http://code.google.com/p/objectify-
appengine/wiki/IntroductionToObjectify#@NotSaved

Morphia currently supports Lists only. Add support for java.util.Set, and 
java.util.Map.

Add support for constructors with arguments. This could surface as a list 
of fields, in order, to be constructed with.

@Property/Reference/Embedded/Serialized (ctorArgs = "id, name")

The mapper would then lookup the types of the fields, get the data from the 
DBObject (converted to java types), and call the constructor with those 
objects.

This is most useful when working with classes that have immutable (final) 
fields; Objects using this patter usually have constructors that require 
those fields be set at construction time.

This is a pretty low-level thing; it is here as a placeholder for 
collecting stars.

Add in support for @AlsoLoad to support loading a field from multiple named 
fields from the datastore. 

This is useful for migrations where you rename something. You may want to run 
in a live system with multiple versions of objects before you batch process 
everything.

http://code.google.com/p/objectify-appengine/wiki/AnnotationReference

If you attempt to serialize a map with a null value, objectToValue throws a 
NullPointerException:

    public Object objectToValue(Object obj) {
        return objectToValue(obj.getClass(), obj);
    }

This is probably an issue in a few other collections as well.

Hi,

MongoDB doesn't have a schema and the model may change for some documents while others 
are not changed in the same collection. This is all right for most cases but if the class is changed 
then the mapping might fail. To be able to change the class over time one should be able to add 
a version field to class. Before the mapping to an java object one should be able to adjust the 
document to the new java class version. So one should be able to add a list of morphers 
(couldn't help myself) that each changes the document from one version to next. That way one 
may update the document when loading it.

E.g.:
A document in the collection is at version 2 and the java class is at version 5. The list of morpers 
are:
1->2 morpher is not applied as the document is at version 2
2->3 morpher is applied 
3->4 morpher is applied
4->5 morpher is applied 
The document is at version 5 and mapped to the java object.

There might be a need for a revert morper as well if one needs to revert a change (but that 
might not be necessary if one apply versioning wisely).

Some of the rules in the system aren't validated during mapping 
(MappedClass/Field.validate). In some cases you don't get an error till you 
map an object and in others it just ignores the field(s) without throwing an 
exception.

Add support for indexing nested fields and multiple fields (in a single 
index: composite).

MappedClass(es) should collect all indexed fields from children during the 
mapping process.

Add warning if more than ~40 indexes (maybe 20 just to warn the user).