Comments (8)
Thanks mitja for the request. I can see how this could be useful.
It's probably the case that most of the in-memory indexes, could be rebuilt in
memory faster than they could be deserialized from disk. So only the objects in
the collection would need to be serialized/deserialized, and then the indexes
re-added. It sounds like a case for adding a readObject() method.
In the meantime, say with version 1.0.3, you could use the
IndexedCollectionSerializer class below, to serialize an indexed collection.
The main catch, is when deserialized, you *need to re-add the indexes!*. See
the SerializerDemo class below for an example.
I'll think about better serialization support for the next release. Thanks!
--------------------------------------------------------------------------------
package com.googlecode.cqengine;
import com.googlecode.cqengine.index.radix.RadixTreeIndex;
import java.io.File;
public class SerializerDemo {
public static void main(String[] args) {
// *************** Build some collection... ***************
IndexedCollection<Foo> myCollection = CQEngine.newInstance();
addIndexesToMyCollection(myCollection);
// Add some objects...
myCollection.add(new Foo("bar"));
myCollection.add(new Foo("baz"));
// *************** Serialize the collection... ***************
IndexedCollectionSerializer.serialize(myCollection, new File("foo.dat"));
// *************** Deserialize the collection... ***************
IndexedCollection<Foo> myDeserializedCollection = IndexedCollectionSerializer.deserialize(new File("foo.dat"));
// Need to add indexes again to the deserialized collection!!...
addIndexesToMyCollection(myDeserializedCollection);
// ************ myDeserializedCollection should now have the same state as myCollection *******
}
static void addIndexesToMyCollection(IndexedCollection<Foo> indexedCollection) {
indexedCollection.addIndex(RadixTreeIndex.onAttribute(Foo.NAME));
}
}
--------------------------------------------------------------------------------
package com.googlecode.cqengine;
import com.googlecode.cqengine.attribute.Attribute;
import com.googlecode.cqengine.attribute.ReflectiveAttribute;
import java.io.Serializable;
public class Foo implements Serializable {
public final String name;
Foo(String name) {
this.name = name;
}
public static final Attribute<Foo, String> NAME = ReflectiveAttribute.forField(Foo.class, String.class, "name");
}
--------------------------------------------------------------------------------
package com.googlecode.cqengine;
import java.io.*;
import java.util.ArrayList;
import java.util.List;
public class IndexedCollectionSerializer {
public static <O> void serialize(IndexedCollection<O> indexedCollection, File destination) {
OutputStream os = null;
try {
os = new BufferedOutputStream(new FileOutputStream(destination));
List<O> objectsList = new ArrayList<O>(indexedCollection);
ObjectOutputStream oos = new ObjectOutputStream(os);
oos.writeObject(objectsList);
oos.flush();
}
catch (Exception e) {
throw new IllegalStateException(e);
}
finally {
if (os != null) {
try { os.close(); } catch (Exception ignore) {}
}
}
}
public static <O> IndexedCollection<O> deserialize(File source) {
ObjectInputStream ois = null;
try {
ois = new ObjectInputStream(new BufferedInputStream(new FileInputStream(source)));
@SuppressWarnings({"unchecked", "UnnecessaryLocalVariable"})
List<O> objectsList = (List<O>) ois.readObject();
return CQEngine.copyFrom(objectsList);
}
catch (Exception e) {
throw new IllegalStateException(e);
}
finally {
if (ois != null) {
try { ois.close(); } catch (Exception ignore) {}
}
}
}
}
--------------------------------------------------------------------------------
Original comment by [email protected]
on 6 Feb 2013 at 6:55
- Changed state: Accepted
- Added labels: Type-Enhancement
- Removed labels: Type-Defect
from cqengine.
Niall,
first of all thank you for very fast response. Unfortunately this is not my use
case. I would use to store IndexedCollection to some Memcache engine(needs to
implement Serializable) and not to file. So due the fact that you stated that
serializing indexes would take to much time, it would be helpful to avoid
"CQEngine.copyFrom" step, so that IndexedCollection could be "stored" directly
without copying and again retrieved without copying, even if indexes must be
added again.
Otherwise I need to say: Great project!!
Original comment by [email protected]
on 6 Feb 2013 at 7:18
from cqengine.
Usually memcache is used to serialize a single object per key. Whereas in this
case you will store an entire collection against a single key? Will this be
retrieved on application startup, or are you planning to do this for each
request?
Memcache is basically "remote RAM". Which indeed is usually faster than local
disk, but slower than local RAM. It might be worth looking at a distributed
cache which supports local RAM with distributed eviction, instead of going
across the network every time. Also take a look at Kryo as an alternative to
Java serialization. It is much faster and does not require classes to implement
the Serializable interface. I've not tested it with IndexedCollection, but I've
had good results from it in the past.
Nonetheless, even if Kryo works with IndexedCollection right now, there are
still a few optimizations to CQEngine which could improve serialization. I will
add support to serialize the indexed collection without copyFrom in the next
release.
Original comment by [email protected]
on 8 Feb 2013 at 12:54
from cqengine.
Not exactly what was requested, but IndexedCollection can now be persisted to
off-heap memory or to a file on disk. It does not rely on Java serialization.
Original comment by [email protected]
on 20 Apr 2015 at 8:25
- Changed state: Fixed
from cqengine.
Reopening this issue, as the current situation is not a complete fix, and could probably be improved.
from cqengine.
I don't understand the status of the issue but I would like to contribute with my use case.
I just need to snapshot a collection along with its already constructed indexes to disk in order to simply survive an application restart.
My collection is fed and indexed with 200.000 records from SQL that mostly never change. Not a huge figure but SQL source could be slow in some scenarios. So I simply would like to:
- Load all SQL records for the first time
- Snapshot them to an opaque binary file
- Restart/kill the application
- Have the application check if the file is up to date with database (
SELECT MAX (last_update)...
) - Load the collection and the indexes from disk to memory
I don't understand if disk store satisfies my scenario. Peeking around the code, it looks like that disk store internally uses SQLite from a disk source, which I don't think loads the data store to memory. What I want to achieve for superfast queries is to snapshot the indices along with the data if possible. Once indices are built, there is no need to recompute them when underlying data has not changed.
Many thanks
from cqengine.
FWIW -
I use the following and it survives restarts without having to do any additional management
ConcurrentIndexedCollection<Job>(DiskPersistence.onPrimaryKeyInFile(Job.ID, new File(dbdir.toString(), "jobs.dat")));
from cqengine.
from cqengine.
Related Issues (20)
- Consider exposing utility static method taking collections
- Reenable cached hashCode
- Support for java.util.Optional
- [Question] How does CQEngine pick the appropriate index to query?
- most longest prefix value exist in the column value but it doesn't intersect with other columns values in the query that we need part of query retrieval HOT 7
- Support STRICT_REPLACEMENT in ObjectLockingIndexedCollection/ConcurrentIndexedCollection
- TransactionalIndexedCollection support for UniqueIndex is broken
- Support for attribute accessors for Java record type HOT 1
- Kryo generates InaccessibleObjectException - Make Dependencies "Java 17 compatible"
- Bug in AttributeOrdersComparator HOT 5
- Offheap cache visibility HOT 1
- Support for ANTLR 4.10.1 HOT 12
- Does it support SQL statistics queries HOT 3
- Thread blocked when using Materialized View Sort For the Longest Prefix for CSV that contain more than 11 K rows HOT 3
- Error while parsing SQL NOT LIKE query HOT 1
- Error while parsing SQL LIKE '%%%' query
- java.lang.IllegalStateException thrown in the getGenericType(Member member) method of AttributeBytecodeGenerator.java
- CQNGrammar Sources missing from master HOT 1
- What is use of sqlite-jdbc in cqengine HOT 2
- Index updated even though objectStore is not modified during IndexedCollection add HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cqengine.