Non-linear performance curve in graph traversal about rdf HOT 5 CLOSED

The chief problem here was that the in-memory RDF::Graph implementation relied on the rather inefficient generic default implementation of the RDF::Queryable#query method (the #first_subject, #first_literal, etc methods all use #query under the hood).

I've fixed this in changesets c4b54f2 and 38aa925 by optimizing the in-memory implementations of RDF::Graph#query and RDF::Repository#query specifically for their respective internal data structures.

This resulted in a 30x overall speed-up when running your test script on my laptop, with the entire script now consistently finishing in about 110 seconds (using Ruby 1.9.1).

Note that there are definitely lots of other low-hanging opportunities for optimization remaining, so in case something's still too slow for you, just let us know. At present there hasn't been that much work on the in-memory implementations, with any and all improvements implemented only in response to user demand. Our recent focus has been more on scalability than performance, with some of the recent API improvements (for enabling the Cassandra storage adapter to scale up to half a billion triples) perhaps even having somewhat adversely affected the performance of the default in-memory storages. These kinds of performance issues are easy enough to fix as long as somebody just lets us know about them - so many thanks for the detailed issue and benchmark script, I hope to see more of these.

I'll release a version 0.2.2 today which will include these optimizations.

from rdf.

I've now released version 0.2.2. Do let me know how it works out for you.

from rdf.

Great - 0.2.2 completed the test in 75s on my machine, which is a huge improvement. Thanks! Perhaps when I get some free time (haha) I'll be able to pitch in and contribute to the low-hanging fruit.

from rdf.

Guess we can close this issue now, and open up new ones for future performance reports.

from rdf.

OK, great; closing the issue, let's open new ones if/when need be.

from rdf.