Heap Limit reached QT5 about lingbm HOT 12 CLOSED

When I remove the last "reviewFor" the server doesn't crash anymore and I get a response atleast. :)

from lingbm.

Just to add a little bit of info behind this.

The queryTemplate states that:

Thus, the query result trees consist of ca. 10,000 leaf nodes that all represent the same product, and the trees contain many duplicate internal nodes (not only within each level of depth but also across levels).

However, this specific product, 405, for me has 17 reviews. And because of the 4 times nesting we get:
171717*17 = 83000:ish. And this is just the amount of reviews we get a the second bottom-most layer. We then also want to get the same product 83000 times. The amount of leaf-nodes here can therefore wildly vary from the estimated 10000 number.

And yes, with optimizations this query will be very, very fast, but the object that has to be returned will still be huge.

from lingbm.

I did a calculation of leaf nodes on an object returned when there were 15 reviews for a product. The amount of leaf nodes were 155'490!! That's 15x the estimation.

from lingbm.

So I think the crashes are happening because of the overhead for each call to the database. Which means the crashing-issue will disappear when caching/batching, mainly caching, is added to the server.

from lingbm.

We are aware that the estimates for the result sizes can be quite far off, in particular for this query template. One of the things we want to do is to analyze the benchmark itself in order to achieve a better understanding of each of our query templates (e.g., what are their characteristics in terms of possible query instances, result sizes, different dataset sizes).

Now to your particular problem: I think it is not a problem if some queries cause issues for the most naive implementations. The test driver may use a timeout threshold and kill query instances that take longer than the threshold. Of course, these cases need to be recorded in the measurements files produced by the test driver.

from lingbm.

Alright, I see. Thanks for the answer.

Regarding your second paragraph it won't really help since it's the server that is crashing, and not the test-driver. And if the server crashes then the test-driver stops since the server is dead. So I have to figure out a way so that the server stops an query itself if it takes to long.

from lingbm.

So I have to figure out a way so that the server stops an query itself if it takes to long.

Either that or there is a way for the server to come back up again after a crash. But perhaps a server-side timeout is easier to implement. On the other hand, this may not always help; the heap limit may be reached before the timeout. However, do you have a rough idea about the time it takes before the out-of-memory error occurs on your server for that query?

from lingbm.

Just for the record, Sijin and I are considering to change QT5 by removing one level of depth. Sijin is currently looking into the impact that such a change would have on the sizes of query results in benchmark datasets of different scale factors.

from lingbm.

Cool! I think that is a good change.

from lingbm.

The QT5 with 3 cycles possesses similar characteristic with current QT5: results grow exponentially with depth, and the size of the result increases with the incremental of the scale factor, which is adequate to capture corresponding choke-points and do analysis.
Therefore, considering the crashing-issue, we decide to remove one level of depth on QT5.

from lingbm.

@ljukas can you please check and confirm that the query instances of the new version of QT5 do not anymore cause the memory problems due to which you had opened this issue. Assuming that's the case, you can close this issue then.

from lingbm.

I've used the naive server to run over 700 different queries generated from QT5 without a crash. So I'd say it is fixed. Closing this issue.

from lingbm.