Faster indexing about aurum-datadiscovery HOT 4 OPEN

Do any of these help? https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html

from aurum-datadiscovery.

Thanks!
Bulk requests should help.

In general, however, we need a more aggressive strategy here, as we are 1 order of magnitude lagging behind profiling. Ultimately, it is about reducing the amount of data we are indexing, and exploiting that we are loading columns of databases---repetition is common---filtering out data we have already seen (per column) should help a lot too.

from aurum-datadiscovery.

I just implemented bulk request. It helped a lot, actually. Indexing is now only 3x slower than profiling (although I haven't optimized profiling yet). In any case this is great news, the gaps is closing.

from aurum-datadiscovery.

Great, glad to hear that guide helped!

from aurum-datadiscovery.