ElasticsearchRest Client not working with scheduler about cogstack-pipeline HOT 4 CLOSED

Note that we should revisit this issue for the refactor milestone.

from cogstack-pipeline.

OK let's take a look at this tomorrow. AFAIK closing after the batch doesn't sound like the right behaviour

from cogstack-pipeline.

cc/ @HeglerTissot

from cogstack-pipeline.

A provisional fix for the scheduler has been implemented in 56080d7 .

The general problem is that the ES REST Client (implemented in ESRestService) needs to be closed at the end of batch processing, otherwise the services and resources deinitialization process gets blocked. The @PreDestroy annotated method in ESRestService is not called automatically hence not releasing the resource.

The current workaround in the code (in JobCompleteNotificationListener::afterJob()) closes the ES REST client at the end of running a job and it works when running single jobs (SingleJobLauncher). However, when running scheduled jobs, the jobs are continuously executed, calling each time JobCompleteNotificationListener::afterJob(). Hence, when trying to write to ElasticSearch using ES REST Client for second time it will fail as the client has been already deinitialized (hence the exception mentioned by @jstuczyn ).

To overcome this issue, a simple fix was implemented -- re-initializing the ES REST client in JobCompleteNotificationListener::beforeJob() method, checking whether the client has been previously deinitialised.

However, this is just a quick fix, as ideally a slight refactoring of handling ES REST client and resources should be implemented.

from cogstack-pipeline.