archive_day function with yahoodayfetcher about datafeed HOT 4 CLOSED

Hi @trbck

This is due to http fetcher will stop the ioloop instance after all request processed. See:
https://github.com/yinhm/datafeed/blob/master/datafeed/providers/http_fetcher.py#L79

When I design the fetcher it is mean to be using as a standalone scripts, so I simply made the choice to using a global ioloop for http client. That is the root cause.

This can be avoid by isolating the server ioloop and the http fetcher ioloop.

I will commit a change to the master shortly.

from datafeed.

Hi,
thanks for the update but now the main loop blocks until the fetcher is finished. Since I want to fetch a lot of symbols it takes a while for the datafeed server to respond to external requests. Do you know a way to fix this?

from datafeed.

Hi @trbck

The server didn't design around async, so the main loop blocks when the archive_day gets called.

I think it may a lot easier to using another scripts to sync the data.

from datafeed.

yes, I integrated it with https://github.com/nvie/rq now. Thanks so much for your help!

from datafeed.