microsoft / qlib-server Goto Github PK

Qlib-Server is the data server system for Qlib. It enable Qlib to run in online mode. Under online mode, the data will be deployed as a shared data service. The data and their cache will be shared by all the clients. The data retrieval performance is expected to be improved due to a higher rate of cache hits. It will consume less disk space, too.

License: MIT License

Python 93.43% Shell 4.19% Dockerfile 2.38%

qlib data server quant python finance stock investment platform

qlib-server's Issues

confusion with config in online mode

I was already using QLIB without the server.
my provider_uri was QLIB_DS2_DATA_DIR

Now I installed the qlib server and it is running. I checked the logs
My config is as follows
`ONLINE_CONFIG = {
# data provider config

"calendar_provider": {"class": "LocalCalendarProvider", "kwargs": {"remote": True}},
"instrument_provider": "ClientInstrumentProvider",
"feature_provider": {"class": "LocalFeatureProvider", "kwargs": {"remote": True}},
"expression_provider": "LocalExpressionProvider",
"dataset_provider": "ClientDatasetProvider",
"provider": "ClientProvider",

# config it in user's own code
"provider_uri": "127.0.0.1:/",
# cache
# Using parameter 'remote' to announce the client is using server_cache, and the writing access will be disabled.

"expression_cache": None,
"dataset_cache": None,
"calendar_cache": None,
"mount_path": QLIB_DS2_DATA_DIR,
"auto_mount": True,  # The nfs is already mounted on our server[auto_mount: False].
"flask_server": "127.0.0.1",
"flask_port": 9710,
"region": "cn",

Now Im not able to query any data.

Seems like the server is not starting correctly.

All logs of the qlib-server container are as follows

docker_support-qlib-server-1  | wait-for-it.sh: waiting 15 seconds for rabbitmq:15672
docker_support-qlib-server-1  | wait-for-it.sh: rabbitmq:15672 is available after 0 seconds
docker_support-qlib-server-1  | [1:MainThread](2023-06-01 05:02:42,283) INFO - qlib.Initialization - [config.py:417] - default_conf: server.

Running docker-compose on windows10 failed

C:\Users\Administrator\Downloads\qlib-server-main>docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env up -d
: Host is already in use by another container
Starting docker_support_redis_1 ... done
Starting 98d81b3aaeec_docker_support_nfs-server_1 ...
Starting 98d81b3aaeec_docker_support_nfs-server_1 ... error

ERROR: for 98d81b3aaeec_docker_support_nfs-server_1 Cannot start service nfs-server: driver failed programming externalRecreating docker_support_rabbitmq_1 ... done
f1c4eec6d8e1): Error starting userland proxy: listen udp4 0.0.0.0:111: bind: address already in use

ERROR: for nfs-server Cannot start service nfs-server: driver failed programming external connectivity on endpoint 98d81b3aaeec_docker_support_nfs-server_1 (345188e9c48d20701a06d4fb86c05435c010915a2069d347a50ef1c4eec6d8e1): Error starting userland proxy: listen udp4 0.0.0.0:111: bind: address already in use
ERROR: Encountered errors while bringing up the project.

client raise exception when read from dataset cache

Hi Team,

we are getting an error when we run qlib online.

Environment

qlib-server
- version: master branch of commit cfd3e4b
- host ip: 192.168.0.165
- OS Version: Ubuntu20.04
- erlang: 23.3.1
- rabbitmq: 3.9.13
- redis: 5.0.4
qlib client
- version: 0.7.1.99
- host ip: 192.168.0.168
- OS Version: Ubuntu20.04

Steps to reproduce

qlib-server(192.168.0.165)

deploy qlib server step-by-step https://qlib-server.readthedocs.io/en/latest/build.html#step-by-step-deployment
pice of config.yaml

provider_uri: /qlib/bin_data/cn_data
flask_server: 127.0.0.1
flask_port: 9710
queue_host: 127.0.0.1
queue_user: admin
queue_pwd: ####
task_queue: qlib_task_queue
message_queue: qlib_msg_queue
max_concurrency: 10
max_process: 30
redis_host: 127.0.0.1
redis_port: 6379
redis_task_db: 1
auto_update: 0
update_time: '23:45'
client_version: '>=0.4.0'
server_version: '>=0.4.0'
dataset_cache_dir_name: dataset_cache
features_cache_dir_name: features_cache

exported uri to NFS clients

/qlib/bin_data/cn_data *(rw,sync,no_subtree_check,no_root_squash)

run the command as user dev

python main.py -c config.yaml

check the files permission in dataset cache

 ls -l /qlib/bin_data/cn_data/dataset_cache/
总用量 1936852
-rw-rw---- 1 dev   dev     344595812 3月   9 18:47 8d1f1755a8b65351ae4fe43016cdf52d
-rwxr--r-- 1 dev   dev        183303 3月   9 18:47 8d1f1755a8b65351ae4fe43016cdf52d.index
-rwxr--r-- 1 dev   dev           235 3月   9 18:47 8d1f1755a8b65351ae4fe43016cdf52d.meta

qlib-client(192.168.0.168)

pip install qlib
mount data cache path manually

sudo mount 192.168.0.165:/qlib/bin_data/cn_data /remote_qlib_data/

modify the config section official example https://github.com/microsoft/qlib/blob/main/examples/workflow_by_code.py

{
                # data provider config
                "calendar_provider": {"class": "LocalCalendarProvider", "kwargs": {"remote": True}},
                "instrument_provider": "ClientInstrumentProvider",
                "feature_provider": {"class": "LocalFeatureProvider", "kwargs": {"remote": True}},
                "expression_provider": "LocalExpressionProvider",
                "dataset_provider": "ClientDatasetProvider",
                "provider": "ClientProvider",
                # config it in user's own code
                #"provider_uri": "192.168.0.165:/qlib/bin_data/cn_data",
                # cache
                # Using parameter 'remote' to announce the client is using server_cache, and the writing access will be disabled.
                "expression_cache": None,
                "dataset_cache": None,
                "calendar_cache": None,
                "mount_path": "/remote_qlib_data",
                "auto_mount": False,  # The nfs is already mounted on our server[auto_mount: False].
                "flask_server": "192.168.0.165",
                "flask_port": 9710,
                "region": "cn",

run the sample code as user robin, raise the exception

File "/workspace/qlib/data/data.py", line 967, in dataset
    df = DiskDatasetCache.read_data_from_cache(mnt_feature_uri, start_time, end_time, fields)
  File "/workspace/qlib/data/cache.py", line 683, in read_data_from_cache
    with pd.HDFStore(cache_path, mode="r") as store:
  File "/workspace/venv/lib/python3.8/site-packages/pandas/io/pytables.py", line 537, in __init__
    self.open(mode=mode, **kwargs)
  File "/workspace/venv/lib/python3.8/site-packages/pandas/io/pytables.py", line 655, in open
    self._handle = tables.open_file(self._path, self._mode, **kwargs)
  File "/workspace/venv/lib/python3.8/site-packages/tables/file.py", line 315, in open_file
    return File(filename, mode, title, root_uep, filters, **kwargs)
  File "/workspace/venv/lib/python3.8/site-packages/tables/file.py", line 778, in __init__
    self._g_new(filename, mode, **params)
  File "tables/hdf5extension.pyx", line 374, in tables.hdf5extension.File._g_new
  File "/workspace/venv/lib/python3.8/site-packages/tables/utils.py", line 158, in check_file_access
    raise IOError("file ``%s`` exists but it can not be read"
OSError: file ``/remote_qlib_data/dataset_cache/8d1f1755a8b65351ae4fe43016cdf52d`` exists but it can not be read

KeyError from config file of qlib

There's no default key in _default_config struction:

Traceback (most recent call last):
File "main.py", line 64, in
init(config)
File "/code/qlib-server/qlib_server/config.py", line 127, in init
redis_host=C["redis_host"],
File "/opt/conda/lib/python3.7/site-packages/qlib/init.py", line 59, in init
if C["auto_mount"]:
File "/opt/conda/lib/python3.7/site-packages/qlib/config.py", line 37, in getitem
return self.dict["_config"][key]
KeyError: 'auto_mount'

the installation methods and steps not defined in markdown readme

the readme.md was not completed, is there a other way to install this? there is no documentation to see or read.

Add CI

Hi, this issue tracks the process of adding CI to the library.

Add test scripts for installing the library.
Add CI script for testing.
Add CI script for docker.

microsoft / qlib-server Goto Github PK

qlib-server's Issues

confusion with config in online mode

Seems like the server is not starting correctly.

Running docker-compose on windows10 failed

client raise exception when read from dataset cache

Environment

Steps to reproduce

qlib-server(192.168.0.165)

qlib-client(192.168.0.168)

KeyError from config file of qlib

the installation methods and steps not defined in markdown readme

Add CI

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent