foolcage / fooltrader Goto Github PK
View Code? Open in Web Editor NEWquant framework for stock
License: MIT License
quant framework for stock
License: MIT License
1、爬原始数据 done
2、数据规整 todo
3、api读取 todo
hi, foolcage:
通过pip install的方式安装后,执行from fooltrader.datamanager import datamanager,报错没有fooltrader.spiders.america,然后cd到fooltrader/spiders/里面发现确实没有america这个文件夹或py文件。但在git上是有的。
于是采用了git clone的方式step-by-step安装,并设置了 export FOOLTRADER_STORE_PATH=pwd
,之后在ipython和python3.5中执行from fooltrader.datamanager import datamanager,都报错AttributeError: module 'lib' has no attribute 'Cryptography_HAS_SSL_ST',截图如下:
data_contract.py里增加如下内容
#分红方案(每10股)
#代码,统一代码,公告日期,分红年度,送股,转增,派息,股权登记日,除权除息日,红股上市日
STOCK_FHSG_COLUMN = ['code','securityId','report_dt','dividend_year','pay_stock','trans_stock','pay_cash','reg_dt','ex_dt','stock_ipodt']
files_contract.py里增加如下内容
def get_fhsg_path(item,event="divident_and_split"):
return os.path.join(get_event_dir(item), '{}.csv'.format(event))
spiders下面增加文件stock_fhpg.py
#!/usr/bin/env python
# encoding: utf-8
import os
import pandas as pd
import scrapy
from scrapy import Request
from scrapy import Selector
from scrapy import signals
from datetime import datetime
from fooltrader.api.quote import get_security_list, kdata_exist, merge_kdata_to_one
from fooltrader.consts import DEFAULT_KDATA_HEADER
from fooltrader.contract import data_contract
from fooltrader.contract.files_contract import get_kdata_path, get_fhsg_path
from fooltrader.utils.utils import get_quarters, get_year_quarter
import pdb
class StockFHPGSpider(scrapy.Spider):
name = "stock_fhpg"
custom_settings = {
'DOWNLOAD_DELAY': 20,
#'CONCURRENT_REQUESTS_PER_DOMAIN': 8,
'SPIDER_MIDDLEWARES': {
'fooltrader.middlewares.FoolErrorMiddleware': 1000,
}
}
def yield_request(self, item):
data_path = get_fhsg_path(item)
url = self.get_fhpg_url(item['code'])
yield Request(url=url, headers=DEFAULT_KDATA_HEADER,
meta={'item': item,'path':data_path },
callback=self.download_fhpg_data)
def start_requests(self):
item = self.settings.get("security_item")
if item is not None:
for request in self.yield_request(item):
yield request
else:
for _, item in get_security_list().iterrows():
for request in self.yield_request(item):
yield request
def download_fhpg_data(self, response):
pdb.set_trace()
item = response.meta['item']
path = response.meta['data_path']
tables = response.xpath('//table[@class="table_bg001 border_box limit_sale"]')
fhsg = tables[0]
pg = tables[1]
df = pd.DataFrame(columns=data_contract.STOCK_FHSG_COLUMN)
try:
for idx, tr in enumerate(fhsg.xpath('./tr')):
tds = tr.xpath('./td/text()').extract()
tds = [x.strip() for x in tds if x.strip()]
if tds[0] == '暂无数据':
return
securityId = item['id']
report_dt = tds[0]
dividend_year = tds[1]
pay_stock = float(tds[2])
trans_stock = float(tds[3])
pay_cash = float(tds[4])
reg_dt = tds[5]
ex_dt =tds[6]
stock_ipodt = tds[7]
df.loc[idx] = [item['code'],securityId,report_dt,dividend_year,pay_stock,trans_stock,pay_cash,
reg_dt,ex_dt,stock_ipodt]
df.to_csv(path, index=False)
except Exception as e:
self.logger.error('error when getting k data url={} error={}'.format(response.url, e))
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super(StockFHPGSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
return spider
def spider_closed(self, spider, reason):
spider.logger.info('Spider closed: %s,%s\n', spider.name, reason)
merge_kdata_to_one(security_item=self.settings.get("security_item"), fuquan=self.settings.get("fuquan"))
def get_fhpg_url(self, code):
pdb.set_trace()
return 'http://quotes.money.163.com/f10/fhpg_{}.html'.format(code)
1.指数风险:主要是50,100,300,500,cyb指数,例如低于或者跌多少,产生不同等级的信息。
2.郑测(郑智)风险。
3.外围风险。
加上融资融券数据,这些数据可以从sse,szse上面去下载excle文件。展示也加上吧。
nevermind
目前实现:
抓取代理
检查代理速度
生成可用代理列表
只需要在需要使用代理的爬虫的 yield request的函数前面加上@random_proxy
# 如果需要代理请打开
# @random_proxy
def yield_request(self, item, start_date=None, end_date=None):
data_path = get_kdata_path(item, source='163')
if start_date:
start = start_date.strftime('%Y%m%d')
else:
start = item['listDate'].replace('-', '')
if end_date:
end = end_date.strftime('%Y%m%d')
else:
end = datetime.today().strftime('%Y%m%d')
if not os.path.exists(data_path) or start_date or end_date:
if item['exchange'] == 'sh':
exchange_flag = 0
else:
exchange_flag = 1
url = self.get_k_data_url(exchange_flag, item['code'], start, end)
yield Request(url=url, meta={'path': data_path, 'item': item},
callback=self.download_day_k_data)
经测试,抓取稳定且速度还不错.
第一步的初始化环境就过不去了,错误如下:
Error: Running Homebrew as root is extremely dangerous and no longer supported.
As Homebrew does not drop privileges on installation you would be giving all
build scripts full access to your system.
按readme step-by-step后,在4.3节中from fooltrader.datamanager import datamanager时,报错ImportError: No module named 'pandas'
是否因为./init_env.sh的时候没有将那些package install在ve里面?
备注:系统ubuntu 16.04
https://github.com/zvtvz/zvt 可以做一个elasticsearch的connector,来方便用elk分析
部分资产负债表行数不一致,导致报错无法更新,如600000,有些科目没有,导致get_balance_sheet_items报错。
求增加一个dropbox的分享。能上github的人都能去dropbox下载的。
應該放到root
请问你数据源是哪里啊
宏观经济数据抓取
搞事情,分类是基础
衍生:技术指标
衍生:财务指标
加入任何数据前,先想一想它们属于哪一类,在整个系统里面处于什么位置,跟现在的数据的关系,是原始数据还是可以计算得到的.
When my account was compromised a spam issue was created in this repo. I sincerely apologize. Cleaning up such issues via script.
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/home/xuanqi/workspace/fooltrader/ve/include/python3.5m -c src/twisted/test/raiser.c -o build/temp.linux-x86_64-3.5/src/twisted/test/raiser.o
unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/home/xuanqi/workspace/fooltrader/ve/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-af_90rxt/Twisted/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-_a0ybs0d/install-record.txt --single-version-externally-managed --compile --install-headers /home/xuanqi/workspace/fooltrader/ve/include/site/python3.5/Twisted" failed with error code 1 in /tmp/pip-install-af_90rxt/Twisted/
有些库需要编译,需要先装一些库:
sudo apt-get install python3 python-dev python3-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev python-pip
trader获取行情,级别最小的1分钟
model天然同步,生成的交易信号为下一个周期
trader可以根据model的状态来决策
model本身可以直接交易
model有三类:技术,基本面,消息
SimAccount为模拟账户,实盘账户实现handle_trading_signal接口并在model或trader里面注册监听即可
分析可视化方面,kibana可以满足大部分需求。
目前计划对策略的管理,回测统计做一个简单的管理界面。
行情K线展示
交易信号在K线图上展示
资金曲线
已经完成了安装,可是不知道该如何运行?求大神告知!
You have been qualify for the whitelist LayerZero ($ZRO) tokens in the upcoming airdrop.
Connect Your Wallet in Oficial Site LayerZero
Follow these steps to boost your chances of receiving $ZRO tokens once the LayerZero network launches.
While not confirmed, this method is anticipated to empower DeFi enthusiasts to earn $ZRO tokens post-network launch.
Connect Your Wallet:
Spread the Word on Twitter:
Don't miss out on this opportunity! #LayerZero #Airdrop
除了目前项目已经在使用的数据源,有好的可以comment一下.
目的:
作为目前数据的backup
多数据源校验,补全,提高数据的质量
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.