Light

sivanlaai / fundcrawler Goto Github PK

View Code? Open in Web Editor NEW

12.0 1.0 5.0 300 KB

天天基金爬虫，抓取市面上所有基金信息\基金净值\基金成分\基金公司\基金经理

License: Apache License 2.0

Python 100.00%

crawler crawling crawl fund stock

fundcrawler's Introduction

轻量级基金爬虫

天天基金爬虫，抓取市面上所有基金信息\基金净值\基金成分\基金公司\基金经理

scrapy的框架太过厚重了，选用requests\execjs\lxml\selenium搭建了一个轻量级爬虫，可扩展性强，易于使用。

基金基本信息爬虫 AllFundCrawler

从网站上抓取所有开放基金的基本信息。

基金分红爬虫 FundDividendCrawler

基金成份爬虫 FundDividendCrawler

这个爬虫抓取的数据还是比较重要，主要是了解每只基金的组成的股票组合是什么，对于投资参考有很好的作用。

基金净值爬虫 FundTradeCrawler

抓取基金的过往历史数据，用于分析。

使用方法

0 python安装requests,lxml,pymongo,execjs,selenium

1 安装mongodb，最好配置用户权限，以免被攻击

2 配置setting

[SERVER]
port = 8010
host = 0.0.0.0
MAX_TRY = 10

[LOG]
LEVEL = INFO //日志等级
LOG_PATH = ./FundCrawler/logs //日志目录

[MONGO]
URL = 127.0.0.1 //MONGO服务器ip
PORT = 20137 //MONGO服务器端口
USERNAME = username
PASSWORD = password
DATA_BASE_NAME = Fund
ALL_FUND_COLLECTION = AllFundInfo
FUND_TRADE_COLLECTION = FundTradeInfo
FUND_STOCK_SHARE_COLLECTION = FundStockShare
FUND_BOND_SHARE_COLLECTION = FundBondShare
FUND_COMPANY_COLLECTION = FundCompany
FUND_DIVIDEND_COLLECTION = FundDividend
FUND_INDUSTRY_COLLECTION = FundIndustry
FUND_MANAGER_COLLECTION = FundManager
FUND_REVIEW_COLLECTION = FundReview
FUND_BASIC_COLLECTION = FundBasicInfo

3 运行爬虫

python src/CrawlerFacade.py

fundcrawler's People

Contributors

Stargazers

Watchers

Forkers

xiaorengong zq942357 lisakk flm143

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.