Giter Club home page Giter Club logo

ncov-supplies's Introduction

新型冠状病毒防疫防控物资信息自动收集器

针对2019年末发现的新型冠状病毒疫情,本项目旨在通过爬虫收集全国最新的各机构所需的物资信息。点击查看最新物资需求表

项目内容

1、自动抓取和更新(暂时手动更新,每天保证至少更新一次)

2、自动解析公告信息以减少人工工作,主要包括:

  • 所需物资:常见的物资已支持(详见物资需求表示例
  • 发布时间(日期)
  • 信息来源:目前只收集政府网站发布的公告以保证信息可靠性
  • 需求方信息:机构、地址、联系信息等(主要功能已开发,待完善)
  • 紧急程度:暂不支持

3、数据编辑、审核、导出(项目介绍中的链接即最新的清单文件,点击下载

4、提供查询功能,如按地区、发布时间、物资需求筛选

5、数据可视化

操作指引

本项目使用Python3开发,请先安装Python3后安装依赖库

pip install -r requirement.txt

爬取最新的公告并更新数据库,同时生成清单(csv格式)

python main.py

此命令相当于分步执行以下三条命令

1、搜索最新公告

scrapy runspider crawler.py
# 此脚本将查询最新(近一天,关键字为:'接受 捐赠 公告')发布的公告,
# 并保存到数据库

2、下载公告内容

python downloader.py
# 此脚本用于下载(数据库中未下载的)公告

3、提取需求内容,并生成物资需求清单

python parse.py
# 此脚本会将所有公告内容进行解析,并生成清单保存到data/demands.csv中

注意:执行此命令会覆盖原清单文件!

开发指引

  • 开发语言:Python3
  • 相关依赖:Scrapy、BeautifulSoup、requests
  • 数据库:SQLite3,并且数据文件提交到代码库,以减少重复工作

    相关表结构,请查看

欢迎fork和提issue,如能提供技术帮助,请联系我

ncov-supplies's People

Contributors

anthony1874 avatar pybeaner avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.