Giter Club home page Giter Club logo

crawl_weixin's Introduction

weixin

ready

config.json

{
  "db":{
    "uris": [mongouri]
  }
}
// 或者传递3个参数,mongo uri, monitor uri, es uri

weixin crawler status

  1. gsdata
初始 --> 0
正在更新id --> 1
公众号id更新成功 --> 2
没有相同公众号 --> -1
username = #########meta######### or '' ornull --> -2
正在采集数据 --> 3
采集报错 --> 4
账号未收录 --> -3
  1. newrank
初始 --> 0
正在更新id --> 1
公众号id更新成功 --> 2
没有相同公众号 --> -1
username = #########meta######### or '' or null --> -2
正在采集数据 --> 3
采集报错 --> 4

scripts

ensure weixin

数据库添加微信 id

  1. ready
    txt 文件,每一行是微信号或者微信文章链接

  2. run

node main/scripts/ensure_weixin.js <filename>
or
npm run ensure <filename>

ensure gsdata

更新 gsdata id

node main/scripts/ensure_gsdata.js

ensure newrank

更新 newrank id

node main/scripts/ensure_newrank.js

crawl gsdata

采集 gsdata 网站数据

node main/scripts/crawl_gsdata.js

crawl newrank

采集 newrank 网站数据

node main/scripts/crawl_newrank.js

crawl content

采集文章内容

node main/scripts/crawl_content.js

crawl_weixin's People

Contributors

soonfy avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.