shanelau / rssspider Goto Github PK

Rss spider by nodejs , rss 爬虫，正文抓取

JavaScript 100.00%

rssspider's Introduction

rssSpider

Design and coding with all the love in the world by ShaneLau.

The simplest way to use rssspide to fetch rss list and site info.
Fetch post'content ,give clean view to you.
rss 爬虫，快速抓取站点信息和文章列表，文章的正文抓取

This project is base on feedparser and node-readability

Usage

npm install rssspider

Then:

var spide = require('rssspider');
var url = 'http://www.bigertech.com/rss';
spide.fetchRss(url).then(function(data){
		console.log(data); // rss  post list
});

API Documentation

1. `fetchRss(url,[options])`

get rss site'post list ,like this www.bigertech.com/rss

url : webiste'rss url
options :what data you need ? default value:

	['title','description','summary','date','link','guid','author','comments','origlink','image','source','categories','enclosures']

response data Array

[{ title: '一个营销人员的自我修养',
  description: '<p></p>',
  summary: '</p>',
  date: Wed Oct 08 2014 17:14:26 GMT+0800 (CST),
  link: 'http://www.bigertech.com/learn-social-media-marketing/',
  guid: 'a623d78a-dae9-4915-9caa-0fd34fb3757c',
  author: '巴依老爷',
  comments: null,
  origlink: null,
  image: {},
  source: {},
  categories: [],
  enclosures: [] },
  ....  // more
	]

2. `siteInfo(url,[options])`

get website info

url webiste'rss url
options what data you need ? default value:

['title','description','date','link','xmlurl','author','favicon','copyright','generator','image']

```

response data Array

{ title: '笔戈科技',
description: '简单、有趣、有价值',
date: Thu Oct 09 2014 18:15:14 GMT+0800 (CST),
link: 'http://www.bigertech.com/',
xmlurl: 'http://www.bigertech.com/rss/',
author: null,
favicon: null,
copyright: null,
generator: 'Ghost 0.5',
image: {},
feedurl: 'http://www.bigertech.com/rss' }

** 以下功能在 1.2.0 才能使用， readability 的库支持不是很好 **

3. `getCleanBody(url)`

Turn any web page into a clean view. This module is based on arc90's readability project.

html url or html code.
options is an optional options object
callback is the callback to run - callback(error, article, meta)

var url = 'http://www.bigertech.com/learn-social-media-marketing/';
spide.getCleanBody(url).then(function(article){
      console.log(article.content);   //clean code view
  });

More info node-readability

article.content is clean view

The article content of the web page. Return false if failed.

4. `getAllByUrl(url,[options])`

This method is similar to fetchRss
####What'more ,it fetch the clean page content. Turn any web page into a clean view. This module is based on arc90's readability project.

url website'rss url
Array respose data

get clean view code , Clean view content


[{ title: '一个营销人员的自我修养',
   content:'clean code view',     // clean code view
  description: '<p></p>',
  summary: '</p>',
  date: Wed Oct 08 2014 17:14:26 GMT+0800 (CST),
  link: 'http://www.bigertech.com/learn-social-media-marketing/',
  guid: 'a623d78a-dae9-4915-9caa-0fd34fb3757c',
  author: '巴依老爷',
  comments: null,
  origlink: null,
  image: {},
  source: {},
  categories: [],
  enclosures: [] },
    ....... // more
	]

test 100%

nodeunit test/index.js

upgrade

Add node 4.x support

Any question shanelau

or
[email protected]

rssspider's People

Contributors

Stargazers

Watchers

Forkers

strucoder thesadboy torome mailyupeng zumaid lll4770 fish1919 ledudu yolio2003 marsbaiyun xbbopen wanghaisheng ocean315 wanily cdrr kevinston vinctzh jasongj aiex medivhna esironal njaulj sxh2000251 robustfengbin finderl sunnycoder lihang1870719 scott2lsb majinhui04 stonegithubs cctshiyu lf2941270 networkrepositories xmanapp kimswe huanglp47 durban89 jason-bai failymiss shenzhenlong1203 podcctv wangyhao wsyandy licong trynget king0222 heixiaoshan sswjzx applelee lykmapipo doubleshift ronald1986 sagacite2 iaceob lixiangnlp yzhenma everywill lizeqiangd yuanjack sunyymq zhoulijie neverwell cobola sevencon wooodhead haichongsihao wesavetheworld youthchinagirl tuian phpsmarter one2ter lrysir veg0s linkfar mionikwang lijinhuai a382695908 jakefeng sean0626 grzhan alpharootbeta webkid760 saoinformaticsteam hhy5277 longtan01 yijun0819 cocworkshop lovessures coffiner kawhiwei mathlon madcc gumpbobo iq-scm joyce0225

rssspider's Issues

你这里的图片完全取决于 RSS 原输出是否包含图片，并没有进一步到新闻的url中去提取featured 图片

http://kissliux.github.io/rssSpider 打不开

如题

nodeunit test/index.js -----Unhandled rejection Error: ETIMEDOUT

[demon@demon-pc:~/rssSpider/rssSpider → master]$ nodeunit test/index.js

index.js
Unhandled rejection Error: ETIMEDOUT
at null._onTimeout (/home/demon/rssSpider/rssSpider/node_modules/request/request.js:909:15)
at Timer.listOnTimeout (timers.js:110:15)

FAILURES: Undone tests (or their setups/teardowns):

fetchRSS

To fix this, make sure all tests call test.done()

lib/rss.js:68 没有判断meta非空

没有判断meta非空的情况。

........./node_modules/rssspider/lib/rss.js:68
        encoding = meta.toString().split('"')[1];
                       ^

TypeError: Cannot read property 'toString' of null
    at Stream.<anonymous>

我本地添加修改后正常：

encoding = meta && meta.toString().split('"')[1];

rss 抓取非标准的http headers 未转码为UTF8

例如：http://news.baidu.com/n?cmd=1&class=civilnews&tn=rss
抓取之后乱码

npm install rssspider 后有问题

npm WARN package.json [email protected] 'repositories' (plural) Not supported. Please pick one as the 'repository' field
npm WARN install Refusing to install rssspider as a dependency of itself

求解，3Q

当我node更新到4.2.1后，这个模块就出问题啦。

当我node更新到4.2.1后，这个模块就出问题啦。
贴部分错误：

gyp ERR! node -v v4.2.1
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok
npm ERR! Darwin 15.0.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "rssspider"
npm ERR! node v4.2.1
npm ERR! npm v2.14.7
npm ERR! code ELIFECYCLE

npm ERR! [email protected] install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-gyp rebuild'.
npm ERR! This is most likely a problem with the contextify package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls contextify
npm ERR! There is likely additional logging output above.

怀疑是作者的包里用的contextify错误。
最后，为作者的贡献表示感谢～