Welcome to the RSS River Plugin for Elasticsearch
RSS River Plugin | ElasticSearch |
---|---|
master (0.0.7) | master (0.19) |
0.0.6 | 0.19.0.RC1 |
0.0.5 | 0.18 |
0.0.4 | 0.18 |
0.0.3 | 0.18 |
0.0.2 | 0.17 |
Just type :
$ bin\plugin -install dadoonet/rssriver/0.0.6
This will do the job…
-> Installing dadoonet/rssriver/0.0.6... Trying https://github.com/downloads/dadoonet/rssriver/rssriver-0.0.6.zip... Downloading ...DONE Installed rssriver
We create first an index to store all the feed messages
:
$ curl -XPUT 'localhost:9200/lemonde/' -d '{}'
We create the river with the following properties :
- Feed URL : http://www.lemonde.fr/rss/une.xml
- Update Rate : every 15 minutes (15 * 60 * 1000 = 900000 ms)
$ curl -XPUT 'localhost:9200/_river/lemonde/_meta' -d '{ "type": "rss", "rss": { "feeds" : [ { "name": "lemonde", "url": "http://www.lemonde.fr/rss/une.xml", "update_rate": 900000 } ] } }'
If you need to get multiple feeds, you can add them :
- Feed1 URL : http://www.lemonde.fr/rss/une.xml
- Update Rate1 : every 15 minutes (15 * 60 * 1000 = 900000 ms)
- Feed2 URL : http://rss.lefigaro.fr/lefigaro/laune
- Update Rate2 : every 30 minutes (30 * 60 * 1000 = 1800000 ms)
$ curl -XPUT 'localhost:9200/actus/' -d '{}' $ curl -XPUT 'localhost:9200/_river/actus/_meta' -d '{ "type": "rss", "rss": { "feeds" : [ { "name": "lemonde", "url": "http://www.lemonde.fr/rss/une.xml", "update_rate": 900000 }, { "name": "lefigaro", "url": "http://rss.lefigaro.fr/lefigaro/laune", "update_rate" : 1800000 } ] } }'
When you create your index, you can specify the mapping you want to use as follow :
$ curl -XPUT 'http://localhost:9200/lefigaro/' -d ' $ curl -XPUT 'http://localhost:9200/lefigaro/page/_mapping' -d '{ "page" : { "properties" : { "feedname" : {"type" : "string"}, "title" : {"type" : "string", "analyzer" : "french"}, "description" : {"type" : "string", "analyzer" : "french"}, "author" : {"type" : "string"}, "link" : {"type" : "string"} } } }'
Then, your feed will use it when you create the river :
$ curl -XPUT 'localhost:9200/_river/lefigaro/_meta' -d '{ "type": "rss", "rss": { "feeds" : [ { "url": "http://rss.lefigaro.fr/lefigaro/laune" } ] } }'
RSS river downloads RSS feed every update_rate
milliseconds and check if there is new messages.
At first, RSS river look at the <channel>
tag.
It reads the optional <pubDate>
tag and store it in Elastic Search to compare it on next launch.
Then, for each <item>
tag, RSS river creates a new document within page
type with the following properties :
XML Tag | ES Mapping |
---|---|
<title> |
title |
<description> |
description |
<author> |
author |
<link> |
link |
<geo:lat> <geo:long> |
location |
ID
is generated from description using the UUID generator. So, each message is indexed only once.
Read RSS 2.0 Specification for more details about RSS channels.
Many many things to do :
- As
<pubDate>
tag is optional, we have to check if RSS River is working in that case and parse each feed message - Support more RSS
<channel>
sub-elements, such as<category>
,<skipDays>
,<skipHours>
- Support more RSS
<item>
sub-elements, such as<category>
,<enclosure>
,<pubDate>
- Support for multi-channel (one per language for instance)
- Use
<ttl>
to manage sleep time of the river - Use
<guid>
as the text to encode to generateID