Welcome to the RSS River Plugin for Elasticsearch

Just type :

$ bin\plugin -install dadoonet/rssriver/0.0.6

This will do the job…

-> Installing dadoonet/rssriver/0.0.6...
Trying https://github.com/downloads/dadoonet/rssriver/rssriver-0.0.6.zip...
Downloading ...DONE
Installed rssriver

We create first an index to store all the feed messages :

$ curl -XPUT 'localhost:9200/lemonde/' -d '{}'

We create the river with the following properties :

• Feed URL : http://www.lemonde.fr/rss/une.xml
• Update Rate : every 15 minutes (15 * 60 * 1000 = 900000 ms)

$ curl -XPUT 'localhost:9200/_river/lemonde/_meta' -d '{
  "type": "rss",
  "rss": {
    "feeds" : [ {
    	"name": "lemonde",
    	"url": "http://www.lemonde.fr/rss/une.xml",
    	"update_rate": 900000
    	}
    ]
  }
}'

If you need to get multiple feeds, you can add them :

• Feed1 URL : http://www.lemonde.fr/rss/une.xml
• Update Rate1 : every 15 minutes (15 * 60 * 1000 = 900000 ms)

• Feed2 URL : http://rss.lefigaro.fr/lefigaro/laune
• Update Rate2 : every 30 minutes (30 * 60 * 1000 = 1800000 ms)

$ curl -XPUT 'localhost:9200/actus/' -d '{}'

$ curl -XPUT 'localhost:9200/_river/actus/_meta' -d '{
  "type": "rss",
  "rss": {
    "feeds" : [ {
			"name": "lemonde",
			"url": "http://www.lemonde.fr/rss/une.xml",
			"update_rate": 900000
    	}, {
			"name": "lefigaro",
			"url": "http://rss.lefigaro.fr/lefigaro/laune",
			"update_rate" : 1800000
    	}
    ]
  }
}'

When you create your index, you can specify the mapping you want to use as follow :

$ curl -XPUT 'http://localhost:9200/lefigaro/' -d '

$ curl -XPUT 'http://localhost:9200/lefigaro/page/_mapping' -d '{
    "page" : {
        "properties" : {
            "feedname" : {"type" : "string"},
            "title" : {"type" : "string", "analyzer" : "french"},
            "description" : {"type" : "string", "analyzer" : "french"},
            "author" : {"type" : "string"},
            "link" : {"type" : "string"}
        }
    }
}'

Then, your feed will use it when you create the river :

$ curl -XPUT 'localhost:9200/_river/lefigaro/_meta' -d '{
  "type": "rss",
  "rss": {
    "feeds" : [ {
		    "url": "http://rss.lefigaro.fr/lefigaro/laune"
	    }
    ]
  }
}'

RSS river downloads RSS feed every update_rate milliseconds and check if there is new messages.

At first, RSS river look at the <channel> tag.
It reads the optional <pubDate> tag and store it in Elastic Search to compare it on next launch.

Then, for each <item> tag, RSS river creates a new document within page type with the following properties :

ID is generated from description using the UUID generator. So, each message is indexed only once.

Read RSS 2.0 Specification for more details about RSS channels.

Many many things to do :

• As <pubDate> tag is optional, we have to check if RSS River is working in that case and parse each feed message
• Support more RSS <channel> sub-elements, such as <category>, <skipDays>, <skipHours>
• Support more RSS <item> sub-elements, such as <category>, <enclosure>, <pubDate>
• Support for multi-channel (one per language for instance)
• Use <ttl> to manage sleep time of the river
• Use <guid> as the text to encode to generate ID