A simple feed aggregator service with sugar on top.
- Single executable with an embedded database.
- Manage feed subscriptions.
- Import/Export feed subscriptions with OPML files.
- Aggressive and tunable aggregation process.
- Manage feed aggregation individually.
- Apply modifications on articles with a pluggable filter system.
- Push new articles to a pluggable output system (STDOUT, HTTP, EMAIL, readflow, Twitter ...).
- Customize the pipeline thanks to a powerful expression language.
- Support of WebSub the open, simple, web-scale and decentralized pubsub protocol.
- REST API with complete OpenAPI documentation.
- Authentication mechanisms such as OpenID Connect.
- Full feature Web UI and CLI to interact with the API.
- Metrics production for monitoring.
- Quota.
Run the following command:
$ go get -v github.com/ncarlier/feedpushr/v3
Or download the binary regarding your architecture:
$ curl -s https://raw.githubusercontent.com/ncarlier/feedpushr/master/install.sh | bash
$ # Or with https://gobinaries.com/
$ curl -sf https://gobinaries.com/ncarlier/feedpushr | sh
Or use Docker:
$ docker run -d --name=feedpushr ncarlier/feedpushr
Feedpushr can be configured by using command line parameters or by setting environment variables.
Type feedpushr -h
to display all parameters and related environment variables.
All configuration variables are described in etc/default/feedpushr.env file.
You can define tags on feeds using the Web UI or the API:
$ curl -XPOST http://localhost:8080/v2/feeds?url=http://www.hashicorp.com/feed.xml&tags=foo,bar
Tags can also be imported/exported in OPML format. When using OMPL, tags are stored into the category attribute. OPML category is a string of comma-separated slash-delimited category strings.
For example, this OMPL attribute <category>/test,foo,/bar/bar</category>
will be converted to the following tag list: test, foo, bar_bar
.
Once feeds are configured with tags, each new article will inherit these tags and be pushed out with them.
New articles are sent to outputs.
Currently, there are some built-in output providers:
Output | Properties | Description |
---|---|---|
stdout |
format |
New articles are sent to the standard output of the process. This can be useful if you want to pipe the command to another shell command. ex: Store the output into a file. Forward the stream via Netcat . Use an ETL tool such as Logstash, etc. |
http |
url contentType format |
New articles are sent to an HTTP endpoint (POST). |
email |
host username password format ... |
New articles are sent by email to a SMTP host. |
readflow |
url (default: official API apiKey |
New articles are sent to readflow instance. |
Outputs can be extended using plugins.
Outputs are activated according to a conditional expression. If the output has no condition it will be activated regardless of the input article. Otherwise the expression will be applied to the article and its result will decide whether the output is used or not.
For example, if we want to use an output only on articles tagged news and mentioning Paris in their title.
We can use the following expression: "news" in Tags and Title contains "Paris"
The conditional expression language is documented here.
For some outputs (stdout
, http
and some plugins) the output format is configurable.
If the format is not specified, the output will be formatted as the following JSON document:
{
"title": "Article title",
"text": "Article text description",
"content": "Article HTML content",
"link": "Article URL",
"updated": "Article update date (String format)",
"updatedParsed": "Article update date (Date format)",
"published": "Article publication date (String format)",
"publishedParsed": "Article publication date (Date format)",
"guid": "Article feed GUID",
"feedTitle": "Feed title",
"meta": {
"key": "Metadata keys and values added by filters"
},
"tags": ["list", "of", "tags"]
}
If, on the other hand, you specify a format, it must comply with the Golang template syntax.
Basically, you have access to the JSON properties above but prefixed with a dot and a capital letter all surrounded by 2 brackets.
For example, the title
property is accessible with the syntax : {{.Title}}
.
Let's say you want to send a JSON payload to a Mattermost Webhook. You can format the payload like this:
{
"text": ":tada: {{.Title}} (<{{.Link}}|more>) cc @all",
}
You can use template functions with some additions:
tweet <text> <suffix>
: Create a small message (text and a suffix) that don't exceed 270 characters. Thetext
parameter can be truncated. Example:"{{ tweet .Title .Link }}"
truncate <length> <text>
: Truncate a text with a max length. Example:"{{ truncate 200 .Text }}"
Before being sent, articles can be modified through a filter chain.
Currently, there are some built-in filter:
Filter | Properties | Description |
---|---|---|
title |
prefix (default: feedpushr: ) |
This filter will prefix the title of the article with a given value. |
fetch |
None | This filter will attempt to extract the content of the article from the source URL. |
interest |
wordlist (default: news ) |
This filter will keep articles that match a (comma-separated) list of words of interest. |
minify |
None | This filter will minify the HTML content of the article. |
Filters can be extended using plugins.
Like outputs, filters are activated according to a conditional expression. Filters are attached to an output, so they inherit the conditional expression of the output.
You can easily extend the application by adding plugins.
A plugin is a compiled library file that must be loaded when the application starts.
Plugins inside $PWD
are automatically loaded.
You can also load a plugin using the --plugin
parameter.
Example:
$ feedpushr --plugin ./feedpushr-twitter.so
You can find some external plugins (such as for Twitter) into this directory.
Feedpushr can be started in "desktop mode" thanks to a launcher.
The purpose of feedpushr-launcher
is to start the daemon and add an icon to your taskbar.
This icon allows you to control the daemon and quickly access the user interface.
You can access Web UI on http://localhost:8080/ui
You can restrict access to Feedpushr using HTTP basic authentication or OpenID Connect.
To activate basic authentication, you have to create a htpasswd
file:
$ # create passwd file the user 'admin'
$ htpasswd -B -c .htpasswd admin
This command will ask for a password and store it in the htpawsswd file.
Please note that by default, Feedpushr will try to load the .htpasswd
file.
But you can override this behavior by specifying the location of the file:
$ export FP_AUTHN=/etc/feedpushr.htpasswd
$ # or
$ feedpushr --authn /etc/webhookd/users.htpasswd
To activate OIDC authentication, you have to specify OIDC issuer URL:
$ export FP_AUTHN=https://accounts.google.com
$ # or
$ feedpushr --authn https://accounts.google.com
Note that you can restrict access for a specific username like this:
$ export FP_AUTHORIZED_USERNAME=xxx
$ # or
$ feedpushr --authorized-username xxx
$ # Start service with default configuration:
$ feedpushr
$ # Start service with custom configuration:
$ export FP_DB="boltdb:///var/opt/feedpushr.db"
$ export FP_DELAY=20s
$ export FP_LOG_LEVEL=warn
$ feedpushr
$ # Add feed with the CLI
$ feedpushr-ctl create feed --url http://www.hashicorp.com/feed.xml
$ # Add feed with cURL
$ curl -XPOST http://localhost:8080/v2/feeds?url=http://www.hashicorp.com/feed.xml
$ # Import feeds from an OPML file
$ curl -XPOST http://localhost:8080/v2/opml -F"[email protected]"
$ # List feeds
$ feedpushr-ctl list feed
$ # Get a feed
$ feedpushr-ctl get feed --id=9090dfac0ccede1cfcee186826d0cc0d
$ # Remove a feed
$ feedpushr-ctl delete feed --id=9090dfac0ccede1cfcee186826d0cc0d
$ # Stop aggregation of a feed
$ feedpushr-ctl stop feed --id=9090dfac0ccede1cfcee186826d0cc0d
$ # Start aggregation of a feed
$ feedpushr-ctl start feed --id=9090dfac0ccede1cfcee186826d0cc0d
$ # Get OpenAPI JSON
$ curl http://localhost:8080/swagger.json
$ # Get runtime vars
$ curl http://localhost:8080/v2/vars
$ # Here a quick ETL shell pipeline:
$ # Send transformed articles to HTTP endpoint using shell tools (jq and httpie)
$ feedpushr \
| jq -c "select(.title) | {title:.title, content:.description, origin: .link}" \
| while read next; do echo "$next" | http http://postb.in/b/i1J32KdO; done
To be able to build the project you will need to:
- Install
makefiles
external helpers:$ git submodule init $ git submodule update
- Install
goa
:$ go get -u github.com/goadesign/goa/...@v1
- Install dependencies if you want to build the launcher:
$ sudo apt-get install gcc pkg-config libgtk-3-dev libappindicator3-dev libwebkit2gtk-4.0-dev libayatana-appindicator3-dev
Then you can build the project using make:
$ make
Type make help
to see other possibilities.
GNU General Public License v3.0
See LICENSE to see the full text.