fkztw / py-feedr Goto Github PK

View Code? Open in Web Editor NEW

This project forked from manvari/py-feedr

24.0 2.0 7.0 66 KB

A Python parser to tweet the latest updates from multiple RSS feeds.

Home Page: https://twitter.com/M157q_News_RSS

License: Do What The F*ck You Want To Public License

Python 100.00%

python rss twitter twitter-bot

py-feedr's Introduction

py-feedr

feedr is a non-asynchronous, WTFPLv2-licensed Python parser to tweet the latest updates from multiple RSS feeds.

Sorry, what does it do?

feedr tweets the newest element from a RSS feed to the Twitter account of your choice, and stores it in a database.

It's pretty easy: you feed it with a list with links to RSS feeds, then it does a bit of formatting to create a nice, readable message, and it tweets it.

A note on feedr's behaviour

feedr tries to stick as much as possible to the RSS feeds it handles but avoids redundancy, all of this without taking too much system resources.

To keep it simple and do just that, it will only care about the latest element in a RSS feed. If it is found to be a duplicate of the last (n-1) element that feedr has already handled, then it will only keep this new element and delete the previous one on Twitter (and in the database).

Not convinced?

If you're unsure whether feedr is the solution you need, check out this Twitter account which uses feedr.

Installation

feedr can either be installed through pip (stable version) or manually (dev or stable version).

Using pip:

$ (sudo) pip install py-feedr

Manually:

Stable version: download the latest release version
Development version: clone this git repository ($ git clone https://github.com/M157q/py-feedr.git)

Then run $ (sudo) python setup.py install to install imgur-scraper.

Requirements

Installation requirements:

Runtime requirements:

A Twitter account with valid API credentials.

Usage

feedr needs to be given an INI configuration file, which contains amongst other parameters a path to a JSON feedlist of RSS feeds that follows the example format.

Then simply invoke feedr:

usage: feedr [-h] config  
  
positional arguments:  
  config      path to the feedr configuration file  
  
  optional arguments:  
    -h, --help  show this help message and exit

You can setup a cron job that runs every minute if you need to call feedr regularly:
*/1 * * * * /usr/bin/bash -c "/usr/bin/python3 ${FEEDR_DIR}/bin/feedr ${FEEDR_DIR}/bin/feedr.ini" >> /home/feedr/feedr.log 2>&1

License

py-feedr is licensed under the WTFPLv2 license; refer to the LICENSE file.

py-feedr's People

Stargazers

Watchers

Forkers

mysteryyyy ysmoe weiweiwoxin postoor gregorycoleman wajika cyrixhero

py-feedr's Issues

Try to not upload the tweet button image when tweets for posts on slashdot

e,g.

Do not send tweet when the entry is too old

Sometimes the source of the feed just suddenly added entries which are very old with unknown reason, and feedr will treat them like new ones. This make feedr send lots of tweets in a short time and banned by Twitter (Remove the access token directly and got {'errors': [{'code': 89, 'message': 'Invalid or expired token.'}]}). To prevent from this, I think adding a check in the monitor and make sure feedr won't send tweets for entries which are too old. (but still need to be updated into db)
Maybe just temporarily set the time to 7 days.

urlretrieve got 403 forbidden while using browser is ok for some images.

example: http://www.dailycold.tw/wp-content/uploads/2015/12/merry-christmasmr-lawrence-680x1024.jpg

Maybe can find another way to get the image successfully.

[Bug] Keep sending duplicated taipei.py meetup info

Linux Update && Upgrade error

root@w3w:~# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up python3.4 (3.4.2-1+deb8u3) ...
File "/usr/lib/python3.4/http/client.py", line 1014
raise InvalidURL(f"URL can't contain control characters. {url!r} "
^
SyntaxError: invalid syntax
dpkg: error processing package python3.4 (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
python3.4
E: Sub-process /usr/bin/dpkg returned an error code (1)

Chinese characters of inside.com.tw becomes gibberish

https://twitter.com/M157q_News_RSS/status/691108372293165056

Remove <img> tag in description

For example: https://twitter.com/M157q_News_RSS/status/680775354928087041
We don't need ![periscope-android](https when the image has alread been uploaded as a medium in the tweet.

Let Twitter users get the latest news of specific feed(s) via DM

Because there's no webhook or streaming Twitter API, it's hard to implement a bot command for this.
Just store a simple user subscription list in a new table (user and subscribed feed mapping) and send the latest news to these users via Twitter DM.

UnicodeEncodeError when encounters Chinese characters in img_url

  File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 464, in open
    response = self._open(req, data)
  File "/usr/lib/python3.4/urllib/request.py", line 482, in _open
    '_open', req)
  File "/usr/lib/python3.4/urllib/request.py", line 442, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 1211, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.4/urllib/request.py", line 1183, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.4/http/client.py", line 1137, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1172, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/lib/python3.4/http/client.py", line 1014, in putrequest
    self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 44-45: ordinal not in range(128)

Crushed when encounters with no <src> in <img> tag

Traceback (most recent call last):
  File "/usr/local/bin/feedr", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/root/M157q_RSS/bin/feedr", line 44, in <module>
    consumer_secret
  File "/root/M157q_RSS/feedr/monitor.py", line 80, in monitor
    self.tweetupdate.tweet_latest_update(self.latest_entry)
  File "/root/M157q_RSS/feedr/tweetupdate.py", line 72, in tweet_latest_update
    img_url = self.get_entry_img_url(feed_entry)
  File "/root/M157q_RSS/feedr/tweetupdate.py", line 55, in get_entry_img_url
    return img_tag['src']
  File "/usr/local/lib/python3.4/dist-packages/bs4/element.py", line 958, in __getitem__
    return self.attrs[key]
KeyError: 'src'

Try to shrink the file size of the oversized media

If media is too large within a tweet, feedr will replace it with the url of the media now.
But users still need to click to see the pic, maybe we can shrink the media file size like generate a thumbnail of the media let it can be uploaded to twitter.
Maybe using the pillow module can do this.