abh1nav / gnippy Goto Github PK
View Code? Open in Web Editor NEWPython library for GNIP
License: Other
Python library for GNIP
License: Other
This does not work if conf['url']
has params.
rules_url = _generate_rules_url(conf['url']) + "?_method=delete"
Example generated URL:
https://api.gnip.com:443/accounts/XXX/publishers/twitter/streams/track/prod/rules.json?client=2?_method=delete
After the #7 merge, even if I have no GNIPPY_
environment variables set, gnippy is still trying to use them and ignores my valid provided config file.
Here is some relevant traceback:
File "configurator.py", line 74, in list_rules
all_rules = rules.get_rules(config_file_path=config_file)
File "/Users/irina/.pyenv/versions/casterisk/lib/python2.7/site-packages/gnippy/rules.py", line 150, in get_rules
rules_url = _generate_rules_url(conf['url'])
File "/Users/irina/.pyenv/versions/casterisk/lib/python2.7/site-packages/gnippy/rules.py", line 13, in _generate_rules_url
if ".json" not in url:
TypeError: argument of type 'NoneType' is not iterable
I have URL defined in my config file, but none of the GNIPPY_
environment variables.
GNIPPY_AUTH_USERNAME
GNIPPY_AUTH_PASSWORD
GNIPPY_URL
With PowerTrack 2.0 migration coming up (deadline is December 1, 2016), the following will not be valid anymore:
Lines 17 to 24 in 9765bbe
Instead of:
https://api.gnip.com:443/accounts/<account-name>/publishers/twitter/streams/track/<stream-label>/rules.json
rules will have to be posted to:
https://gnip-api.twitter.com/rules/powertrack/accounts/<account-name>/publishers/twitter/<stream-label>.json
.
It would be better if gnippy accepted rules_url
as part of the config.
In addition, rule deletion will now be done with a POST request and &_method=delete
parameter (from here).
I encountered a rare "Connection reset by peer" when consuming PowerTrack stream:
https://github.com/abh1nav/gnippy/blob/master/gnippy/powertrackclient.py#L75
The exception was thrown in iter_lines
; unfortunately, I don't have the traceback available.
Because this error is happening within a worker thread, and there's no way to catch that, the calling thread also dies. I need a way to capture the error event and restart the connection to PowerTrack.
Suggestion:
exception_callback
parameter when creating PowerTrackClient
run
exception_callback
provided, that would be called, letting the caller know that a reconnect may be neededexception_callback
is provided, re-raise.If that sounds ok, I'd implement that.
Thoughts?
It would be great if the client could throw an error if the credentials are wrong. Right now looks like the client is just waiting for data to come back. Happy to work on this if you like.
Thanks for your great work on this!
@abh1nav We're using gnippy
in parse.ly. We need a release after this PR was merged, otherwise our dependencies end up in a bad state:
Any chance you could do a release, or make me or @dan-blanchard a gnippy
maintainer on pypi?
When Worker
is requested to stop it will do so at once when possible, which is when r.iter_lines
wakes up and returns data. Depending on the stream and rules this could either be just an ack with high probability, or precious JSON.
for line in r.iter_lines():
if self.stopped():
break
elif line:
self.on_data(line)
It is a bit of an opinionated subject, but to me it seems like it should handle the possibly received data and then stop.
Is there any plan to add Replay API wrapper to this library? Seems that it's the only well-maintained GNIP python library out here
Hi,
I am facing a delay of approximately 10 minutes from the time I tweet to the time I get ithe tweet in callback. Is there a way out where this delay can be avoided.
Thanks,
Shruti
If I do this:
all_rules = rules.get_rules()
rules.delete_rules(all_rules)
I get an exception:
Traceback (most recent call last):
File "configurator.py", line 93, in <module>
delete_rules(popts.file_name)
File "configurator.py", line 43, in delete_rules
rules.delete_rules(all_rules)
File "/Users/irina/.pyenv/versions/casterisk/lib/python2.7/site-packages/gnippy/rules.py", line 180, in delete_rules
_delete(conf, rules_list)
File "/Users/irina/.pyenv/versions/casterisk/lib/python2.7/site-packages/gnippy/rules.py", line 96, in _delete
_check_rules_list(built_rules)
File "/Users/irina/.pyenv/versions/casterisk/lib/python2.7/site-packages/gnippy/rules.py", line 50, in _check_rules_list
fail()
File "/Users/irina/.pyenv/versions/casterisk/lib/python2.7/site-packages/gnippy/rules.py", line 33, in fail
raise RulesListFormatException(msg)
gnippy.errors.RulesListFormatException: rules_list is not in the correct format. Please use build_rule to build your rules list.
In my case, all_rules
only contains one item and looks like this:
[{u'tag': None, u'value': u'url_contains:"mashable.com"'}]
It seems that {'tag'=None} is the problem when submitting to delete API. If my list looks like this:
[{u'value': u'url_contains:"mashable.com"'}]
then there's no exception and rules are deleted.
Right now, Gnippy uses requests strictly pinned to 2.8.1:
requests==2.8.1
This is a conflicting requirement with one of the packages we use that requires latest requests (not pinned).
Can this be changed to requests>=2.8.1
?
The examples in the README for deleting rules are broken due to the addition of the id field in the rules returned by GNIP.
from gnippy import rules
from gnippy.errors import RuleDeleteFailedException, RulesGetFailedException
try:
rules_list = rules.get_rules()
# Suppose I want to delete the first rule in the list
rules.delete_rule(rules_list[0])
# OR ... I want to delete ALL rules
rules.delete_rules(rules_list)
except RuleDeleteFailedException, RulesGetFailedException:
pass
At the moment a user needs to either manually remove the id
field
del rule['id']
rules.delete_rule(rule)
or rebuild the rule using rules.build
rules.delete_rule(rules.build(rule['value'], rule.get('tag')))
When this happens it would be good to be able to recover gracefully.
I'm using:
$ python -V
Python 2.7.12
BTW Many thanks for a great library.
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/lib/python2.7.12/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/home/cabox/env/g2/lib/python2.7/site-packages/gnippy/powertrackclient.py", line 133, in run
for line in r.iter_lines():
File "/home/cabox/env/g2/lib/python2.7/site-packages/requests/models.py", line 720, in iter_lines
for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
File "/home/cabox/env/g2/lib/python2.7/site-packages/requests/models.py", line 679, in generate
raise ChunkedEncodingError(e)
ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read, 512 more expected)', IncompleteRead(0 bytes read, 512 more expecte
```d))
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.