9b / chirp Goto Github PK

Common keywords can generate a lot of associated stories. Showing all stories for all time is not an efficient way to interact with the system. Default limits should hide stories past a certain date or a paging function should be in place by each day.

I deleted a monitor via google-alerts CLI and added a new monitor via CLI, Chirp has not reflected the change. I expected the deleted monitor to be removed from Chirp and the new monitor to be added.

There's benefit to understand a breakdown of how much information is being processed, who the top sources are and other summary details. These should be contained within a simple dashboard and provide a quick-view of platform activity.

The below error prevents new tasks from running.

[2018-05-21 15:30:00,101: WARNING/ForkPoolWorker-16]
/root/venv3/lib/python3.5/site-packages/pymongo/topology.py:149:
UserWarning: MongoClient opened before fork. Create MongoClient only after
forking. See PyMongo's documentation for details:
http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
"MongoClient opened before fork. Create MongoClient only "
[2018-05-21 15:30:06,375: ERROR/ForkPoolWorker-16] Task
process_all_rss[64432368-c40a-4708-98c1-51e125ecdcf1] raised unexpected:
TypeError("the JSON object must be str, not 'bytes'",)
Traceback (most recent call last):
File "/root/venv3/lib/python3.5/site-packages/celery/app/trace.py", line
374, in trace_task
R = retval = fun(*args, **kwargs)
File "/root/venv3/lib/python3.5/site-packages/celery/app/trace.py", line
629, in protected_call
return self.run(*args, **kwargs)
File "/root/chirp/app/tasks/init.py", line 104, in process_all_rss
response = get_article(item, source['href'], reprocess)
File "/root/chirp/app/tasks/init.py", line 75, in get_article
article['sentiment'] = get_sentiment(text_content)
File "/root/chirp/app/utils/helpers.py", line 117, in get_sentiment
loaded = json.loads(response.content)
File "/usr/lib/python3.5/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'bytes'

I commonly see news sources I consider unreliable or "spammy". I'd like to be able to blacklist or mute those sources.

Hi, I want to design a new logo for the project completely free..

Users should have the ability to receive notifications from Chirp. Queries could be built out to look for the presence of specific tags or combinations of metadata in order to generate an in-platform alert or email alert.

Exposed tokens:

• Monitor tag
• Monitor category
• Monitor term
• Article source
• Article tag
• Article text

Evaluation criteria:

• Contains
• Does not contain
• Equals
• Not equals

Actions to support:

• Send email
• Tag monitor
• Post to domain
• Delete or hide

:~/chirp# celery worker -A celery_worker.celery --loglevel=info -B
Traceback (most recent call last):
File "/root/venv3/bin/celery", line 11, in
sys.exit(main())
File "/root/venv3/lib/python3.5/site-packages/celery/main.py", line 14, in main
_main()
File "/root/venv3/lib/python3.5/site-packages/celery/bin/celery.py", line 326, in main
cmd.execute_from_commandline(argv)
File "/root/venv3/lib/python3.5/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/root/venv3/lib/python3.5/site-packages/celery/bin/base.py", line 279, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/root/venv3/lib/python3.5/site-packages/celery/bin/base.py", line 481, in setup_app_from_commandline
self.app = self.find_app(app)
File "/root/venv3/lib/python3.5/site-packages/celery/bin/base.py", line 503, in find_app
return find_app(app, symbol_by_name=self.symbol_by_name)
File "/root/venv3/lib/python3.5/site-packages/celery/app/utils.py", line 355, in find_app
sym = symbol_by_name(app, imp=imp)
File "/root/venv3/lib/python3.5/site-packages/celery/bin/base.py", line 506, in symbol_by_name
return imports.symbol_by_name(name, imp=imp)
File "/root/venv3/lib/python3.5/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/root/venv3/lib/python3.5/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
return imp(module, package=package)
File "/root/venv3/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 665, in exec_module
File "", line 222, in _call_with_frames_removed
File "/root/chirp/celery_worker.py", line 4, in
import app.tasks
File "/root/chirp/app/tasks/init.py", line 97
if not reprocess or if not processed:
^
SyntaxError: invalid syntax

If I add a new monitor, I get this error. "
Sever Error

Your request resulted in an error."
This is in the logs.

ERROR init:server_error():56 2018-11-10 03:48:19,133| 500 triggered: Failed to create monitor: b'\n\n \n \n <title>Error 500 (Server Error)!!1</title>\n <style>\n {margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px} > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n </style>\n \n

500. That\xe2\x80\x99s an error.\n

The server encountered an error and could not complete your request.

If the problem persists, please report your problem and mention this error message and the query that caused it. That\xe2\x80\x99s all we know.\n'

I can't tell if the issue is external to chirp, like maybe google changed something on their side. My google-alerts library is up to date.

There are times when celery will spawn several workers and fetch articles over and over. Due to no lock on processing information, there's the chance that the article count for a monitor could become incorrect. This should be fixed using a post-process action that resets counts based on database queries.

Google seems to eventually identify the scripted accounts and forces a captcha. There is no way to automate around this, though 2FA does appear to be a good solution, though not perfect. This is a major issue to keeping the functionality within Chirp going. While annoying, one can swap accounts and still retain access to existing monitors as the RSS feeds are public.

When viewing settings, it would be nice to allow users to test their credentials and identify if a session token has been saved. This should eliminate some confusion when the application throws an error due to invalid credentials.

The following administrative items needed to be added within the codebase.

• Mark first user as an administrator
• Allow administrators to add other users (generate email invite)
• Allow administrators to define and manage groups
• Allow administrators a way to delete or disable users

There should be a way to filter the list of monitors by details contained within them. This may include the monitor name, category, tags or last processed time. Added a filter makes it easier to find the monitor of interest when many are deployed.

Information displayed for each article is meant to be sparse and enough to capture the idea. If users want to share this with peers, they should be able to do so with ease through social buttons.

Give users the ability to see the monitors they have deployed, update them and allow for deletion. This is just a user element on top of the standard functions built into the Google Alerts library.

I get this error when I try to add new monitor. I tried to install with Docker and without it, but an error was the same.
2018-06-15 16:18:57,052] ERROR in app: Exception on /monitors/add-monitor [POST]
Traceback (most recent call last):
File "/var/chirp/venv3/lib/python3.5/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/var/chirp/venv3/lib/python3.5/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/var/chirp/venv3/lib/python3.5/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/var/chirp/venv3/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/var/chirp/venv3/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/var/chirp/venv3/lib/python3.5/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/var/chirp/venv3/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
return func(*args, **kwargs)
File "/var/chirp/app/core/monitors.py", line 28, in add_monitor
ga = GoogleAlerts(gdata['email'], gdata['password'])
TypeError: 'NoneType' object is not subscriptable

My mongoDB became corrupt, so I had to wipe the DB, which wiped all connection info. Adding my google account to the GUI errors that there is a captcha and authenticating via the browser does not allow me to get around the captcha.

The google-alerts CLI works as expected and pulls info with no issues. This appears to be unique to the chirp GUI.

"Authentication Failed! Google is forcing a CAPTCHA. To get around this issue, authenticate within your web browser, pass the CAPTCHA and try to run this script again. Once authenticated, this module will cache your session and load that in the future"

After starting Celery, this warning is given about Mongo being opened after a fork:

[2018-05-26 11:00:38,234: WARNING/ForkPoolWorker-8] /Users/adam/Research/projects/google-alerts-pipeline/chirp/venv3/lib/python3.6/site-packages/pymongo/topology.py:149: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
  "MongoClient opened before fork. Create MongoClient only "

I believe this is due to the celery processes and multiple Mongo instances being opened. You can usually get around this by checking if the PID is 0 and if yes connect to Mongo, if not re-use an existing connection since one will already be opened - though since you're using Flask-Mongo I'm not sure if this is something you can deal with on the Chirp side..

I took a quick glance through the code but couldn't find an obvious spot where this would be happening on your end but figured I'd let you know since it will just keep creating new Mongo connections whenever it's called