news-ai / knowledge Goto Github PK

Key/value store for topic -> information about the topic

Python 100.00%

knowledge's Introduction

news-ai

A specification for building NewsAI

What is NewsAI?

Exploring the long-term changes that can happen in the consumption and creation of news through the emergance of AI techniques, and advancements in Machine Learning.

Long-term projects that can built

Giving context to articles discussed on Facebook Messenger. Similar to the @fbchess. Gives you an overview of the current afairs news by doing @currentnews or @news which gives you back a summary and some contextual information.
Consumer Apps:
- A new experience of reading the news. It provides not just the news article, but contextual information on top of it.
- A way for consumers to want to stay updated on a news topic. This is if you want to stay updated to the ISIS bombing. We know that ISIS bombing is a "topic" in the database that has several "articles" underneath it. Then when a new article is added to that topic you can push to that user.
- Catch up on X or something like "You missed news X" while you were gone.

Problems

There's a lot of news. How can you get it, analyze it, and use it to predict things?
How can you model things like Timelines and key topics that occur in news?

Notes

Individual contributors are now able to build up their own brand given the new social media aspects of things. This is a way to centralize and provide a platform for them to be able to get up to date news information, but also be able to explore archival information for news.

We can generate things like timelines for entities, entity information, old news articles, articles written about same perspective by different publishers, etc.

knowledge's People

Contributors

Watchers

knowledge's Issues

Using bios of people on Twitter to construct employee data about companies

Broken pipe error

127.0.0.1 - - [12/Apr/2016 05:25:19] "POST /knowledge_server HTTP/1.0" 200 -
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe

Integrate with API to get requests whenever new article is added

Go back and process unprocessed articles

Issue with duplicates again

https://publisher.newsai.org/articles/10201

KeyError in script

[2016-04-12 05:22:32,517: ERROR/MainProcess] Task articles.process_single_article[301883f4-2ddb-4478-827f-b7ce1f101b1b] raised unexpected: KeyError('id',)
Traceback (most recent call last):
  File "/var/apps/knowledge/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/var/apps/knowledge/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/var/apps/knowledge/knowledge/articles.py", line 27, in process_single_article
    response = entity_extract(article, types, token)
  File "/var/apps/knowledge/knowledge/knowledge/entity_extraction.py", line 21, in entity_extract
    entity, types, token, single_entity_api, api_entity_id_added)
  File "/var/apps/knowledge/knowledge/knowledge/internal/context.py", line 123, in add_entityscore_to_api
    if api_entity['id'] not in api_entity_id_added:
KeyError: 'id'

Add Flask to make it easier to POST url and id data from the API

Cache entity lookup from the API

Cache Entity Name -> ID in the API.

Automatically add types not working

[2016-03-19 05:38:58,201: ERROR/MainProcess] Task articles.process_single_article[7fa43e35-805c-4392-af5a-6f5a7fcb7277] raised unexpected: KeyError(u'BasketballTeam',)
Traceback (most recent call last):
File "/var/apps/knowledge/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(_args, *_kwargs)
File "/var/apps/knowledge/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
return self.run(_args, *_kwargs)
File "/var/apps/knowledge/knowledge/articles.py", line 24, in process_single_article
response = entity_extract(article, types, token)
File "/var/apps/knowledge/knowledge/knowledge/entity_extraction.py", line 190, in entity_extract
single_entity_api = add_entity_to_api(entity, types, token)
File "/var/apps/knowledge/knowledge/knowledge/entity_extraction.py", line 85, in add_entity_to_api
types_url + str(types[sub_type]['id']) + '/')
KeyError: u'BasketballTeam'