Investigate Approaches to Query News Articles Via Web about geopoliticsdashboard HOT 4 CLOSED

News API provides multiple highly helpful api calls to use in our project including the ability to fetch news articles by country, by category, by news source, etc. Also provides methods to find trending articles or sort by popularity (number of clicks). More info on the site: https://newsapi.org/.

Results from api calls are returned in the form of a json with each item being a news article. Each item contains the article author, article name, article link, and a description.

"source": {
     "id": "bbc-news",
     "name": "BBC News"
 },
 "author": "BBC News",
 "title": "Afghanistan: Taliban announce new rules for female students",
  "description": "Students will be segregated by gender and have to observe an Islamic dress code",
  "url": "http://www.bbc.co.uk/news/world-asia-58537081",
  "urlToImage": "https://ichef.bbci.co.uk/news/1024/branded_news/B54C/production/_120521464_talibanwomensmaller.jpg",
  "publishedAt": "2021-09-12T13:07:20.6935178Z",
 "content": "image captionPro-Taliban students attend a Taliban rally at the Shaheed Rabbani Education University in Kabul on Saturday.\r\nUniversities in Afghanistan are to be segregated by gender and a new Islami… [+2599 chars]"

The summary can be used in place of an ML summarizer solution or as a baseline comparison for us to quantitatively validate our model. The categories could also realistically scrap the need for a topic clustering solution. This might be preferable since it seems like this kind of undefined n-type classifier seems quite difficult to pull off.

from geopoliticsdashboard.

In terms of parsing an article's HTML most websites have a very simple layout with the article contents under a single div container, although some websites have multiple sub-containers inside or bundle the lower block on author information into the same overall div container, making things more complicated.

edit: irrelevant

from geopoliticsdashboard.

Aside:
Articles are really long and I'm not sure how sustainable it is to pass multiple articles into a complex neural network and have reasonable runtime. Not sure if this is an actual problem though, perhaps certain servers provide inference accelerators which would make this issue pretty much irrelevant.

from geopoliticsdashboard.

Aside:
Look into fake news classification https://www.kaggle.com/mrisdal/fake-news/discussion

from geopoliticsdashboard.