motazsaad / feedsucker Goto Github PK
View Code? Open in Web Editor NEWThis project forked from dkorenci/feedsucker
Tool for collecting texts from a set of feeds and storing them in a relational database
License: Other
This project forked from dkorenci/feedsucker
Tool for collecting texts from a set of feeds and storing them in a relational database
License: Other
Feedsucker is an application for collecting texts from a set of feeds over a period of time and storing them in a relational database. Main use case is collection of texts from web media outlets. Feedsucker is a free software, licensed under the Apache License, Version 2.0 Documentation is contained in the following folders: doc/structure/ : Structure of the main application code, and supporting functionality. doc/deploy/ : Deployment instructons doc/todo/ : List of functionality to add, todo lists and bug lists. Feedsucker is written in Java (Java 7 or above is required), with interface to newspaper written in Python and with control and support functionality written in Bash. For this reason, it is deployable on Linux only, without some modifications (these are on todo list, see doc/todo/new_functionality.txt). Supported feeds are rss and atom feeds and a class of html pages (containing URLs with specific structure common for news sites). For text scraping, feedsucker relies on newspaper, a tool written in Python. In general, a feed can be viewed as any source of URLs or other addresses pointing to resources containing text that can be scraped/extracted. Feedsucker is extensible and new scrapers (IArticleScraper classes) and feed readers (IFeedReader classes) can be written and used within app workflow. Feedsucker was developed as a research tool for creating media text corpora. It enables a researcher to collect texts from a set of outlets/feeds of interest. Following research articles are based on data collected with Feedsucker: "Getting the Agenda Right: Measuring Media Agenda using Topic Models" "Issues and their Salience in the 2015 Parliamentary Election in Croatia: A Topic Model based Analysis of the Media Agenda" (to appear) Largest deployment up to date collected 1.1 million texts from 73 feeds (25 outlets) over a period of 14 months, but the app should easily handle at least hundreds of feeds.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.