Giter Club home page Giter Club logo

news-aggregator's Introduction

News Aggregator

This project is the backend side of Brave News, and it fetches the articles from the Brave-defined publishers and shows their feeds/News in the Browser.

For more details: https://brave.com/brave-news-updates/

Setup

Dependencies

Required setup:

virtualenv -p /usr/bin/python3.9 .venv
. .venv/bin/activate
pip install -r requirements.dev.txt

Running locally

To generate sources and list of feeds:

export PYTHONPATH=$PWD:$PWD/src
NO_UPLOAD=1 NO_DOWNLOAD=1 python src/csv_to_json.py

To generate browser feed and images:

export PYTHONPATH=$PWD:$PWD/src
NO_UPLOAD=1 python src/feed_processor_multi.py feed

To update the favicon urls:

export PYTHONPATH=$PWD:$PWD/src
NO_UPLOAD=1 NO_DOWNLOAD=1 python src/favicons_covers/update_favicon_urls.py

Committing code

We configured the pre-commit hooks to ensure the quality of the code. To setup the pre-commit hooks run the following commands:

pre-commit install
pre-commit run --all-files

organization

This service organizes as follows:

news_aggregator/
├── bin/                # This dir contains the helping shell scripts for Dockerfile.
├── lib/                # This dir contains the utility modules.
├── models/             # This dir contains the dataclasses.
├── sources/            # This dir contains the sources files.
├── src/                # This dir contains all the python script to run the new aggregator.
├── tests/              # This dir contains the tests.

wasm_thumbnail

The wasm_thumbnail.wasm binary comes from https://github.com/brave-intl/wasm-thumbnail.

news-aggregator's People

Contributors

mattmcalister avatar renovate[bot] avatar fallaciousreasoning avatar mschfh avatar aurangzaib048 avatar renovate-bot avatar nvonpentz avatar petemill avatar sangwoo108 avatar dependabot[bot] avatar

Stargazers

Roman avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.