pollfax's Introduction

Pollfax

Pollfax

Summary

This is an educational project for the continued learning of:

Data ingestion pipelines
ETL
Go(lang)

Architecture

Golang as the server language paired with sqlx package for interfacing with the database.

Data ingestion

The server starts a cron job to ingest data from api.congress.gov. The main data points to ingest are:

Current congress number. e.g 118th congress, etc.
Get first ~250 bills belonging to the 118th congress.
Persist the bills to a bills table.

Roadmap

The pollfax MVP will accomplish the following

Let users browse the most recently updated bills.
Users can up/downvote bills -> this is persisted in a database
Dashboard TBD

Database Schema Version Control

Create New Migration

run migrate create -ext sqlx -dir db/migrations -seq create_bills_sentiment_table

Commands

`server/main`

Starts the http server.

`ingest/main`

Runs the bills igestion command ad-hoc command.

pollfax's People

Contributors

Watchers

pollfax's Issues

Include new data ingest command in README

Add Bill Id to Pipeline response

Bills now have a auto-generated ID field, it now needs to be exported along with the bills json content file.

Implement bills upsert in pipeline

The existing bills import behavior was to delete all bills from the data base and add the new ones. This worked well with the previous bills schema, where the bill number and type were the composite primary keys. Now that this is no longer the case and the primary key is an auto generated int id column, the import behavior should be to upsert.

Recommend Projects