This is a Golang (go) program to read from a Message Queue System and Store into Database using concurrency as you can see into the image below
- The number of queue consumers could be different from the numbers of storage workers
- The process (job) of consume one message from queue and store into the database is synchronous because every message needs to be acknowledge (confirm as storage).
- Prometheus metrics for consumers, storage workers, go statistics and database.
- Grafana dashboard for prometheus metrics
- Dockerfile multi-stage build
- Makefile to facilitate the project builds
- docker-compose file and configuration to test all elements
source code
git clone https://github.com/christiangda/mq-to-db.git
cd mq-to-db/
go run -race ./cmd/mq-to-db/main.go --help
# and then
go run -race ./cmd/mq-to-db/main.go --configFile config-sample.yamlNOTE: the parameter -raceis to check race conditions because we are using Go Concurrency
./mq-to-db --helpdocker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-managementNOTE:
- RabbitMQ web console: http://localhost:15672
- Username: guest
- Password: guest
docker run --rm --name postgresql -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres
# logs
docker logs postgresql -f
# remember to stop and remove (--rm in docker run do it for you)
docker stop postgresqlgo run -race ./cmd/mq-to-db/main.go --configFile config-sample.yamlo
make
./mq-to-db --configFile config-sample.yamldocker-compose up --builddocker-compose down -v- mq-to-db-01 home page
- mq-to-db-02 home page
- Prometheus Dashboard
- Grafana Dashboard
- RabbitMQ Dashboard
docker-compose logs mq-to-db-01
docker-compose logs mq-to-db-02Manually
go build \
-o mq-to-db \
-ldflags "-X github.com/christiangda/mq-to-db/internal/version.Version=$(git rev-parse --abbrev-ref HEAD) -X github.com/christiangda/mq-to-db/internal/version.Revision=$(git rev-parse HEAD) -X github.com/christiangda/mq-to-db/internal/version.Branch=$(git rev-parse --abbrev-ref HEAD) -X github.com/christiangda/mq-to-db/internal/version.BuildUser=\"$(git config --get user.name | tr -d '\040\011\012\015\n')\" -X github.com/christiangda/mq-to-db/internal/version.BuildDate=$(date +'%Y-%m-%dT%H:%M:%S')" \
./cmd/mq-to-db/main.goAutomatic
makeNOTES related to make
-
- This create a linux binary (64bits) and also Docker Image (linux 64 bit)
-
- Check the
Makefileto see themake targets
- Check the
- https://www.openmymind.net/The-Little-Go-Book/
- https://golang.org/doc/effective_go.html#generality
- https://golang.org/doc/code.html
- https://golang.org/pkg/database/sql/
- https://golang.org/s/sqldrivers
- https://astaxie.gitbooks.io/build-web-application-with-golang/content/en/05.4.html
- https://gist.github.com/divan/eb11ddc97aab765fb9b093864410fd25
- https://www.alexedwards.net/blog/interfaces-explained
- https://golang.org/doc/effective_go.html#interfaces_and_types
- https://github.com/go-yaml/yaml
- https://www.sohamkamani.com/golang/2018-07-19-golang-omitempty/
- https://ubuntu.com/blog/api-v3-of-the-yaml-package-for-go-is-available
- https://gigi.nullneuron.net/gigilabs/rabbitmq-who-creates-the-queues-and-exchanges/
- https://www.rabbitmq.com/queues.html#optional-arguments
- https://www.rabbitmq.com/dlx.html
- https://www.rabbitmq.com/vhosts.html
- https://www.rabbitmq.com/tutorials/tutorial-one-go.html
- http://www.inanzzz.com/index.php/post/0aeg/creating-a-rabbitmq-producer-example-with-golang
### Metrics
- https://prometheus.io/docs/guides/go-application/
- https://eltonminetto.net/en/post/2020-03-13-golang-prometheus/
- https://scot.coffee/2018/12/monitoring-go-applications-with-prometheus/
- https://dev.to/ahmedash95/understand-golang-channels-and-how-to-monitor-with-grafana-part-2-2-3o42
- https://dlmiddlecote.io/exposing-gos-sql-stats/
- https://github.com/andrestc/demo-go-instrument
### Tests
- https://blog.alexellis.io/inject-build-time-vars-golang/
- https://goenning.net/2017/01/25/adding-custom-data-go-binaries-compile-time/
- https://ewencp.org/blog/golang-iterators/
- https://pkg.go.dev/google.golang.org/api/iterator?tab=doc#example-package-ServerPages
- https://github.com/googleapis/google-cloud-go/wiki/Iterator-Guidelines
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent