This is a Golang (go) program to read a specific JSON Payload message from a Message Queue System and Store into Database using concurrency

This is a close image of how it works:

• RabbitMQ

• PostgreSQL

• The number of queue consumers could be different from the numbers of storage workers, see config-sample.yaml
• The process (job) of consuming one message from the queue and store into the database is synchronous because every message needs to be acknowledged (confirmed as stored).
• Golang pprof enabled via --profile command line when starting the service
• Prometheus metrics for consumers, storage workers, go statistics, and database
• Grafana dashboard for Prometheus.io metrics
• Dockerfile multi-stage build
• Makefile to facilitate the project builds
• docker-compose file and configuration to testing all elements
• docker images at docker-hub and Github Packages
• CI/CD Github Action pipeline workflow

There are many ways to do it, but always is necessary PostgreSQL and RabbitMQ dependencies, the easy way to see how it works is using containers.

The program and all dependencies and visibility systems at once

docker-compose up --build

docker-compose down -v

After docker-compose start all the services, you have the following links ready to be curious, I prepared a simple grafana dashboard to show you part of the prometheus metrics implemented

• mq-to-db-01 home page
• mq-to-db-02 home page
• Prometheus Dashboard
• Grafana Dashboard
• RabbitMQ Dashboard

# terminal 1, for mq-to-db-01 inside the docker-compose-file
go tool pprof http://127.0.0.1:8080/debug/pprof/goroutine

# terminal 2, for mq-to-db-02 inside the docker-compose-file
go tool pprof http://127.0.0.1:8081/debug/pprof/goroutine

# once you are into tool pprof, execute the command web
(pprof) web

docker-compose logs mq-to-db-01
docker-compose logs mq-to-db-02

First install dependencies

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management

NOTES:

• RabbitMQ web console: http://localhost:15672
• Username: guest
• Password: guest

docker run --rm  --name postgresql -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres

# logs
docker logs postgresql -f

# remember to  stop and remove (--rm in docker run do it for you)
docker stop postgresql

git clone https://github.com/christiangda/mq-to-db.git
cd mq-to-db/
go run -race ./cmd/mq-to-db/main.go --help

# and then
go run -race ./cmd/mq-to-db/main.go --configFile config-sample.yaml

NOTE: the parameter -raceis to check race conditions because we are using Go Concurrency

compiling for your ARCH and OS

# make executable first
make

# check the available options
./build/mq-to-db --help

# execute
./build/mq-to-db --configFile config-sample.yaml

Cross-compiling

make build-dist

# cross-compiling files
ls -l ./dist/

NOTES related to make

• 1. This create a cross-compiling binaries and also Docker Image (linux 64bits)
• 2. Check the Makefile to see The make available targets options
• 3. Remember to start dependencies first

Here I use latest tag, but you can see all releases here

# pull the image first
docker pull christiangda/mq-to-db:latest

# see available option
docker run --rm --name mq-to-db christiangda/mq-to-db:latest - --help

# run with a config file mapped and with profile option
docker run --rm -v <path to config file>:/etc/mq-to-db/config.yaml --name mq-to-db christiangda/mq-to-db:latest - --profile

NOTES:

• Remember to start dependencies first

The application expose different endpoints via http server

• http://localhost:8080/
• http://localhost:8080/metrics
• http://localhost:8080/health
• http://localhost:8080/debug/pprof

Internals

• Configuration
• Messages Type

Externals

• references

This module is released under the GNU General Public License Version 3:

• http://www.gnu.org/licenses/gpl-3.0-standalone.html

• Christian González Di Antonio