Eavesdropper is project that used for consuming in message queue and storing in database.

• Consume message from Nats
• Store message in timescaleDB

Eavesdropper configure message queue and database with input data from config file. After that, eavesdropper will parse schema file to understand the structure of message. Then, eavesdropper will consume message from message queue and store it in database.
Eavesdropper will create table in database with migration files and create column in table with name of message queue's field.

Schema file is a file that used for understanding the structure of message. It's contain two parts:

• Subject (required)
• Storage (required)

Subject is a topic that message queue should consume on it.

Storage is a metadata of example message that received from message queue. It's contain list of two parts:

• Table (required)
• Field To DB Column (required)

Table name is a name of table that message should store in it for selected subject.

Field To DB Column is a list of field that message queue should store in database. It's a dictionary that key is name of field in message and value is name of column in database.

- subject: "test-subject-1"
  storage:
    - table: "test_one"
      field_to_db_column:
        data: "data"
        status: "state"

- subject: "test-subject-2"
  storage:
    - table: "test_two"
      field_to_db_column:
        var1: "var_one"
        var2: "status"
    - table: "test_three"
      field_to_db_column:
        var1: "var_one"
        var3: "var_three"

In this example, Eavesdropper has three table in database with name of test_one and test_two. It's store data of message that received from test-subject-1 and test-subject-2. In "test-subject-2" topic, Eavesdropper store data of message in two table. var1 and var2 store in test_two and var1 and var3 store in test_three.

After cloning the project, you can run it locally by using the following commands:

make mod
make build-eavesdropper
docker-compose -f docker-compose.dev.yaml up -d
eavesdropper --config config.yaml

1. Clone this repository
2. Install dependencies
3. Write your own configuration
4. Write your own schema
5. Run the project with docker-compose

graph LR
A[Clone this repository] --> B[Install dependencies]
B --> C[Write your own configuration]
C --> D[Write your own schema]
D --> E[Run the project with docker-compose]

MIT

• @n25a

• GitHub actions
• app package
• DSN
• CLI
• license
• security
• issue templates
• feature templates
• pull request templates
• gitignore
• query builder
• logger
• config file
• dockerized
• makefile
• godoc
• readme
• test
• Nats config
• helm
• changelog
• release
• contributing