This repository contains the data project —datasources, and endpoints— and data-generator scripts for a log example of using Tinybird.
To clone the repository:
git clone [email protected]:tinybirdco/demo_logs.git
cd demo_logs
To start working with data projects as if they were software projects, let's install the Tinybird CLI in a virtual environment. Check the CLI documentation for other installation options and troubleshooting.
virtualenv -p python3 .e
. .e/bin/activate
pip install tinybird-cli
tb auth --interactive
Choose your region: 1 for us-east, 2 for eu
Go to your workspace, copy a token with admin rights and paste it. A new .tinyb
file will be created.
├── datasources
│ ├── build_log.datasource
│ ├── lambda_log.datasource
│ └── rewrite_log.datasource
├── endpoints
│ ├── count_log.pipe
│ ├── get_filter_values.pipe
│ └── query_logs.pipe
In the /datasources
folder we have three Data Sources:
- build_log: where we'll be sending build log events.
- lambda_log: where we'll be sending lambda log events.
- rewrite_log: where we'll be sending rewrite log events.
And three .pipe files in the /endpoints
folder:
- query_logs: Retrieve timestamp and message data every events filtered by channel, project_id, deployment_id and log_level.
- count_log: Retrieve the number of rows in all the logs.
- get_filter_values_channel: Retrieve distinct values for channel, project, deploymert and log level.
Note: Typically, in big projects, we split the .pipe files across two folders: /pipes and /endpoints
/pipes
where we store the pipes ending in a datasource, that is, materialized views/endpoints
for the pipes that end in API endpoints.
You'll need cli-authentication before executing the next steps cli-authenticate, given that the token will be readed from the file .tinyb.
tb auth
Copy the admin token from https://ui.tinybird.co/tokens and paste it here: <pasted token>
** Auth successful!
** Configuration written to .tinyb file, consider adding it to .gitignore
Here you must paste your Tinybird token.
Push the data project —datasources, pipes and fixtures— to your workspace.
tb push
Your data project is ready for realtime analysis. You can check the UI's Data flow to see how it looks.
Let's add some data through the HFI endpoint.
To do that we have created a python script to generate and send dummy events.
python3 data_generator/demo_log_events.py --datasource build_log --sample 100000 --events 100 --silent
Feel free to play with the parameters. You can check them with python3 data_generator/demo_log_events.py --help
You now have your Data Sources and pipes that end in API endpoints.
The endpoints need a token to be consumed. You should not expose your admin token, so let's create one with more limited scope.
pip install jq
TOKEN=$(cat .tinyb | jq '.token'| tr -d '"')
HOST=$(cat .tinyb | jq '.host'| tr -d '"')
curl -H "Authorization: Bearer $TOKEN" \
-d "name=endpoints_token" \
-d "scope=PIPES:READ:query_logs" \
-d "scope=PIPES:READ:count_log" \
-d "scope=PIPES:READ:get_filter_values" \
$HOST/v0/tokens/
You will receive a response similar to this:
{
"token": "<the_newly_ceated_token>",
"scopes": [
{
"type": "PIPES:READ",
"resource": "query_logs",
"filter": ""
},
{
"type": "PIPES:READ",
"resource": "count_log",
"filter": ""
},
{
"type": "PIPES:READ",
"resource": "get_filter_values",
"filter": ""
}
],
"name": "endpoints_token"
}
If you want to create a token to share just query_logs
with, let's say, the channel with value build, you can do so with the row level security:
curl -H "Authorization: Bearer $TOKEN" \
-d "name=build_logs_token" \
-d "scope=PIPES:READ:query_logs" \
-d "scope=DATASOURCES:READ:query_logs:channel=build" \
$HOST/v0/tokens/
This project shows just some of the features of Tinybird. If you have any questions, come along and join our community Slack!