This repository contains the data project —datasources and endpoints— and script for a UX personalization example using Tinybird.
In this example, we adjust the order of images on a website based on realtime sales or website clicks. This Observable app shows the idea.
Check out this video of the live-coding session walking you through the whole process.
To clone the repository:
git clone [email protected]:tinybirdco/demo-ux-person.git
cd demo-ux-person
To start working with data projects as if they were software projects, let's install the Tinybird CLI in a virtual environment. Check the CLI documentation for other installation options and troubleshooting.
virtualenv -p python3 .e
. .e/bin/activate
pip install tinybird-cli
tb auth --interactive
Choose your region: 1 for us-east, 2 for eu
Go to your workspace, copy a token with admin rights and paste it. A new .tinyb
file will be created.
├── datasources
│ ├── sales.datasource
│ ├── articles.datasource
│ ├── clicks.datasource
│ ├── shoes_and_shirts.datasource
│ └── fixtures
│ └── shoes_and_shirts.csv
├── endpoints
│ ├── api_ranking.pipe
│ └── check.pipe
In the /datasources
folder:
- sales.datasource: where we'll be ingesting sales event data via a python script
- articles.datasource: descriptions of the articles sold
- clicks.datasource: where we will record website activity
- shoes_and_shirts.datasource: containing links to the image files used in the demo
In the /fixtures
folder:
- shoes_and_shirts.csv
In the /endpoints
folder:
- api_ranking: a pipe to rank sales or clicks for the articles in shoes_and_shirts and join them to the info in articles.
- check: a single node pipe to check how ingestion of the sales data is progressing.
Note: Typically, in big projects, we split the .pipe files across two folders: /pipes and /endpoints
/pipes
where we store the pipes ending in a datasource, that is, materialized views/endpoints
for the pipes that end in API endpoints.
Push the data project —datasources, pipes and fixtures— to your workspace
tb push --fixtures
You will see something like this:
** Processing ./datasources/shoes_and_shirts.datasource
** Processing ./datasources/clicks.datasource
** Processing ./datasources/articles.datasource
** Processing ./datasources/sales.datasource
** Processing ./endpoints/api_ranking.pipe
** Processing ./endpoints/check.pipe
** Building dependencies
** Running shoes_and_shirts
** 'shoes_and_shirts' created
** Running articles
** 'articles' created
** Running clicks
** 'clicks' created
** Running sales
** 'sales' created
** Running check
** => Test endpoint at https://api.tinybird.co/v0/pipes/check.json
** 'check' created
** Running api_ranking
** => Test endpoint at https://api.tinybird.co/v0/pipes/api_ranking.json
** 'api_ranking' created
** Pushing fixtures
** Checking ./datasources/shoes_and_shirts.datasource (appending 1.9 kb)
** OK (1 blocks)
** Warning: datasources/fixtures/articles.csv file not found
** Warning: datasources/fixtures/clicks.csv file not found
** Warning: datasources/fixtures/sales.csv file not found
You can check the UI's Data flow:
Add sales data through the HFI endpoint.
Download transactions_train.csv to datasources/fixtures from the Kaggle competition H&M Personalized Fashion Recommendations.
Then run the ingestion script
python3 script/hfi_sales.py
Leave this running while you look at the website to see changing rankings.
Download articles.csv from the Kaggle competition H&M Personalized Fashion Recommendations. Upload this via the UI and append to the articles Data Source.
You now have your Data Sources and pipes that end in API endpoints.
You need a token to consume the api_ranking endpoint and write to the clicks Data Source. You should not expose your admin token, so let's create one with more limited scope.
pip install jq
TOKEN=$(cat .tinyb | jq '.token'| tr -d '"')
HOST=$(cat .tinyb | jq '.host'| tr -d '"')
curl -H "Authorization: Bearer $TOKEN" \
-d "name=ux_demo" \
-d "scope=PIPES:READ:api_ranking" \
-d "scope=DATASOURCES:READ:clicks" \
-d "scope=DATASOURCES:APPEND:clicks" \
$HOST/v0/tokens/
You will receive a response similar to this:
{
"token": <new_token>,
"scopes": [
{
"type": "PIPES:READ",
"resource": "api_ranking",
"filter": ""
},
{
"type": "DATASOURCES:READ",
"resource": "clicks",
"filter": ""
},
{
"type": "DATASOURCES:APPEND",
"resource": "clicks"
}
],
"name": "ux_demo"
}
The demo website is based on Vercel's Next.js commerce. "The all-in-one starter kit for high-performance e-commerce sites. With a few clicks, Next.js developers can clone, deploy and fully customize their own store."
Use node version 16.14.0
nvm use 16
git clone [email protected]:tinybirdco/commerce.git
cd commerce
git stash
git checkout live_coding
npm install
vim .env.local # add your region and ux_demo token
NEXT_PUBLIC_TINYBIRD_API_2=https://api.tinybird.co # or https://api.us-east.tinybird.co
NEXT_PUBLIC_TINYBIRD_TOKEN_2=< token >
npm run dev
The website will run on http://localhost:3000/search
With Auto Refresh selected:
- Sales ranking changes as data is ingested - note the green and red locations. This is the website responding to external data.
- Image positions change in response the the number of clicks. This is the website responding to what is happening on the web.
This project shows just some of the features of Tinybird. If you have any questions, come along and join our community Slack!