mikenikles / your-analytics Goto Github PK

os_name for example records an empty string when it can't be determined based on the user agent string.

Check all columns to make sure the DB columns are NULL instead of empty strings.

Convert all existing values from empty strings to NULL

Make sure the authenticated user has access to the domain they request. The website prevents users from looking at domains they are not authorized for. However, the query-api needs to enforce that too.

When accessing the dashboard landing page, the following errors are displayed in the query-api service:

Error: User did:ethr:xyz tried to access domain dashboard but is not authorized.

Solution
Configure the website service not to send requests for the dashboard.

https://magic.link/

With #55 done, let's update the Visitors chart and include only unique visitors.

Structure:

• A top-level your-analytics database used by the SaaS platform
  • [Collections
    • users: Used for authentication. Also contains an array of sites a user has access to.
  • Child databases (one per site)
    • E.g. example.com, your-analytics.org
      • Collections
        • users
        • settings

On hold until I review the new Cube Cloud at www.cube.dev.

If Cube Cloud does not work out, use the following SQL queries:

Use https://www.tablesgenerator.com/markdown_tables to edit the table

Use the ClickHouse docker container (https://hub.docker.com/r/yandex/clickhouse-server/) for local development.

Set up the events-api and query-api services to work locally too.

It's likely a good idea to ensure authenticated users see the analytics for their own domains only.

Blocked by #7.

Enable operational monitoring.

• admin-api
• db-analytics
• events-api
• query-api

Use this service as an experiment to evaluate feasibility.

Start with a single VM. Expanding to a production-ready Zookeeper setup can be done in a separate step.

Start with a select with the following options

• Today
• Last 7 days
• Last 30 days
• This year (Jan - Today)
• Custom

Develop the API so it requires from/to date parameters. Deal with presets (e.g. last 7 days) on the client and create the from/to dates before calling the API

Dogfooding

Opening the GEO-IP database takes roughly 70 to 80ms. Improve the code to cache the open instance so it does not slow down each individual request.

Current request processing times:

Instead of tracking users with a cookie, we use a more rudimentary approach based on a hash of the following data:

• IP address
• Browser User-Agent

The IP address is not persisted, it is only used to generate the hash and helps to differentiate multiple users behind a NAT gateway.

Follow up to #85.

When querying the screen size, group by mobile (640px), tablet (768px), laptop (1024px), desktop (1280px) as per https://tailwindcss.com/docs/breakpoints#app

When a user logs in who does not yet have a site registered, walk them through a wizard-like onboarding experience. Keep this as lightweight as possible.

• Ask for domain name
• Provide JS snippet with installation instructions

Related to #7.

Given Your Analytics' openness, we want to allow anyone to publish their dashboard publicly.

The first use case is for our own marketing landing page where we want to display live stats for your-analytics.org to demonstrate the features.

For requests to the query-api's /example.com/* endpoints, validate the user is authorized to access stats for the given website.

Do this with a call to the Fauna DB to validate.

We want to serve them from the referrer's domain. Also, DDG displays favicons, let's see if that's a simpler approach.

Check https://en.wikipedia.org/wiki/Favicon for additional educational material

Source: https://excalidraw.com/#json=4861555866861568,hTdmd2q6BxzU6S1C51-n9g

• Events API (#4) - https://events-api.your-analytics.org
• ClickHouse (#2)
• Query API (#5) - https://query-api.your-analytics.org
• Events script (ya.js) (#6) - https://www.your-analytics.org/ya.js
• FaunaDB (#7)
• Website (#21) - https://www.your-analytics.org
• Admin API (#23) - https://admin-api.your-analytics.org

This helps identify situations where for example os_name can't be determined.

#46 ensures NULL is recorded for the os_name in the DB. By logging an error, an alert pops up and UA strings can be manually reviewed to determine how to improve the logic to make sure more UA strings can be parsed properly.

Due to #31 where we now have dynamic pages at www.your-analytics.org/example.com, SSG no longer works, unless we auto-generate URLs for all configured sites. At this point, this on-the-fly SSG is not really feasible with a reasonable effort.

Moreover, adopting a SSR approach will open the doors to hosting the APIs and website on the same domain, hence eliminating the need for CORS. If not on Vercel (to be investigated), then definitely on Firebase where static assets are hosted on Firebase Hosting and dynamic requests routed to Cloud Run instances, i.e. for the APIs and SSR.

Until Vercel supports monorepos (https://twitter.com/mikenikles/status/1281704513351618563), we can only host one service per GitHub repository on Vercel.

Because of that, we need to make sure the generated ya.js file in packages/events-script gets copied to services/website/static/ya.js. This ensures the file is available at https://your-analytics.org/ya.js.

Once Vercel monorepo support is available, we can likely move the ya.js to https://script.your-analytics.org/ya.js and create a permanent redirect from https://your-analytics.org/ya.js to https://script.your-analytics.org/ya.js. Traffic to the old URL can be monitored and once no more traffic hits that file, the redirect can be removed.

1. Do not record
2. Update table to remove it from all recorded events
3. When querying the screen size, group by mobile (640px), tablet (768px), laptop (1024px), desktop (1280px) as per https://tailwindcss.com/docs/breakpoints#app

Tutorial: https://codechips.me/svelte-auth0-integration-in-66-loc/

#68 opened the doors to hosting the APIs and website on the same domain, hence eliminating the need for CORS. If not on Vercel (to be investigated), then definitely on Firebase where static assets are hosted on Firebase Hosting and dynamic requests routed to Cloud Run instances, i.e. for the APIs and SSR.

See https://vercel.com/docs/configuration#routes/dest