mini-services / miniurl Goto Github PK

Publish as a plugin NPM package with programmatic usage, also suitable as Fastify & Express route plugins

@SnirShechter
Change Error to ErrorConstructor in /node_modules/fastify-error/index.d.ts

Allow picking a custom id (e.g http://url.sh/my-nice-link).

This route should be for admins only

Consider using https://draw.io

• Design the UrlInformation structure
  • Include at least: url_visit_count, info_visit_count, updates, creator identifiers (IP, region etc.),
• Add a new url_information table via migrations
• Record the relevant info data in the other routes (visit_count etc.)
• Retrieve URL should also return the visit counts etc. (all the info data without admin-restricted data)
• Create a RetrieveUrl alias using a "+" sign at the end of the short URL
• updatedAt should not be seen by non-admins
• RetrieveUrl should return the whole info data if an admin secret is supplied (should also work via a query string or other browser-friendly methods)

• Make sure eslint & prettier properly run before committing
• Make sure tests properly run before pushing (committing)
• Make sure eslint properly adds end-of-file newlines

Background
The Storage service supports multiple drivers (connectors), which means it can use several databases as a storage target.

Task
Create a new driver which uses MongoDB and implement all of its functionalities similar to the other drivers (use the StorageDriver interface). Try to keep the file/folder structure as similar as possible to the other drivers for consistency.

Allow links to be related to certain users/groups/roles via headers/auth and used only by them

helm Makefile, docker Makefile etc.

Also consider just replacing it with existing GitHub Actions.

• Remove unused dependencies
• Sort dependencies and dev-dependencies
• Make sure CI/CD does not install dev-dependencies on production
• Use ts-node's transpile-only option

Ingress route should also take into account the port of baseRedirectUrl (not just host)

The task is to integrate e2e testing into the project, configure it, structure & design it and divide it into smaller tickets for others to assist.

Steps:

• Find an e2e testing library & methodology (it's possible to use ava for it)
• Create the required folders for the tests files & edit the package.json to run the tests
• Write a first sanity general test for showcasing.
• Create issues for all test sections, among them:
  • All endpoints
  • Database deployment
  • Different configurations (all storage & auth drivers, different databases, existing databases, etc.)
  • Test a Node.js deployment, a Docker deployment & a Kubernetes (helm) deployment

Tip:
See this for how to deploy a Postgres db in CI/CD tests. It might help.

Such as Code climate etc.

Should have a code coverage rating which will be shown as a tag on the README

For the routes documentation.

Should be directed towards either localhost or the demo

As an initial documentation and guide.

To be included:

• Intro
• Features
• Deployment
  • Ways to deploy
  • Environment variables
• API
  • Environment variables
  • Routes
  • Storage
• Contributing

Deployment CI/CD : run tests, build & push docker image, build & push helm, deploy docs & demo.

Test CI/CD: on PR, run tests, GitGuardian, some code quality tool and to be approved only by core team.

TODO: fix the .github/workflows, adjust the Makefile, setup Github Container Registry.

Background

The MiniUrl storage service has multiple drivers with different storage targets. When running, the storage driver uses only one target, as configured by the user in the config module.

Problem

Even though only one driver is used, all the drivers are imported (loaded into memory), which results in unnecessary code loaded in production.

Solution

Make sure the drivers are imported only when needed.

• Go over all the storage drivers, make sure no one imports them besides the main storage/index.ts (their types.ts files can be imported as long as the types.ts does not import the other driver files).
• Replace the imports in storage/index.ts with dynamic imports (`await import('..')) and use it only after checking which driver is picked.

• Upload docker image to dockerhub & Github Container Registry
• Upload helm chart to helm hub (Artifact hub) and Github (our own separate helm repo)
• Add the real docker image (after the CI/CD deploys it) on the helm chart

See the Node.js repo for a good example

• Get the admin secret from env as config
• Generate a secret on helm if not provided (also export it on NOTES.txt)
• Check the secret on any admin endpoints

• A nice-looking homepage with some explanation and a showcase.
• Docs (Vuepress?) - should have a guide section and a technical API section, along with the demo

• Use "last used" instead of "created at"

In a managed Kubernetes environment. Also add a "deploy to demo" Github action.

How do we update the config live AND keep it up-to-date with other instances and the codebase config?

The problem is that changing configuration in the dashboard should be synced with the static config (environment variables etc.) that are injected into the process so that future deployments will stay in sync.

We need to implement a "graceful shutdown" upon receiving 'SIGINT', 'SIGTERM'
Our project also running docker. Unfortunately, sending these signals doesn't automatically translate the signals to a graceful shutdown of the running docker.

The solution to this problem is explained here:
https://snyk.io/blog/10-best-practices-to-containerize-nodejs-web-applications-with-docker/
Also, make sure our docker file follows the best practices mentioned in the article.

Relevant files:
src/index.js

you can also search for the comment "// Graceful shutdown" that will take you to the relevant location in the projects' folder

Allow the user to destroy the app, stop its functionality and cleanup since we might be using shared resources (database, etc.) - delete everything from the database and shutdown (removing a self-deployed database will be performed via helm uninstall)

Services, Routes, Config in particular.

Problem

The database is being accessed many times, which slows down the database and server quite heavily.

Solution

Make a list of all the non-live updates - incVisitCount, incInfoCount etc.

Design an asynchronous batch updating functionality for the frequent non-live updates. e.g incInfoCount should keep an in-memory count inside the driver and flush them all every 5-10 seconds.

NOTE Before the design, check if there's a better way to handle it in the connector libraries (e.g Knex) instead of implementing our own.

EDIT

After checking, there's no such functionality in Knex. The PostgresStorageDriver should implement a batch-update mechanism for incInfoCount and incVisitCount. each of these should save its own cache and flush it every 10 seconds. Their usage in routes should also be non-blocking.