Website

Design: figma.com/file/BfYs1TMPaEx8Mr9haEPE6H (incomplete)
Preview: figma.com/proto/BfYs1TMPaEx8Mr9haEPE6H

Stack

• NextJS
• Supabase Auth
• Docusaurus

UI

• Minimal
• Dark Theme
• Use MDX for rich-text content

Auth

• Email/Password (primary)
• GitHub
• GitLab
• Magic Link

The email is required for all the auth methods because it is used to link your account to all auth methods. the email will not be exposed in the api (or anywhere really).

Pages

Home

• Hero
  • Title: Nest
  • Sub: The immutable module registry for Deno
• Description
• Features
• Get started
• Standard modules

User

• Avatar
• Username
• Name
• tabs
  • Modules
  • settings
    • [ pending ]

X

• Search
• Featured modules
• All modules (default sort: updated)

Module

• Name
• Tabs
  • Code
    • Version selector
    • Link for the latest version
    • Nest URL breadcrumbs
    • Files
    • Link to raw (arweave) version
    • Link to type docs
    • Highlighted code
  • Insights
    • Full Name
    • Description
    • Downloads (chart for a week)
    • Dependencies (Maybe "dependents" too?)
    • Licence
    • Quality
  • Settings
    • [ pending ]

Blog

• Minimal MDX blog
• For detailed release notes and announcements

Post

• Title
• Date
• Read time
• Content
• Edit link

Orgs

• Hero with logo, name and description
• Link to their homepage
• Link to their GitHub/GitLab
• List of all modules

Routes

nest.land # home page
nest.land/u # redirect to current user's profile
nest.land/u/:user # user profile & list of modules
nest.land/u/:user/settings # user settings

nest.land/x # modules
nest.land/x/:user # redirect to /u/:user
nest.land/x/:user/:module # redirect to latest version
nest.land/x/:user/:module/:filepath # redirect to latest version
nest.land/x/:user/:module@:version # details for specific version
nest.land/x/:user/:module@:version/:filepath # filepath for specific version

nest.land/- # redirect to /x
nest.land/-/:module # vanity url for details for latest version (redirect)
nest.land/-/:module/:filepath # vanity url for filepath for latest version (redirect)
nest.land/-/:module@:version # vanity url for details for specific version
nest.land/-/:module@:version/:filepath # vanity url for filepath for specific version

nest.land/docs # homepage for docs
nest.land/docs/nest # website docs
nest.land/docs/eggs # eggs docs
nest.land/docs/cli # nest cli docs
nest.land/docs/api # api docs
nest.land/docs/terms # usage policies, etc.

nest.land/blog # latest posts and search
nest.land/blog/:slug # Blog post

orgs.land # info about the service
<org>.orgs.land # organisation profile and modules list
<org>.orgs.land/:module # details for latest version
<org>.orgs.land/:module/:filepath # filepath for latest version
<org>.orgs.land/:module@:version # details for specific version
<org>.orgs.land/:module@:version/:filepath # filepath for specific version

Notes

Why Next.js and not Vue/Nuxt or Svelte/Sapper?

1. Incremental Static Regeneration (IRS)
2. We use Vercel for most of our user-facing infrastructure and Vercel has native support for Next.js
3. We also depend on Vercel's sponsorship and as per the sponsorship conditions, we cannot use SSR (Nuxt, Sapper) and using client-side rendering greatly impacts performance (and obviously SEO).
4. Also, Next.js has "API Routes" features which can be used to create serverless functions to go alongside the website.

PS: Why Next.js is Great for egghead.io - Joel Hooks

Why Supabase Auth?

Because we are already using Supabase for our backend and it handles everything.

Why /x/:user/:module and not /x/:module?

People have to be able to publish a variants of an already existing module with the same name but under their account. Similar to how GitHub does it. you can have "forks" of an already existing project.

There are only so many sensibly short combinations of alpha numeric characters that make sense as a name.

It also helps with "name squatting" and "typo squatting" which have been an issue with npm in the recent past.

ref:

• Recent incident of typo squatting on npm
• strawpole for how to handle squatting on deno.land/x (leading answer: "delete the module even if a version is published (with no considerable content)")

Why is /-/:module a vanity address?

This address will be provided to good quality and trustable modules. So whenever you see a module on that address, you know it's good. To get a vanity URL, a module will have to meet certain criteria that ensures code quality and authenticity. The vanity address will be granted after a manual review.

Criterion (in discussion):

• Properly formated
• properly linted
• Properly documented
• Includes license file
• includes a lock file
• 90%+ test coverage

Authentication

• Primary login would be a email/password pair (paired with HCaptcha).
• Secondary options would be to use GitHub, GitLab or a "Magic Link" to login.

Emails are used to identify you if you use multiple login methods.

But what about GDPR?

GDPR doesn't restrict the collection of user info. The only thing restricted is the sharing of that info with third parties without the user's explicit consent.

Module Analytics

this will be moved to it's own rfc once we settle on a solution

Module import analytics may be the hardest part to implement correctly. We absolutely don't want someone to just do watch curl <Module URL> and fake the numbers.

Checking user agent for Deno won't work cuz you can do watch deno cache -r <Module URL>

So we need the analytics to be bound to the IP address of the user and have a cooldown before that IP can be counted as a real user. I'd suggest one or two days cuz you don't need to refresh your cache that often.

But with that approach, there is a problem of people abusing the CI/CD services as every instance has a separate IP. You could write a cron job to deno cache the URL every minute.

An obvious question arises here:

How does NPM and other registries do it? they got legit download metrics, right?

Nope. NPM themselves claim that the download counts are not accurate and having accurate analytics is hard.

tl;dr: they count every HTTP 200 request as a download.

Same for GitHub, PyPi and almost all kinds of registries. This is because they didn't need it to be accurate.

Now someone would ask,

how does YouTube count views? I tried some bots, they filter it out just fine.

Agreed. So does Google analytics or any analytics service for that matter. But the two products and distribution methods are fundamentally different.
They have the advantage of being on a website that can run javascript code on the browser to figure out if the client is a bot. an HTTP server has no such feature.

This is the whole request that is sent to the URL from Deno when caching or running a remote file:

> GET /-/[email protected]/mod.ts HTTP/1.1
> Accept: */*
> User-Agent: Deno/1.5.4
> Accept-Encoding: gzip, br
> Host: localhost

There can be no captcha, no cookie, no cursor tracking, nothing to differentiate a bot from a user. At this point I think we should look for another metric for deciding the payout.

misc

• Module authors can apply to manually verify a module. Only verified modules will be featured at the top of the "modules" page. "Verification" here means that we check if the module isn't malicious or a "not useful enough" module (looking at you isOdd()).

• Lazy load the lists only on user input i.e. when the client presses "Load more" (ref. design).

Serverless Functions

• Use Next.js API Routes
• Functions:
  • Magic URL
  • Webhooks

Magic URL

• Same URL for web and import
• Handle X-Deno-* headers
• Proxy arweave and cache data on Vercel

Notes

What is this "Magic URL" you mention?

I don't know what else to call it (suggestions?). It is just a function that checks if the client is a browser and sends raw data or webpage accordingly. similar to what deno.land/x does.

Why Next.js API Routes and not Vercel Functions

Because Vercel Functions are "proprietary" and in case we ever need to migrate to some other platform, we would have to rebuild the functions for the platform we migrate to. Using Next.js would mean that the functions remain portable and can be run by any serverless platform (or even bare metal servers).

Then why Vercel at all? Why not run it on our own server?

Because with a great server comes great maintenance requirements 😛. If we choose to use a custom server we would have to:

• Set up and maintain SSL
• Setup auto-deployment for all the branches and create a companion GitHub bot for it
• Configure CDNs to properly cache requests
• Set up load balancers and other (D)DOS protection
• Deal with extra bandwidth costs in case of increased usage
• And all of the above while paying for each of them in time and man-power in addition to money

The whole point of serverless is the is no one dedicated server. It's a cluster of many on-demand servers that run your code and scrap everything after that. Using Vercel save us from all of that headache and more.

Moreover, if you use Next.js, Vercel automatically pushes certain security patches. ref: GHSA-X56P-C8CG-Q435

Copyright Management

All public modules published on Nest assume you want people to be able to use them freely (freedom 0 only). Further license info MUST be included in the readme and the module should be published with a LICENSE file.

The license info will be available through the API. For that the LICENSE file should be in the root directory of the module. Then the license will be analyzed at publish time. If it is an unknown license, the API will return UNKNOWN for the license name.

If a module is reported for copyright infringement through a legit DMCA (or equivalent) request, the module should be immediately unlisted and blocked from being accessed or updated (still accessible through arweave and listed in the API) and a clear notice should be displayed on the module's page and the cli using the X-Deno-Error header. Then if the user wants to dispute the claim, they can request a manual review from the Nest team (not sure what happens here). And if someone wants to access the module through Nest for a legit reason, they will be granted a limited access key on request, which can be used the same way as private repositories do. After the request has been accepted and enacted, the request documents should be pushed to a public dmca repository where people can refer to. This is to ensure that people don't blame us for removing content without reason.

Now the obvious question arises:

You promised to be immutable!?

Yes, by definition all the modules are still immutable as they cannot be edited (mutated) but in extreme cases we need to take action against bad actors to prevent actions against us. And as we cannot remove it from the blockchain, all we can do is block access through our gateway to the artifact on the blockchain.

Nest CLI

• Use the latest standard modules as much as possible
• Use ASCII for stdout as much as possible (looking at you cliffy)
• Avoid using --unstable APIs
• Store all config in a temporary .nest directory
• Exclude all dotfiles by default while publishing
• Enforce semver
• Companion server for gui (see #9)

Commands

nest register # opens the register page in browser

nest login # prompt for login info
nest login --token <token> # add access to an account

nest logout # logout of current account
nest logout --token <token> # remove access to an account

nest # asks to initialize a new module or link an existing one

  # if a module is already initialized and linked, just sync the metadata

  # Nest CLI 1.0.0
  # ? setup "~/mod"? [Y/n]:
  # ? link to an existing module? [Y/n]:
  # ? what's the name of your existing module? (mod):
  # i linked to "user/mod" (created .nest and added it to .gitignore)

  # Nest CLI 1.0.0
  # ? Setup "~/mod"? [Y/n]:
  # ? Link to an existing module? [Y/n]: n
  # ? Module name (mod):
  # ? Version (0.1.0):
  # ? Description:
  # ? Homepage:
  # i Linked to "user/mod" (created .nest and added it to .gitignore)

nest init # initialize a new module in current directory

nest init <name> # initialize a new module in specified directory

nest setup <name> # link current directory to an existing module

nest settings # opens user settings in editor

nest config # opens module config in editor

nest publish --git # publish the latest local git tag

nest publish [ <version> || patch || minor || major ] [ --pre <id> ] # publish the specified version

  # say the current version is [email protected]

  # nest publish                    [email protected]
  # nest publish 1.2.3              [email protected]
  # nest publish patch              [email protected]
  # nest publish minor              [email protected]
  # nest publish major              [email protected]

  # nest publish --pre              [email protected]
  # nest publish 1.2.3 --pre        [email protected]
  # nest publish patch --pre        [email protected]
  # nest publish minor --pre        [email protected]
  # nest publish major --pre        [email protected]

  # nest publish --pre alpha        [email protected]
  # nest publish 1.2.3 --pre beta   [email protected]
  # nest publish patch --pre broke  [email protected]
  # nest publish minor --pre asdf   [email protected]

nest audit # returns an audit report

nest upgrade # upgrade nest cli

# to do any task in the gui, add a --gui or -G flag
# to disable confirmation prompts, add a --yes or -Y flag

Configuration

// .nest/module.json

{
  "$schema": "https://nest.land/-/nest-cli/schema.json",

  "name": "mod",
  "full_name": "My Awesome Module",
  "description": "I swear my module is the best out there",
  "homepage": "https://my.awesome.module",
  "license": "LICENSE",

  "hooks": {
    "presync": "command",
    "postsync": "command",
    "prepack": "command",
    "postpack": "command",
    "prepublish": "command",
    "postpublish": "command",
    "preaudit": "command",
    "postaudit": "command"
  },

  "unlisted": false,
  "private": false
  // ...
}

// .nest/module.json interface
interface Module {
  $schema: string;

  name: string;
  fullName?: string; // default: `${name}`
  description?: string; // default: ""
  homepage?: string; // default: ""
  license?: string; // default: "UNKNOWN"
  unlisted?: boolean; // default: false
  hooks: {
    presync: string; // default: ""
    postsync: string; // default: ""
    prepack: string; // default: ""
    postpack: string; // default: ""
    prepublish: string; // default: ""
    postpublish: string; // default: ""
  };
  // ...
}

// .nest/data.json

// THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.

{
  "meta": {
    "name": "mod",
    "full_name": "My Awesome Module",
    "description": "I swear my module is the best out there",
    "homepage": "https://my.awesome.module",
    "license": "LICENSE",
    "unlisted": false,
    // ...
    "hooks": {
      "presync": "command",
      "postsync": "command",
      "prepack": "command",
      "postpack": "command",
      "prepublish": "command",
      "postpublish": "command",
    },
  },
  // cache the api for offline use
  "api": {
    "versions": ["1.0.0", "0.9.8", "0.9.7", "..."],
    "latestVersion": "1.0.0",
    "lastPublished": 1603171976111,
    "license": "MIT"
    // ...
  },
  "version": "1.0.0", // nest cli version
  "lastSync": 1602831956336,
  "nextAutoSync": 1602832005131
  // ...
}

# .nest/ignore

@extends .gitignore

docs/
tests/
CHANGELOG
CODE_OF_CONDUCT.md
CONTRIBUTING.md
funding.yml

# all dot files are ignored by default so user needs to include them
!.shared_dot_file
!.shared_dot_directory/

<!-- .nest/README.md -->

> Why do I have a directory named ".nest" in my project?

The ".nest" directory is created when you link a directory to a Nest module.

> What does this directory contain?

The "module.json" file contains metadata about the module

The "ignore" file contains a list of files that are ignored or included

The "data.json" file is an auto generated file. This file should **NOT** be directly edited by the user.

> Should I commit the ".nest" directory?

**No**, you should not commit the ".nest" directory to your repository.
Upon creation, it will be automatically added to your ".gitignore" file.

Publishing Programmatically

People can programmatically publish the modules using the nest module (same as cli)

import { publish } from "https://nest.land/-/[email protected]/src/utils/publish.ts"

const options = {
  // ...
})

// auth details are taken from the `DENO_AUTH_TOKENS` env var

publish("module", options);

Notes

• Anything that could lead to unwanted consequences, should be handled by the GUI or the website. These features should never be able to be automated and require human interaction.

• yolk should be merged into cli, though still be usable as a module.

Why are Nest CLI and eggs separate?

Nest CLI is the main utility you would use to create and publish modules. eggs on the other hand, is just a reference module manager that has first-class support for Nest including private modules. Other module managers can use this reference to integrate Nest into their software.

Nest GUI

• PWA (offline first)
• Depends on and Extends the Nest CLI (see #2)
• Dependency analyzer

Notes

When you run nest [command] --gui, the cli would:

1. spin up a server at http://0.0.0.0:3995
2. open https://gui.nest.land in the browser.
3. the gui website hooks into the the cli server.

Why not bundle the GUI with the CLI?

Because it would be a lot larger than the CLI itself. We could also independently push updates to the GUI without any need for manual updates.

There is a disadvantage to this approach too. The GUI cannot support all the versions of the Nest CLI because it would be thae same for everyone. so everyone will have to be on the same major version of the cli otherwise the CLI server would be incompatible with the GUI.

Private modules

Because of the nature of arweave, we cannot "hide" the data. The option left is encrypting the data. furthermore we can push the data to arweave in multiple pieces making it harder to put together the encrypted data.

Publishing

• User sets private: true in config while creation
• User runs nest publish <version>
• Nest CLI uploads the tarball
• Backend encrypts the tarball contents
• Backend uploads the encrypted tarball contents to arweave

Using

• Set DENO_AUTH_TOKENS to <api_key>@nest.land (nest switch would handle this env var)
• User runs deno cache/deno run
• Deno sends a GET request with the active account's access token using Authorization: bearer <api_key> header
• Backend fetches the requested file from arweave and decrypts it
• Backend returns the unencrypted file to Deno
• Deno caches the fetched file.

Notes

• private modules probably will also be accessible using an ?auth=<token> query to access from the browser.
• ?auth=<token> takes precedence over the Authorization header.

Eggs

The go-to dev tool for Deno module devs.

• Avoid using --unstable APIs
• CLI only
• Use /.well-known/deno-import-intellisense.json to get modules
• Support semver constraints (similar to udd)
• Commands
  • add - add module
  • update - update module
  • cache - cache all dependencies
  • run - script runner
  • check - type check all files
  • lint - lint all files
  • fmt - check formatting for all files
  • suggest more...

Configuration

# ./.eggs.yml

# local deno installed to `./.eggs/deno`
deno: 1.12

lock_file: ./mod.lock.json
import_map: ./modules.json

# for deps management
dependencies: ./deps.ts
# or
dependencies:
  - ./deps.ts
  - ./dev-deps.ts

registries:
  - nest.land
  - deno.land

deno:
  compilerOptions:
    strict: true

  lint:
    rules:
      tags: ["recommended"]

env:
  PORT: 3000
# or
env: ./.env.local

# what shell to run commands in
shell: bash

# eggs run <script>
scripts:
  start: deno run --allow-net server.ts
  test: deno test --allow-net server_test.ts
  fmt: deno fmt --ignore="node_modues"

# tools will be installed to `./.eggs/tools/`
tools:
  - name: nest
    src: https://nest.land/-/[email protected]/nest.ts
    location: https://nest.land
    allow:
      read: ./
      net: ['gql.nest.land']
      run: ['deno']
      # ...

hooks:
  pre_commit: deno fmt --check --ignore="node_modues"

Commands

eggs # setup the project environment (cache dependencies, setup git hooks, etc)

eggs cache # caches all (public/private) dependencies

eggs add [...modules | url to file] # add export to the deps file

eggs update [options] [ <target_file> or deps.ts ] # bump all the modules to the specified limit
    Options:
        -r, --registry         name of the registry to lookup

eggs remove [options] [ <target_file> or deps.ts ] [modules] # removes export from the target file

eggs purge [options] [ <target_file> or deps.ts ] # analyse and remove unused deps
    Options:
        -m, --map              use an import map instead

eggs upgrade # update eggs to the latest version

Notes

• hatcher should be merged into eggs, though still be usable as a module.

Why are Nest CLI and eggs separate?

Nest CLI is the main cli you would use to create and publish modules. Eggs is a set of dev tools like script runner, module manager, etc.

Publishing a module

Publishing a module on nest.land can be done in three different ways.
Each of these options may be found in the nest module #2.

Side note: since this module is not just a CLI, a name for this module must be discussed.
(I don't thing cli or nest-cli would be a good name.)

CLI

The command line interface can be used independently or with configuration files.

nest publish [name] [...options]

# Global options
-h, --help                                - Show this help.
-d, --debug                               - Print additional information.
-q, --quiet                               - Suppress diagnostic output.
-o, --output-log                          - Create a log file after command completion.
-G, --gui                                 - Perform the task in the gui.

# Miscellaneous
--dry-run                                 - No changes will actually be made, reports the details of what would have been  
                                          published.
--hands-free                              - Don't display the confirmation message with the staged files.

# Common config
--entry               <value:string>      - The main file of your project.
--full-name           <value:string>      - The name of your project, if long. Not a description.
--description         <value:string>      - A description of your module that will appear on the gallery.
--homepage            <value:url>         - A link to your homepage. Can also be a repository.
--version             <value:version>     - Set the version.
--release-type        <value:release>     - Increment the version by the release type.
--unstable                                - Flag this version as unstable.
--unlisted                                - Hide this module/version on the gallery.

# Files
--files               <values...:string>  - All the files that should be uploaded to nest.land. Supports file globbing.    
--ignore              <values...:string>  - All the files that should be ignored when uploading to nest.land. Supports
                                          file globbing.

# Checks
--check-format        [value:string]      - Automatically format your code before publishing
--check-tests         [value:string]      - Test your code.
--check-installation                      - Simulates a dummy installation and check for missing files in the dependency
                                          tree.
--no-check                                - Don't perform any check.

By default all boolean options are set to false.

Checks are a special case. By default all checks are performed.

To disable this behavior, use --no-check. Individual checks take precedence over option --no-check.
This means that with nest publish --no-check --check-installation, only the installation check will be carried out.

Default values

the default values are not defined as such in the CLI options.
Indeed, they must be determined further by merging all the parameters.

• --entry: Defaults to ./mod.ts, if exists. Automatically added to the list of included files.
• --unstable: Default value is determined by Semantic Versioning rules.
• --check-format: Defaults to deno fmt --check && deno lint --unstable. Disabled if --no-check is used.
• --check-tests: Defaults to deno test -A --unstable. Disabled if --no-check is used.

Config Files

Configuration files can be used so you don't have to rewrite the same configuration every time you publish.

egg

The egg file exists in two formats, json and yaml.

To discuss:

• Should the file be located at the root of the project or in the .nest folder?

The command line options have priority over the fields in the egg file.
The file will be rewritten if the parameters have changed at runtime.
For example if a new version is determined or if command line options have been provided that differ from the fields in the configuration file.

interface Config {
  $schema: string; // Validates the structure of JSON data

  name: string;
  fullName: string;
  entry: string;
  description: string;
  homepage: string;

  version: string;
  releaseType: semver.ReleaseType;

  unstable: boolean;
  unlisted: boolean;

  files: string[];
  ignore: string[];

  checkFormat: boolean | string;
  checkTests: boolean | string;
  checkInstallation: boolean;
  noCheck: boolean;
}

The options are the same as on the command line.

The egg file is automatically added to the list of ignored files.

.eggignore

The .eggignore file is an extension of the egg file. It takes precedence over ignore.

To discuss:

• Should the file be located at the root of the project or in the .nest folder?
• .eggignore, .nestignore or nestignore ?

Files and directories to be ignored are specified
one by line, as a glob expression.
They follows the same rules as the .gitignore pattern format.

In addition to this syntax, the keyword @extends allows to import other ignore files.

When extending the .gitignore file, .git*/** is automatically added to the list of ignored file.

The .eggignore file is automatically added to the list of ignored files.

Programmatically

Publishing a module is possible by calling the publish function provided by nest.

This is the same function used by the CLI after parsing the options.
This means that egg and .eggignore files are taken into account.

import { publish } from "https://x.nest.land/[NEST]@[VERSION]/commands/publish.ts"

publish(name, options);

/********************************/

async function publish(name: string, {
  help: boolean,
  debug: boolean,
  quiet: boolean,
  outputLog: boolean,

  dryRun: boolean,
  handsFree: boolean,

  fullName: string,
  entry: string,
  description: string,
  homepage: string,

  version: string,
  releaseType: semver.ReleaseType,

  unstable: boolean,
  unlisted: boolean,

  files: string[],
  ignore: string[],

  checkFormat: boolean | string,
  checkTests: boolean | string,
  checkInstallation: boolean,
  noCheck: boolean,
})

GUI

The graphical interface will allow authors to ensure that all the parameters of their module are correct and modify fields on the fly.
Uses the publish function under the hood.

Once all files have been read and prioritized, some fields are required.
If they are not present, the publication will fail.

• name;
• version and/or release-type;
• files and/or ignore.

The only data that will not be undefined for sure when publishing are:

• The name of the module;
• The version of the module;
• The filenames and their content;
• The stability of the module.

API

• api.nest.land REST API
• gql.nest.land GraphQL API

Rest API

• users

  api.nest.land/users # list of all users
  api.nest.land/users/:user # user profile
  api.nest.land/users/:user/modules # list of modules by :user

• modules

  api.nest.land/modules # list of all modules
  api.nest.land/modules/:user/:module # module details
  api.nest.land/modules/:user/:module/versions # list all versions
  api.nest.land/modules/:user/:module/:version # version details
  api.nest.land/modules/:user/:module/:version/files # list of files

• import intellisense

  api.nest.land/imports/- # for vanity modules
  api.nest.land/imports/x # for scoped modules

• publish webhooks

  api.nest.land/webhook/github/:user/:module # publish from github
  api.nest.land/webhook/gitlab/:user/:module # publish from gitlab

Database

schema: schema.prisma

Notes

Why would you need a REST API when you have a GraphQL API?

REST is a bit more accessible to beginners than GraphQL, and having more options is not a bad thing.

We can also deploy serverless functions for badges and stuff in the future on api.nest.land.