Giter Club home page Giter Club logo

matters-server's Introduction

Matters Server

Deployment Status Test Status Commitizen friendly

Development

Local

  • Install dependencies: npm install

  • Start Postgres, Redis, stripe-mock, and IPFS daemon

  • Setup Environments: cp .env.example .env

  • Run all migrations: npm run db:migrate

  • Populate all seeds data if needed: npm run db:seed

  • Run npm run start:dev, then go to http://localhost:4000/playground to GraphQL Playground.

  • Run test cases: npm run test

  • Run db rollup process; use the same psql command line parameters if modified in .env; (hint -d database and -U username, and -w to read saved password of psqlrc)

    (cd ./db; PSQL='psql -h localhost ... -w' bash -xe bin/refresh-lasts.sh )
    

Docker

  • cp .env.example .env
  • docker-compose -f docker/docker-compose.yml build
  • docker-compose -f docker/docker-compose.yml run app npm run db:rollback
  • docker-compose -f docker/docker-compose.yml run app npm run db:migrate
  • docker-compose -f docker/docker-compose.yml run app npm run db:seed
  • docker-compose -f docker/docker-compose.yml up
  • Run test cases: docker-compose -f docker/docker-compose.yml run app npm run test
  • Init search indices: docker-compose -f docker/docker-compose.yml run app npm run search:init

DB migrations and seeds

  • Create a new migration: npm run db:migration:make <migration-name>
  • Create a new seed file: npm run db:seed:make <seeds-name>, seed files are run sequential so please pre-fix with order
  • Rollback a migration: npm run db:rollback

Email Template

We use MJML to develop our SendGrid email template.

Please refer to the repo matters-email for details.

Test Mode

To make the login flow testing easier, the login-related mutations have hardcoded input values with respective behaviors in the non-production environment.

see test_mode.md for detail

NOTE

AWS resources that we need to put in the same VPC

  • Elastic Beanstalk
  • RDS PostgreSQL
  • ElastiCache Redis instances
    • Pub/Sub
    • Cache
    • Queue
  • IPFS cluster EC2 instances

matters-server's People

Contributors

byhow avatar carolusian avatar denkeni avatar dependabot-preview[bot] avatar dependabot[bot] avatar devformatters avatar gary02 avatar guoliu avatar jasmine-liang avatar jeremyok avatar pitb2022 avatar robertu7 avatar tx0c avatar williamchong avatar zeckli avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

matters-server's Issues

Could you allow developers to register a oauth2 applicaion for matters?

Describe the solution you'd like
A clear and concise description of what you want to happen.

Hi, I want to make an application for users of matters to manage their article at matters.news, could I get a oauth2 application for user to authorize their article scope for me? For now, my needs is only to read the authorized user's article.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

For now, may be I can get the articles of user through scraping the graphql api, but I don't think it's a good use case.

Additional context
Add any other context or screenshots about the feature request here.

thanks!

Notice logic inconsistency

It seems there are two notice types have logic inconsistency. Below are possible problems based on my understanding:

๐Ÿ‘๐Ÿป comment_new_upvote

Before queue trigger inserts notice data into DB, it generates different data objects according to different notice types. Like this:

-------------------------------------------------
File: src/connectors/notificationService/index.ts
-------------------------------------------------

private getNoticeParams = async (params): Promise<any> => {
  switch (params.event) {
    case 'user_new_follower':
    case 'comment_new_upvote':
      return {
        type: params.event,
        recipientId: params.recipientId,
        actorId: params.actorId
      }
    ...
 }

In above, there is no entities in comment_new_upvote's returned object. Howerver, entities are required in the query API and they will be displayed in notice digest:

-------------------------------------------------
File: src/common/utils/notice.ts
-------------------------------------------------

const actorsRequired = {
  ...
}

const entitiesRequired = {
  comment_new_upvote: true
  ...
}

export const filterMissingFieldNoticeEdges = (params): any => {
  return edges.filter(({ node: notice }) => {
    const noticeType = notice.type
    ...
    // check entities
    if (entitiesRequired[noticeType] && _.isEmpty(notice.entities)) {
      return false
    }
    ...
    return true
  })
}

So, there is logic inconsitency in it. Except that, it causes another problem. Since comment_new_upvote are always filtered out of query result and state still stay unread, our notice service will try to bundle all comment_new_upvote as one even upvote targets are different. ๐Ÿค”

๐Ÿ… comment_pinned

Pretty similar to previous case:

private getNoticeParams = async (params): Promise<any> => {
  switch (params.event) {
    case 'comment_pinned':
      return {
        type: params.event,
        recipientId: params.recipientId,
        entities: params.entities
      }
    ...
}

Our filter will kick comment_pinned out of query result becasuse of lacking actors:

const actorsRequired = {
  comment_pinned: true
  ...
}

export const filterMissingFieldNoticeEdges = (params): any => {
  return edges.filter(({ node: notice }) => {
    const noticeType = notice.type
    // check actors
    if (actorsRequired[noticeType] && _.isEmpty(notice.actors)) {
      return false
    }
    ...
    return true
  })
}

And I found we tried to get actor from comment data so that we don't have to record actor in DB:

-------------------------------------------------
File: src/queries/notice/index.ts
-------------------------------------------------

  ...
  CommentPinnedNotice: {
    id: ({ uuid }) => uuid,
    actor: ({ entities }, _: any, { dataSources: { userService } }) => {
      const target = entities.target
      return userService.dataloader.load(target.authorId)
    },
    target: ({ entities }) => entities.target
  },
  ...

But the target here is the comment, so target_authorId is the comment creator instead of article author. In notice digests, they look like comment authors pinned their own comments. ๐Ÿ˜‚

New registration link provided in email

To simplify registration process, we will provide an email attached a link in order to confirm and activate THE following steps.

We will need:

  • New email template
  • Link generator
  • Confirm mechanism
  • Retire old verification

tag cover fallback runs incorrectly

Describe the bug
Some covers of tag are still vacant although covers of articles in the tag can be found.

Screenshots

image

Desktop (please complete the following information):

  • MacOS
  • Chrome

Automatically follow tags

An user should automatically follow a tag when s/he:

  • submit an article to the tag
  • become tag owner
  • become tag collaborator

[Server] Onboarding Tasks: authors recommendation

thematters/matters-web#1556

In one of new user tasks, there will be three new feed types most trendy, most appreciated and most active for new registered users to follow.

Logics:

  • Most trendy: List creators who has over 60 followers in 90 days.
  • Most appreciated: List top 60 creators who got HKD donations, but exclude creators in most trendy.
  • Most active: List top 60 creators who has more comments in 90 days, but exclude creators who receive downvote rate > 10%. (downvote/(upvote + downvote))

Personal search history records @ user, connect articles and add article to tag

Describe the bug
Search API should only record searches in search bar, but is currently recording every search including @ user, connect articles and add article to tag.

To Reproduce
Steps to reproduce the behavior:

  1. @ user, connect articles or add article to tag
  2. see the recent query when clicking on search bar

Expected behavior
Queries from @ user, connect articles or add article to tag should not be included in search history

Additional context
Search API input should include record: boolean to notate if the current search should be recorded, and frontend should use it accordingly.

Re-designing home page feeds

Is your feature request related to a problem? Please describe.

Currently two home article feeds, ใ€Ž็†ฑ้–€ใ€andใ€Ž็†ฑ่ญฐใ€, have vague design goals and significant overlap with each other. ใ€Ž็†ฑ้–€ใ€should be a collective choice for articles that are worth reading at a given moment, and ใ€Ž็†ฑ่ญฐใ€should be a collective choice for articles that are worth discussing at a given moment.

Forใ€Ž็†ฑ้–€ใ€, in the past we do not have a direct measurement of reading time. Now that we are recording read time, we can move away from appreciations/Likes, which does not have any cost for the actor and signifies many different things such as friendship, support or greetings, and move towards read time and donation, the former being a direct measurement of reading and the later with cost and therefore resilient to spams. We can also start recording impressions, which are the times an article card appear to our user, to calculate the efficiency of ready given a number of impressions.

ใ€Ž็†ฑ่ญฐใ€still requires more discussions. A general direction might be more focus on number of participates, number of votes on comments, or different ways of measuring weights of commenters.

User blacklist for preventing re-registration

Discussion context: https://mattersnews.slack.com/archives/CF78WGNNM/p1582930152004800

We need a mechanism to reduce re-registration of banned users. We still need to confirm with lawyers for legal regulations, but here's some initial ideas.

Record canvas fingerprint after ban

When a banned user logon, backend record ip, canvas fingerprint and email in blacklist table. Fingerprint can be passed to backend either through a mutation or through a particular header.

In this way, we don't have to track the fingerprint of every user, but only for the banned user.

Inheritance of ban

When requesting verification code during registration, frontend also send canvas fingerprint to backend. If backend finds a match in fingerprint, ip or email, it adds the other two to blacklist and decline sending verification code.

In this way, a banned user is likely to inherit his/her banned state across agents. For example, when a banned user use the same browser or ip to register with a new email, verification code is not send and new email is also banned; when he/she changed browser and ip to try again, the email is matched and the new ip and canvas fingerprint is also banned. This makes it harder to crack.

Blacklist table

  • id: increments
  • uuid: uuid, used to mark the same user across different types
  • type: enum(['ip', 'canvas', 'email'])
  • value: string
  • created_at: timestamp

Fix article to draft relationship

Current article doesn't record draft id when publishing. We need to backfill these ids before implementing version control of article.

Medium migration flow

Since we are reshaping our feature from calling Medium API to uploading Medium source files, couple things need to discuss.

๐Ÿ“ฆ Files

The packed files downloaded from Medium is quite big because it has lots of irrelevant files. Below are unpacked files:

โ”œโ”€โ”€ blocks
โ”‚ย ย  โ””โ”€โ”€ blocked-users-0001.html
โ”‚ย ย 
โ”œโ”€โ”€ bookmarks
โ”‚ย ย  โ””โ”€โ”€ bookmarks-0001.html
โ”‚ย ย 
โ”œโ”€โ”€ claps
โ”‚ย ย  โ””โ”€โ”€ claps-0001.html
โ”‚ย ย 
โ”œโ”€โ”€ highlights
โ”‚ย ย  โ””โ”€โ”€ highlights-0001.html
โ”‚ย ย 
โ”œโ”€โ”€ interests
โ”‚ย ย  โ”œโ”€โ”€ publications.html
โ”‚ย ย  โ”œโ”€โ”€ tags.html
โ”‚ย ย  โ”œโ”€โ”€ topics.html
โ”‚ย ย  โ””โ”€โ”€ writers.html
โ”‚ย ย 
โ”œโ”€โ”€ ips
โ”‚ย ย  โ””โ”€โ”€ ips-0001.html
โ”‚ย ย 
โ”œโ”€โ”€ posts
โ”‚ย ย  โ”œโ”€โ”€ 2018-04-02_-----------Arendt----51bc52c880f3.html
โ”‚ย ย  โ”œโ”€โ”€ 2018-04-12_-----------------3a905851316e.html
โ”‚ย ย  โ”œโ”€โ”€ 2018-04-12_------------1253c94fa6ac.html
โ”‚ย ย  โ”œโ”€โ”€ 2018-05-31_Matters--------------ae9f9aa98249.html
โ”‚ย ย  โ”œโ”€โ”€ 2018-11-10_-Matters--------------70c1ab6d47e2.html
โ”‚ย ย  โ”œโ”€โ”€ 2019-03-22_Matters-------------------6dc72e6753f9.html
โ”‚ย ย  โ”œโ”€โ”€ 2019-03-27_---------------------c4336ab683df.html
โ”‚ย ย  โ”œโ”€โ”€ 2019-04-02_------------------12bdf59fe4a9.html
โ”‚ย ย  โ”œโ”€โ”€ 2019-04-24_----------------------2fd7c25b0934.html
โ”‚ย ย  โ””โ”€โ”€ draft_nn-813be4d2bd80.html
โ”‚ย ย 
โ”œโ”€โ”€ profile
โ”‚ย ย  โ”œโ”€โ”€ memberships.html
โ”‚ย ย  โ”œโ”€โ”€ profile.html
โ”‚ย ย  โ””โ”€โ”€ publications.html
โ”‚ย ย 
โ”œโ”€โ”€ pubs-following
โ”‚ย ย  โ””โ”€โ”€ pubs-following-0001.html
โ”‚ย ย 
โ”œโ”€โ”€ sessions
โ”‚ย ย  โ””โ”€โ”€ sessions-0001.html
โ”‚ย ย 
โ”œโ”€โ”€ topics-following
โ”‚ย ย  โ””โ”€โ”€ topics-following-0001.html
โ”‚ย ย 
โ””โ”€โ”€ users-following
    โ”œโ”€โ”€ users-following-0001.html
    โ””โ”€โ”€ users-following-0002.html

As you see, there are some user information and settings. All we need is posts folder. Comments are considered as posts from Medium view, so comments are also packed in posts folder. Do we want user to upload one package? Or just upload real posts by picking ?

๐Ÿง‘๐Ÿปโ€๐Ÿ”ฌ Process flow

Possible process flows are here:

image

Based on current design, uploading packed files would be the easiest way for user but not for us. Also, some comments' contents will be listed in drafts. In opposite, uploading multi-files would be the simplest way for us and we could get the right uploaded files (real posts), but users might need to drop files couple times.

FYI, editor has an upload button on the right sidebar.

Love to hear you guys ideas ๐Ÿง‘๐Ÿปโ€๐Ÿ’ป

Performance of pagination using offset

Problem

Our pagination is based on offset and limit, but the performance of it will get worse when records accumulated. Because:

  • offset always shifts from scratch. (page 1 -> page 2 -> page 3)
  • Based on point one, the cost and execution time will increase drastically in tables that we insert frequently.

Take follower as an example:

findFollowers = async (...) =>
  this.knex
    .select()
    .from('action_user')
    .where({ targetId, action: USER_ACTION.follow })
    .orderBy('id', 'desc')
    .offset(offset)
    .limit(limit)

In this example, users can follow multiple users so it accumulates records quickly. Now we have 197,989 records, and CPU usage of the query usually stays on Top 5. ๐Ÿคฆ๐Ÿปโ€โ™‚๏ธ

screen

Possible fix

In order to fix this, we need to change how we do pagination. An easier solution is to use id as cursor:

findFollowers = async (...) =>
  this.knex
    .select()
    .from('action_user')
    .where({ targetId, action: USER_ACTION.follow })
    .andWhere('id', '<', id)
    .orderBy('id', 'desc')
    .limit(limit)

In this query, the order of id exactly matches the order of followers. So, we can take advantages of comparing id:

  • No more shift from scratch.
  • Comparing id (a number) can have more stable and consistent performance than using offset.
  • If we try to retrieve pretty old record, the database will utilize id (pk or index) to search reversely.

I've used PG's explain to analyze, and the cost looks ok. Once I commit the new query, we can observe the result. At the same time, we might need to think rest of queries that cannot apply this fix case.

@robertu7 @guoliu any idea?

Selected tags should appear at the front of the list

Is your feature request related to a problem? Please describe.
It's better to highlight the selected tags

Describe the solution you'd like
Selected Tags (with green background) should appear at the front of the list in article detail page

image

Re-design GraphQL schema to separate public/system and private/viewer data

Our GraphQL schema has grown very large, maintenance and optimization are needed. With a better partition between public data and private data, we should be able to fit the need of JAMstack pattern on frontend, simplify private cache pattern on backend, make it easier for developers to start using after we open source, better support future iterations on follow page, and much more.

We can separate public and private data into different types. For example, each Node type can have a viewer field, which holds the corresponding type for private/viewer data. Therefore Article type can be:

Article {
  ...
  viewer: ViewerArticle
}

ViewerArticle {
  isBookmarked
  appreciationCount
  appreciationLeft
  ...
}

User type can be:

User {
  ...
  viewer: ViewerUser
}

ViewerUser {
  isFollower
  isFollowee
  isBlocked
  ...
}

Comment type can be:

Comment {
  ...
  viewer: ViewerComment
}

ViewerComment {
  isCollapsed
  ...
}

In this way, we do not need to keep a special keyword for CSR on frontend as proposed in thematters/matters-web#1051 (comment), but need to assemble Viewer${NodeType} fragments with client side query. This could be a cleaner logic, and improves cache hit rate and share rate.

We can also separate private and public data under different root fields, so that we can apply different auth and cache patterns. For example, we can group all private data under viewer and all public data under system, and the schema could look like:

query: {
  system: { 
    node(input: {
      id
    }): Node
    article(input: {
      mediaHash
      dataHash
    }): Article
    user(input: {
      userName
    }): User
    feeds: {
      icymi: ArticleConnection
      hottest: ArticleConnection
      ...
    }
  },

  viewer: {  
    // followers feed
    feeds {
      // follower publish feed
      articles: ArticleConnection
      // follower comment feed, group by article and user
      discussions: { 
        ...
        edges { 
          // comment on which article
          article: Article
          // the grouped comments
          comments: CommentConnections
          cursor: String
        }
      }
      // follower donation feed, group by article
      donations: {     
        ...
        edges { 
          article: Article
          users: UserConnection
          cursor: String
        }
      }
    }
    setting {   
      language
      ...
    }
    status {
      ...
    }
  }
}

Visitors and SSR would only need to query system root field. If we want to be safe and strict, we can even enforce on the backend that queries of type Viewer${NodeType} and viewer root field only return when fetched from client.

There should also be other optimization patterns we can apply.

Evaluate Knex.js to Prisma migration

Is your feature request related to a problem? Please describe.

We are currently using Knex.js without an ORM. This has provided flexibility in the beginning, but as our database schema grows more and more complex, we need a cleaner model for database objects.

Describe the solution you'd like

We can migrate to Prisma, which has matured in the past few years. It automates most of the mapping between GraphQL schema and database schema, which will make our codebase much cleaner.

However, Prisma Migrate for database migration is still experimental. We can take some risk and try it on, or keep knex-migrate for migration and then do introspection separately (see example).

Since Knex is a query builder and Prisma is similar to an ORM, there need to be a change in design pattern. We need to decide what migration method we want to use, and what would be the easiest path switching to Prisma.

Additional context
Related to #897

Store fingerprint

This issue involves:

  • Create data structure for storing info
  • Redesign or merge current table
  • Revise logic of store limits

Difficulties of cache tuning

After checking queries listed in card, there are some difficulties on tuning cache. For instance:

query {
   user {
     articles {
       title
       isSubscribed
     }
   }
}

Most fields in the query could be public, but personalized data isSubscribed makes entire response private. Besides that, our digest components like ArticleDigest, UserDigest and Comment more or less include personalized data. More examples:

query {
  comment {
    author {
      name
      isBlocking
    }
  }
}

and

query {
  user {
    name
    isFollower
  }
}

As you see, it's quite hard to separate data like Author from Article, Comment and Response (Article | Comment) in queries we have now. And it also a little bit conflicts GQL philosophy if we try to separate those fields. ๐Ÿค”

Probably, increase TTL might be one temporarily solution for now ?

@robertu7 @guoliu any idea?

Version control of articles on backend

We will implement article edit functionality on the frontend. On the backend, we need to implement a minimal version control. The idea is using draft table as versions, and article table as pointer pointing to the newest version.

Reject certain high frequency mutation

We need to reject certain operations if the frequency is higher than given threshold. This might not be useful for preventing spam in general, but is still good practice to mitigate attacks.

Operations currently under discussion include:

  • comment, 2 per minute
  • appreciate, 5 (transaction) per minute (need confirmation)
  • publish, 10 per 2 hours

  • global rate limit for mutation
  • use redis to record operation log

Separate webpage CDN cache for different user groups

We have been delivering one single version of SSR webpages on CDN, and certain user group refetch data for A/B test. This has slow response time. We can instead write user group in cookie, differentiate user group by cookie in CDN, and implement A/B test behind GraphQL resolvers.

Reduce google translation API calls

Currently we are caching translation result for 10 days. This is wasting API calls, since we only need to call detect language once for each article, and call translation once for each language of each article.

We also have the issue of high volume concurrency. If we update cache key schema or cache service, we will have a high volume of translation API calls, which result in many (403) User Rate Limit Exceeded errors. Although we haven't surpass GCP quotas, it looks like GCP have some hard limit in API call frequency peak. Since we also query Article.language to determine whether we should show translation button or not, it also slows down SSR when hitting rate limit.

A better long term strategy is that we store language in database, Translation.title and Translation.content in database or even s3.

We can probably progress in several steps according to our need:

  • add a timeout for Article.language, so that query returns null and do not block SSR, but still fires API call and fill cache storage
  • store language and translation to database or s3 after API call, and serve from database or s3 if possible
  • after publishing article, store language and translation to database, and randomly store language and translation for older articles

Prevent concurrent transactions

Here are flows for preventing concurrent transactions to Postgres:

queue

Couple things to be clarified since flows are in sync style:

  • Pending transaction are always returned, which means client side need to revise feedback message. Client get notices when transaction state has been updated.
  • Transaction history must includes pending transaction.

Any supplementation to those flows? @robertu7 @guoliu

cc @gyuetong

Modeling subthreads

In the new design of comment section, 2nd level comments are bundled together based on reply relationships.

image


We can view the newly added structure as sections within subthreads, or as subthreads of subthreads that are flattened into the same level (level 3 flattened into level 2).

subthead 001

The structure on the left has subthreads with section, so subthreads are lists of comment list. It is more similar with the UI layout.

The structure on the right is fractal, with subthreads having their subthreads. It is more similar with the actually relationship of comments, which is a directed acyclic graph.

With the mental model on the right, we won't need to update our API, since Comment type has comments field. But we need to either resolve in API all remaining comments at level 3, or recursively call child comments of comments on frontend util no more comments are returned.

They both have pros and cons, but I think the right one is more concise and closer to the actual relationship of data.

Email Workflow

Current Workflow

excalidraw-2020216114836 (1)

TODO

  • Build & deploy templates with CI/CD
  • Separate to different templates for different languages
  • Separate different environments

SendGrid?

Pros

  • More developer-friendly than MailChimp: Template Engine Support & Easy to Use API
  • Reduce Server Load: Rendering & Network Costs
  • Sender Reputation: To avoid emails being in spam folder
  • Statistics

Cons

Metabase security for public dashboard

We need to display certain data visualizations for current and potential users, and need to evaluate if it is secure to embed metabase public dashboard as iframe. Another option is to display visualization as static image, which is secure but will involve manual updates.

After first glance of current open security issues on metabase, it seems that none is related to public dashboard, but further evaluation is still needed.

Some measures we can take:

  • Use other domains to proxy to dashboard url (for example, data.matters.news/about and data.matters.news/community ?)
  • NGINX stop public dashboard other than the ones in whitelist
  • Setup CDN with very long TTL for dashboard
  • See if cloudfront can keep alive when origin dies

OAuth Scope: display texts

Is your feature request related to a problem? Please describe.
Display (human-readable) texts for OAuth scope are hard coding at client now. It's hard to maintain.

Describe the solution you'd like
Move to matters-server and use field-level directive or a single file to declare related texts.

Clean Code: DataServices

DataServices such as userService.ts and articleService.ts have a lot of code in a single file, and as our business logic grows, they become difficult to maintain.

Optimization: Images Processing

Background

We have lots of images uploaded by users: avatars, profile covers, article embedded images, and etc. Based on Lighthouse Report, images have big negative impact for performance score:

image

In general, there are three sides to be optimzed:

Source

  • Compressing
  • Resizing: different sizes fit different needs
  • Formats: WebP, JPEG 2000, etc.
  • Progressive

Application

Proxy

Solutions

Currently, we will do simple image processing (compressing & resizing) in connectors/aws, while image uploading. But there are cons:

  1. It's synchronous, uploading is blocked by processing;
  2. Growing complexity, we need add more and more codes for above needs;

Lambda To Rescue!

With AWS Lambda, processing image will be asynchronously and separately. There are two ways to implement:

1) Lazy processing

Use Serverless Image Handler, process image on client requests, and cached by CDN.

Pro & Cons:

  • Lower S3 cost, only store raw images;
  • Higher CDN cost by low hit rate;
  • Longer response time, process on-demand;

2) Post-processing

  1. The client calls the API to upload image, server forward it to AWS S3 directly, now raw image is accessible.
  2. AWS Lambda post-process the raw image from AWS S3, asynchronously, now optimized images are accessible.

Steps

  1. Formating: WebP
  2. Resizing
    • avatar: raw, 144w
    • embed raw, 1080w, 540w, 360w, 144w
    • profileCover: raw, 1080w, 540w
  3. Compressing

Results

# AWS S3 
/matters-server-stage
โ”œโ”€โ”€ 1080w
โ”‚ย ย  โ”œโ”€โ”€ uuid.jpeg
โ”‚ย ย  โ””โ”€โ”€ uuid.webp
โ”œโ”€โ”€ 540w
โ”‚ย ย  โ”œโ”€โ”€ uuid.jpeg
โ”‚ย ย  โ””โ”€โ”€ uuid.webp
โ”œโ”€โ”€ 360w
โ”‚ย ย  โ”œโ”€โ”€ uuid.jpeg
โ”‚ย ย  โ””โ”€โ”€ uuid.webp
โ”œโ”€โ”€ 144w
โ”‚ย ย  โ”œโ”€โ”€ uuid.jpeg
โ”‚ย ย  โ””โ”€โ”€ uuid.webp
โ””โ”€โ”€ uuid.jpeg

Usage

<!-- <ArticleDetail.Content> -->
<figure>
  <picture>
    <source type="image/webp" media="(min-width: 768px)" srcset="https://xxx.cloudfront.net/embed/1080w/uuid.webp" alt="...">
    <source type="image/webp" srcset="https://xxx.cloudfront.net/embed/540w/uuid.webp" alt="...">
    <source media="(min-width: 768px)" srcset="https://xxx.cloudfront.net/embed/1080w/uuid.jpeg" alt="...">
    <img src="https://xxx.cloudfront.net/embed/540w/uuid.jpeg" alt="...">
  </picture>
  <figcaption>...</figcaption>
</figure>

<!-- <ArticleDigest.Cover>, View Mode = default -->
<picture>
  <source type="image/webp" media="(min-width: 768px)" srcset="https://xxx.cloudfront.net/embed/1080w/uuid.webp" alt="...">
  <source type="image/webp" srcset="https://xxx.cloudfront.net/embed/540w/uuid.webp" alt="...">
  <source media="(min-width: 768px)" srcset="https://xxx.cloudfront.net/embed/1080w/uuid.jpeg" alt="...">
  <img src="https://xxx.cloudfront.net/embed/540w/uuid.jpeg" alt="...">
</picture>

<!-- <ArticleDigest.Cover>, View Mode = compact -->
<picture>
  <source type="image/webp" media="(min-width: 768px)" srcset="https://xxx.cloudfront.net/embed/360w/uuid.webp" alt="...">
  <source type="image/webp" srcset="https://xxx.cloudfront.net/embed/144w/uuid.webp" alt="...">
  <source media="(min-width: 768px)" srcset="https://xxx.cloudfront.net/embed/360w/uuid.jpeg" alt="...">
  <img src="https://xxx.cloudfront.net/embed/144w/uuid.jpeg" alt="...">
</picture>

<!-- <UserProfile.Cover> -->
<picture>
  <source type="image/webp" media="(min-width: 768px)" srcset="https://xxx.cloudfront.net/profileCover/1080w/uuid.webp" alt="...">
  <source type="image/webp" srcset="https://xxx.cloudfront.net/profileCover/540w/uuid.webp" alt="...">
  <source media="(min-width: 768px)" srcset="https://xxx.cloudfront.net/profileCover/1080w/uuid.jpeg" alt="...">
  <img src="https://xxx.cloudfront.net/profileCover/540w/uuid.jpeg" alt="...">
</picture>

<!-- <Avatar> -->
<picture>
  <source type="image/webp" srcset="https://xxx.cloudfront.net/avatar/144w/uuid.webp" alt="...">
  <img src="https://xxx.cloudfront.net/profileCover/144w/uuid.jpeg" alt="...">
</picture>

[1] We do had set a long cache TTL in CloudFront, but LightHouse doesn't think so.
[2] https://css-tricks.com/responsive-images-css/
[3] https://dev.to/jsco/a-comprehensive-guide-to-responsive-images-picture-srcset-source-etc-4adj
[4] https://css-tricks.com/using-webp-images/#article-header-id-3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.