omnivore-app / omnivore Goto Github PK

example: https://omnivore.app/settings/emails

TODO:

• update add/delete labels api to include color and description
• setLabels api to add/remove labels to link
• update search article api to handle label: type queries

References:

• https://github.com/omnivore-app/omnivore/issues/labels
• https://docs.github.com/en/graphql/reference/objects#label

Currently Omnivore relies on a few GCP services to run, but open source users will likely want to deploy the api, web, and content fetching (puppeteer-parse) service to another platform. We need to come up with a list of target platforms and supported deployment configurations that are realistic for users wanting to deploy a minimalistic configuration.

Some of the services we currently rely on:

• Pubsub & Cloud Task -- task manager is used for schedule jobs such as content fetching
• CloudStorage -- we use CloudStorage as our main blob storage for PDF files
• Cloud Functions -- we deploy the pupeteer-parse service to a google cloud function

Other services we are using:

• SendGrid Inbound Parse -- to receive income email and invoke a webhook (running on a GCP Cloud Function)

There are two properties we should add to our Pages model:

• site_name: we are actually fetching this but not saving it:
• site_icon: There are a few ways to get site icons, apple-touch-icon-image looks like the best initial option, and then we can callback to favicon. Some details: https://stackoverflow.com/questions/20440512/why-use-link-rel-apple-touch-icon-image-src

We will also want to add these to the API response so they can be displayed in the UI. Notice how the icon of the site is displayed in the library view.

Here is an example image proxy URL that fails: https://proxy-demo.omnivore-image-cache.app/100%x0,sCh5EyC35VQVAZdNY4byf0b_pccwDyG5LeHsgI10LSLc/https://datawrapper.dwcdn.net/E8vUS/fallback.png

this came from my saved copy of the newsletter here: https://demo.omnivore.app/jacksonh/axios-pm-gen-z-surprise-17f0988918c

with an original URL here: https://link.axios.com/view/61eef5e5492d41078a32440cfxaw4.a1v4/df289925

it looks like the issue is the image sizing is set to "100%x0" so it does not get resized properly.

Original URL: https://www.cbsnews.com/amp/news/brigham-and-womens-hospital-boston-refusing-heart-transplant-man-wont-get-vaccinated/

My saved copy: https://omnivore.app/jacksonh/hospital-refusing-heart-transplant-for-man-who-won-t-get-vaccina-17f03c89f06

Note the top image is not correctly fetched by the image proxy.

Omnivore has an installation page that helps users installed native apps (iOS and Mac) and browser extensions. The current version is available at /settings/installation. We have a temporary version of this page, but need to update to the new design / layout.

Figma design (Use option three): https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=444%3A34100

Implementation Notes

• We can ignore the settings button (Navigation to Settings) information in the Figma designs
• For mobile design use the card option in figma Apps and Extensions 2

Requirements

• Dark and light mode
• Desktop and mobile mode
• Component for App installation
• Component for Browser installation

Reported by a user. I just signed up on demo so we can see their format.

some logs:

2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 headers: {
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 received: [
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 'by mx0087p1iad2.sendgrid.net with SMTP id rEM6NEJjwh Mon, 24 Jan 2022 21:24:35 +0000 (UTC)',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 'from pmta237-200.sailthru.com (unknown [192.64.237.200]) by mx0087p1iad2.sendgrid.net (Postfix) with ESMTP id 069588C0962 for <[email protected]>; Mon, 24 Jan 2022 21:24:35 +0000 (UTC)',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 'from njmta-149.sailthru.com (173.228.155.149) by pmta237-200.sailthru.com id htsch62tq4oi for <[email protected]>; Mon, 24 Jan 2022 16:24:34 -0500 (envelope-from <[email protected]>)',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 'from nj1-oddturquoise.flt (172.18.48.8) by njmta-149.sailthru.com id htsch41qqbsv for <[email protected]>; Mon, 24 Jan 2022 16:24:34 -0500 (envelope-from <[email protected]>)'
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 ],
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 'dkim-signature': 'v=1; a=rsa-sha256; c=relaxed; s=sailthru; d=axios.com; h=Date:From:To:Message-ID:Subject:MIME-Version:Content-Type:List-Unsubscribe; [email protected]; bh=yRTtlQ779D4/8js3cCWBEtwu411k9z6vcWUnSXZp7zk=; b=B44kCReNtBMYsjuORWXOI7iIXpZdAvZ2WytFejnoUs3r78o3G8pA5g+sQMVUAG9gHa7Jqg/PCDGR GIcbh6HGW9XAcmX2uKfOD+lg45Qw0OnKsKxySoMVxNZdgKsEZ0KPfQFInpALCjlKLdJ2cs+ILTNU 96NpscdJHjzqxpQ9i9E=',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 date: 'Mon, 24 Jan 2022 16:24:34 -0500 (EST)',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 from: 'Mike Allen <[email protected]>',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 to: '[email protected]',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 'message-id': '<[email protected]>',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 subject: 'Axios PM: Crisis on the Eastern Front',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 'mime-version': '1.0',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 'content-type': 'multipart/alternative; boundary="----=_Part_35505561_1026271291.1643059474872"',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 precedence: 'bulk',
Default
2022-01-24T21:24:37.139ZinboundEmailHandlerc7gx6zcmbpx6 'x-feedback-id': '7053:26472575:campaign:sailthru',

Details in discussion here: https://github.community/t/github-actions-are-severely-limited-on-prs/18179/21

Omnivore has a design for a floating action button for actions on an article. We'd like to implement these controls in multiple phases, starting with the button having a single action. This will give us an opportunity to test out the FAB and see if the design concept works well for us.

Note for this ticket we only implement the Phase One functionality, the additional details are to understand future requirements.

Collapsed:

Expanded:

We have seven actions available in this button but we can implement in multiple phases phases.

Phase one: Adjust font size

Phase two:

• Adjust margin size
• Adjust line height
• View all highlights
• Archive

Phase three: Edit labels

Phase four: Edit title and description

Requirements

• Add collapsed floating action button to the reader view. Should be always visible in lower right hand portion of the reader
• On tap, expand the view to expose action buttons (Using the set of phase 1 buttons)
• On tap, main menu button becomes a X close button
• Tapping the X closes the menu
• Tooltips and larger display on hover of each action item

• Implement the adjust font size control:

• When the adjust font size control is open, tapping the X button should close the whole floating actions and go back to the original ... state.

Figma designs: https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=344%3A32055

Note we use Phosphor Icons and should be using:

• DotsThree
• X
• Text-aa

Some code that handles adjusting font size: https://github.com/omnivore-app/omnivore/blob/main/packages/web/components/templates/article/ArticleContainer.tsx#L72 -- note there is a min size of 10 and max size of 28

There are a few issues with PDFs in Safari. The main one is the blob type isn't supported so we can't transfer the PDF data from the tab to the background extension context. Sending the full data in the message will fail as there is a max message size for passing data between the two.

return new Promise((resolve, reject) => {
      const xhr = new XMLHttpRequest();
      // load `document` from `cache`
      xhr.open('GET', '', true);
      // xhr.responseType = 'blob';
      xhr.responseType = 'arraybuffer';

      xhr.onload = function (e) {
        if (this.status === 200) {
          // `blob` response
          console.log('response', this.response);
          var arrayBuffer = this.response; // Note: not oReq.responseText
          var byteArray = new Uint8Array(arrayBuffer);
          console.log('sending arraybuffer', byteArray);
            // const blobUrl = URL.createObjectURL(this.response)

          resolve(byteArray);
        } else {
          reject(e);
        }

Steps to reproduce:

1. Save a URL from iOS
2. Go into the web app and remove the Link
3. Save the same URL again from iOS

Note that the link is not in the library.

Omnivore allows users to create addresses that receive email and add it to a user's inbox. A user can create multiple email addresses. There is a temporary version of this page available at settings/emails.

A new design for this page has been created here:

https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=444%3A33319

Notes

The design for this view is slightly off. Visually it is correct, but the user experience is inaccurate. Email addresses are immutable. They can be created, and they can be deleted, but they can not be modified. Also, they are created by Omnivore's backend, so there should be no user input on this page.

The confirmation codes are created when a GMail address is linked with an Omnivore Inbox address. So in testing they will normally not be set.

Requirements

• List emails
• Display confirmation code if set on an email
• Create an email
• Delete an email
• Copy an email address
• Copy a confirmation code
• Light and dark mode support
• Desktop and mobile support

We have an improved design for the library items cards.

Main improvements:

• reading progress bar moved to the top of the car in the grid display
• added labels to the bottom of the card
• new reading progress bar in the list view layout
• change to image size
• new icon for the MoreOptionsIcon (DotsThree from https://phosphoricons.com/ )
• A new hover state for selected cards (hover or activated with keyboard controls)

wont do:

• There is a new site icon on the cards in the top left corner, our backend doesn't support this feature so we will not implement it in this iteration

Validation

Match figma designs for:

• Grid card light theme
• Gird card dark theme
• List card light theme
• List card dark theme

Links

Code

• Main page that loads the library: https://github.com/omnivore-app/omnivore/blob/main/packages/web/pages/home.tsx
• Main component that lays out the library: https://github.com/omnivore-app/omnivore/blob/main/packages/web/components/templates/homeFeed/HomeFeedContainer.tsx
• Where the cards are created: https://github.com/omnivore-app/omnivore/blob/main/packages/web/components/templates/homeFeed/HomeFeedContainer.tsx#L602
• the LinkedItem card that we will need to update: https://github.com/omnivore-app/omnivore/blob/main/packages/web/components/patterns/LinkedItemCard.tsx

Design

• Figma Design (Grid): https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=160%3A4803
• Figma Design (List): https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=160%3A5096
• Figma design for hover: https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=270%3A27361
• Icons: https://phosphoricons.com/

Omnivore has an updated design for desktop that will display notes in the right hand margin. To create a note in Omnivore first select some text, then press the note button. A dialog will open and you can enter text. Currently that text is not visible until you click the note icon.

We would like to update the layout to display the notes in the margin on large displays:

The Highlight objects have an annotation?: string property. If set, the highlight has a note and it should be displayed in the margin.

Note that we will only display notes if there is enough margin space.

https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=121%3A8424

There should be three ways of using Labels in search. We can use GitHub as a guide for each of these methods:

• contains: label:foo -- the returned items have the specified label
• OR: label:foo,bar,baz -- the returned items have any of the specified lablels
• NOT -label:foo -- the returned items do not have the specified label

Steps to reproduce:

1. Navigate to the reader
2. Adjust the font size
3. Navigate back to the library
4. Go back to the reader -- note that the font size has not been changed

These should at least be saved in the personalization cookie so they are maintained per-device.

We should be able to receive these PDFs and add them to user's libraries.

Since creating the public repo we need to update our vercel configuration to integrate with this repo instead of the old.

Lets just have the share extension say Saved to Omnivore. I think the reason Apple had issue with the previous mac version was just because the refresh button didn't show up.

On iOS we need the ability to create, and copy email addresses (based on https://omnivore.app/settings/emails ) . This can be a simple tableview type layout linked to the profile page.

The two pieces of functionality on the view are a Create Email Address button which would call the createNewsletterEmail API, a list of all the existing emails (fetched with the newsletterEmails API), and the ability to copy an email address to the pasteboard (tap an address, and display a Snackbar after copying).

When users import large libraries we should switch over to the low priority queue to keep performance predictable for all users.

To do this we can add a rate limit type check to the import path. In before adding to the task queue based on some heuristics we can decide to use either the normal or the low priority task queue. There is a task queue omnivore-prod-queue-low that dispatches to the cloud function at a lower rate (3 per second vs 500).

To do this we will need to create a low priority queue in the demo environment also.

As a first pass at our logic I think we can do:

5 articles added in the last minute: use low queue
default: use normal queue

You can see how we pass a priority param to the cloud task function here:

Reported by a user, can sign up here: https://golangweekly.com/

Omnivore is adding an extensible settings system for user options. Right now we do not have a temporary version of this page, but it should be located at /settings. This view will only be displayed on mobile, we will handle this in the dropdown menu, by making the settings button go to /settings/installations on desktop, and /settings on mobile.

Figma (use Option Three): https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=444%3A34100

Implementation notes

This task involves updating settings/installation, settings/labels, and settings/emails to display the left menu on desktop and creating a new page settings that will be displayed on mobile.

Requirements

• Dark and light mode support
• Add left menu to all settings pages
• Create new page for settings on mobile
• Update dropdown menu to link to correct page based on device

I think Mono's is really well done: https://github.com/mono/mono/blob/main/.github/CONTRIBUTING.md

Reported by a user. I signed up on demo to validate.

Based on the updated designs we can improve the typography on the reader view:

As discussed. Trying out a collectionview and going to just two panes instead of three.

When saving pages such as https://www.feynmanlectures.caltech.edu/I_01.html from the feynmanlectures site we fail to fetch images. The server is sending a 403 response. What is happening is on the site when you load a page there is a cookie set by the page, that cookie authorizes fetching the images, so individual requests, like the ones made by the image proxy are rejected.

These don't include a list-url and have private domains such as milkroad.com so its hard to tell what is a newsletter vs an informational post. It looks like the best thing we might have to match on is a Read Online URL, and potentially some tracking pixels in content.

Tells us beehiiv is the newsletter platform:

<img alt="" height="auto" width="340" style="display:block;width:100%;height:auto;border:0;outline:none;text-decoration:none;" src="https://media.beehiiv.net/uploads/asset/

Tells us the message is a newsletter, not an informational post:

<a target=3D"_blank" class=3D"link" style=3D"color:#ffff=
ff;text-decoration:underline;text-decoration-color:#ffffff; " href=3D"http:=
//link.mail.beehiiv.com/ss/c/....">Read Online</=
a></div></td></tr></table><!--[if mso | IE]></td></tr></table>

Important to note that the link is not always hosted at beehiiv.com:

<a target="_blank" class="link" style="color:#222222;text-decoration:underline;text-decoration-color:#222222; " href="https://u23463625.ct.sendgrid.net/.......">Read Online</a></div></td></tr></table>

A user just testing Gmail forwarding didn't receive their code. Looking at logs (on production) we see this message:

Error: invalid confirmation email at handleConfirmation (/workspace/build/src/newsletter.js:79:15) at /workspace/build/src/index.js:110:59 at /workspace/node_modules/@sentry/serverless/dist/gcpfunction/http.js:80:16 at /workspace/node_modules/@sentry/serverless/dist/utils.js:33:20 at Object.apply (/workspace/node_modules/@sentry/serverless/dist/utils.js:51:28) at /workspace/node_modules/@google-cloud/functions-framework/build/src/invoker.js:98:17 at processTicksAndRejections (internal/process/task_queues.js:77:11)
Error: invalid confirmation email at handleConfirmation (/workspace/build/src/newsletter.js:79:15) at /workspace/build/src/index.js:110:59 at /workspace/node_modules/@sentry/serverless/dist/gcpfunction/http.js:80:16 at /workspace/node_modules/@sentry/serverless/dist/utils.js:33:20 at Object.apply (/workspace/node_modules/@sentry/serverless/dist/utils.js:51:28) at /workspace/node_modules/@google-cloud/functions-framework/build/src/invoker.js:98:17 at processTicksAndRejections (internal/process/task_queues.js:77:11)

We use the react-hot-toast library for displaying toast messages in a few places in the web app. You can see its usage in the code by searching for toast.error and toast.success.

These toasts are using default UI from the library that doesn't fit our UI.

In the designs you can see we have a new UI for success UIs:

This can be seen in Figma here: https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=444%3A34018

We should create a new showSuccess function in utils that can be imported into components/pages and display the above Toast UI. We can then update all uses of toast.success to use the new UI.

There isn't a design for failure yet but we can use the same design but with the icon WarningCircle and a background colour #cc0000

The service has its own URL validation, but we should use the utility method.

We need to upgrade the API server to apollo-server 3 to remove some deprecated packages.

It looks like the main issue with this will be the @sanitize directive, but we might be able to use an OSS equivalent.

Report from a user:

"https://golangweekly.com/leave/8b299ffecf", 
"reportComment": 
"The extracted content is incomplete.  Compare to the original: https://golangweekly.com/issues/401

The omnivore version includes the main section but omits the "Code & Tools" section."

Steps to reproduce:

• Create a new Label foo on /settings/labels
• Add the label to a link (navigate to link, press l, check box beside label)
• Go back to /settings/labels
• Delete the label

You'll note the label is still on the article if you refetch it

Based on design discussed on discord, lets add more readable space on iOS by collapsing the navigation bar on the reader view when the user scrolls.

• Collapse when the user scrolls up
• Use the reader background colour for the statusbar
• On tap of the statusbar display the navigation bar without scrolling to top

We have a few places we should automatically add a label to a link. In some cases we might want to automatically remove a label too.

The most obvious and easiest to do is a Newsletter label on links we receive via email newsletters. This would make it easy for users to filter them out.

But we could also consider some other cases like Recently Read (this could be done via another query also). If in search we added a Recently Read label it would be another nice filter,

These are generating 404s now. Likely because they can't join on pages

Omnivore is cleaning up the settings and installation views in our web app based on new designs. Most of these pages are already created but need layout and display changes. The settings view work is prioritized and split into separate issues below based on the importance of each view.

Figma: https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=444%3A30941

Implementation Plan

1. Emails and Installation: The most time sensitive view is the Emails view. This can be implemented before the main index as we will link directly to it in our dropdown menu. This means we do not need to implement the side menu on the emails desktop page right away.

2. Index After Emails, and installation are implemented we can add the side menu on the emails and installation page, and add the index page for the mobile view.

3. Labels view

Prioritized implementation order:

• Emails view
• Installation View
• Index View
• Labels View

Based on latest designs: https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=121%3A8139

A few design suggestions:

• Set the opacity on the profile picture to 48% if there is a picture
• Use a white background if there is no photo
• Set opacity to 100% on hover

Design for the avatar in light and dark mode

Figma Design: https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=344%3A30092

Code for the avatar header: https://github.com/omnivore-app/omnivore/blob/f7e3a1657eb9c7af1a67bf25faf8265f75dc3d4a/packages/web/components/elements/Avatar.tsx

Original URL https://www.theatlantic.com/ideas/archive/2022/03/putins-war-dispelled-the-worlds-illusions/623335/

Omnivore allows users to create labels and attach them to library items. A label has a name, colour, and description , users can create labels, delete labels. They can also edit a labels color and description.

This page has a temporary version available at settings/labels

Figma (start at option three): https://www.figma.com/file/ILgs6aXrPfukXOMf5Yibra/Omnivore---Deliverables?node-id=444%3A34097

Implementation Notes

1. We can ignore the USES column in the design.

Requirements

• Light and dark mode support
• List label w/columns: Name, Description, Color, Actions
• Edit button to enter edit mode
• Save/Cancel buttons when in edit mode
• Allow editing descriptions
• Allow selecting colors from a drop down (7 colors listed)
• Allow creating a custom color
• Refresh button to choose a random color (from the 7 listed)