Giter Club home page Giter Club logo

kindle-to-notion's Introduction

🚀 Kindle to Notion

Seamlessly transfer your Kindle highlights to a Notion Database!

🔁 Usage

Before you can run the sync, you need to complete the setup section.

To sync your highlights, just upload the new My Clippings.txt file into your GitHub repo. It will trigger a GitHub Action to sync the newly added highlights to your Notion books database.

⚙️ Setup

  • Duplicate my Notion books database as a template in your Notion account. You don't need to clone the contents of my database. You only need to database table. This will serve as your Notion books database.

  • Create a new internal integration at https://www.notion.so/my-integrations and copy the Notion API key.

  • Go to your Notion dashboard. Open your books database as a page.

  • Click on the three dots icon (more options) and go to Add connection. Search for the integration you created in the previous step and add it as a connection.

  • On the Notion books database page, go to Share and click on Copy link to get the URL of the Notion database. Note: You must open the books database as a page before copying the link.

  • Now, extract the Database Id from the link as shown in the example below:

    Book Database Link: https://www.notion.so/arkalim/346be84507ff482b80fceb4024deadc2?v=e868075eaf5749bc941e617e651295fb
    Book Database Id: 346be84507ff482b80fceb4024deadc2
    
  • Connect your Kindle E-Reader to your computer. Navigate to Kindledocuments and copy My Clippings.txt.

  • Create a GitHub repository and add your My Clippings.txt to the root of the repository. Now, add a file sync.yaml to the repository at location .github/workflows with the following content:

    name: Sync Kindle Highlights
    on: [push, workflow_dispatch]
    jobs:
      sync-highlights:
        runs-on: ubuntu-latest
        steps:
          - name: Checkout repository
            uses: actions/checkout@v3
    
          - name: Sync highlights
            uses: addnab/docker-run-action@v3
            with:
              image: ghcr.io/arkalim/kindle-to-notion:master
              run: node /code/dist/main.js
              options: |
                -v ${{ github.workspace }}:/code/resources 
                -e BOOK_DB_ID=${{ vars.BOOK_DB_ID }}
                -e NOTION_API_KEY=${{ secrets.NOTION_API_KEY }}
    
          - name: Commit cache changes
            uses: EndBug/add-and-commit@v9
            with:
              message: Commit cache changes
              add: sync.json
    
          - name: Push cache changes
            uses: ad-m/github-push-action@master
            with:
              github_token: ${{ secrets.GITHUB_TOKEN }}
    
  • The above step will trigger a GitHub action to sync the highlights. But, it will fail as you haven't yet setup the NOTION_API_KEY and the BOOK_DB_ID. To do that, go to the GitHub repository settings, click on Secrets and Variables and then click on Actions. Now, create a secret by the name NOTION_API_KEY and a variable by the name BOOK_DB_ID, with their corresponding values as obtained in the previous steps.

  • Now, you need to allow GitHub Action to save the cache (make updates) to the repository. For that, go to repository settings. Click on Actions and then on General. Scroll down to Workflow permissions and select Read and write permissions.

  • You can now manually trigger the GitHub Action by going to the Actions tab in the GitHub repo. This will sync the highlights for the first time and create a sync.json file (cache) in the repository. Afterwards, whenever you want to sync your highlights, just upload the new My Clippings.txt file into your GitHub repo. It will automatically trigger the action to sync the newly added highlights.

For reference on how to setup your repo, take a look at my Kindle Highlights Repo

❗️For Nerds

  • Every highlight made on Kindle is appended at the end of My Clippings.txt
  • Book Name is used as the primary key to facilitate upsert operation in Notion database. So, this field should be left untouched. However, the other fields like Title, Author, Date Started, Date Finished, Satus and Genre could be modified as per your wish.
  • This tool only syncs the new highlights made to each book. If no new highlights have been made, no sync takes place.
  • The info about the last sync is stored in sync.json present in your repository containing My Clippings.txt.
  • In case you wish to sync every book all over again, delete the sync.json file from the repo and delete all the highlights present in your Notion.
  • GitHub Action is used for CICD. Whenever a push is made to the master branch, the Docker image is rebuilt and pushed to GitHub container registry.
  • Your GitHub repo containing My Clippings.txt runs the GitHub action to sync the highlights to Notion everytime you push a change to it. It can also be triggered manually from the Actions tab.

kindle-to-notion's People

Contributors

alfredchenpiano avatar anubhavshrimal avatar anuraagdjain avatar arkalim avatar chrisfartek avatar fnavales avatar juninhofreitas avatar manigandan-dorairaj avatar thaotpphung avatar yedrup avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

kindle-to-notion's Issues

Sync Date Started and Date Finished?

Goal: in Notion Database, automatically sync:

  • Date Started = date of earliest note
  • Date Finished = date of latest note

It seems like I'll need to alter the /interfaces/notion.ts file and /interfaces/clipping.ts file, but I'm not sure about that. Can you point me in the right direction?

Problem when using "My Clippings.txt"

In the end of the process, where it says

Connect your Kindle to your computer. Navigate to Kindle arrow_right documents and copy My Clippings.txt. Replace my My Clippings.txt in resources folder with yours.

After doing it, running npm start, I get:

image

No books and no clipping have been imported as a result. I've tested with the Clippings that came in the resources folder, and it worked. So, I have no ideia what is happening...

Maybe it has something to do with this:

image

Apparently, the clippings from my kindle it's not a plain/text file and maybe thats whats happening.

My Clippings.txt

error message when running manually sync.yaml

Hello
i have a pb when running sync.yaml. here is the error message. could you help me ? It is in Sync Higlights part :

Unable to find image '3425285a4d654d7ebb8dd6c24903a639:latest' locally
docker: Error response from daemon: pull access denied for 3425285a4d654d7ebb8dd6c24903a639, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.

See 'docker run --help'.

I've checked variable and secret : there both OK. 3425285a4d654d7ebb8dd6c24903a639 is my BOOK_DB_ID.

Error if more than 100 highlights

Got the following error after the command "npm start". I guess, it's because I have more than 100 highlights in the book "The Brothers Karamazov". Is there a way to fix it? Also I see another error: body.parent.page_id should be defined. I don't know how to fix this. HELP please.

Screenshot 2022-09-18 at 05 02 03

[Tutorial] Code changes to support for older Kindle devices

I have an older kindle (d01200) and the current regex was unable to capture the data from the clippings file. Below are the changes one could do to sync their data to Notion.

// src/models/parser.ts

private regex =
    /(.+) \((.+)\)\r*\n- Your Highlight (location|Location|page)( |(.+))([0-9]+)-([0-9]+) \| Added on ([a-zA-Z]+), ([a-zA-Z]+|[0-9]+) ([0-9]+|[a-zA-Z]+),? ([0-9]{4}), ([0-9]+):([0-9]+\s? A?P?M?)\r*\n\r*\n(.+)/gm;
    
// src/models/parser.ts line 38-39

const time = `${match[12]}:${match[13]}`;
const highlight = match[14];    

Input data looks something like below:

No Rules Rules (Reed Hastings;Erin Meyer)
- Your Highlight Location 2620-2620 | Added on Sunday, January 22, 2023, 08:20 PM

farm for dissent, socialize the idea, and for big bets, test it out. 
==========
No Rules Rules (Reed Hastings;Erin Meyer)
- Your Highlight Location 3725-3725 | Added on Saturday, January 28, 2023, 05:10 AM

Erin’s book, The Culture Map. 
==========

Thanks to @arkalim for making this project. Solves the most crucial piece for me, just had to fix the regex for my device.

My clippings are not recognized

I followed the instructions of the README.md to configure all steps. At last, I found some clippings were not recognized and some were recognized correctly. I extracted the clippings were not recognized and pused them up again. Unfortunately, they still were not recognized. @arkalim Can you help me test it? I added the My Clippings.txt. Thx for any help!

image

image

My Clippings.txt

Highlights not updating

Screenshot 2022-08-07 at 02 47 49
Screenshot 2022-08-07 at 02 49 39
Screenshot 2022-08-07 at 02 48 33

Good morning,

what a great project. Thanks for your work :)

Unfortunately the updating of highlights does not work (Looks like it does not read the updated My Clippings file properly)

Do you have an idea :)?

Cheers, Phil

Update Highlight Count

Hey, since the code already uses highlightCount in business logic, could you update it to add it as a property to the book page? And update it whenever one syncs new highlights.

Chinese synchronization failed

I followed your video on YouTube, but the synchronization keeps failing? Does this version support Chinese? There is something wrong with my operation. T T, can you help me find out?
PixPin_2024-01-06_12-53-11
My Clippings.txt

Problem after npm start

Hey Abdur, great work and a great idea. However it seems there's a problem when I run npm start, the highlights are not syncing, I think the problem is on my side, but I'm not very good at coding so I can't figure it out. Could you help me out?

Attaching the terminal log.

Thanks, Marek

terminal.txt

Highlights from My Clippings.txt is not syncing anymore

Hi @arkalim ,

I just setup the kindle-to-notion sync but everytime I run the workflow, nothing gets synced to Notion and I see the following in the log:
📋 Parsing Clippings

➕ Grouping Clippings

💹 Stats for Clippings

🟢 Every book is already synced!

But nothing really gets synced to Notion DB. Could you please help me debug this issue?

Thanks!

No match for regex of clippings

Hi,

Thanks a lot for this tool! It's super cool.

Found a small issue (may be due to my Kindle Paperwhite version) and resolution.

The clippings are in a different format than the one used in this code.

Eg.
'=======
Kindle Paperwhite User's Guide, 3rd Edition (Amazon)

  • Your Highlight at location 347-348 | Added on Wednesday, 28 June 2017 22:45:15

tap the sides of the screen. To exit Kindle
'=======

Key differences in the format are:

  • on Location vs at location
  • Month dd, yyy HH:mm:ss AM/PM vs dd Month yyyy HH:mm:ss (24Hr format)
  • CR and line-breaks after date and time

I have replaced the regex code with the below, and it worked.

/(.+) \((.+)\)\r*\n- Your Highlight (at|on) (location|Location|page)( |(.+))([0-9]+)-([0-9]+) \| Added on ([a-zA-Z]+), ([0-9]+) ([a-zA-Z]+) ([0-9]{4}) ([0-9]+):([0-9]+):([0-9]+)\r*\n\r*\n(.+)/gm

Maybe if you can modify the Regex to match the above example too, it would help.

Thanks

Syncing notes as well as highlights

Hi,

Thanks for this, it works great in my experience so far!

Less an issue, more a question... any plans to include notes, along with highlights?

I tested this out today and found that, whilst the notes are present in the MyClippings.txt file, they didn't appear in Notion after sync. I couldn't see any reference to notes in the codebase when I had a brief look, so assuming this isn't part of the functionality, but wondered if it might be in future?

Thanks!

Issues with Syncing

Hi, after following your Yt tutorial I sadly still have issues...
my Workspace key looks a bit different, but I took the 32char code between library- and the ?

still I get an error : Failed to sync highlights APIResponseError: Could not find database with ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

https://www.notion.so/[workspacename]/Library-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx?pvs=x

(ofc instead of xxx there is the "real" key)

Do you have any ideas why I get this issue?
Best regards

Automate

Hi!
I was wondering if there is a way to automate the syncing process where every time you plug in your kindle and type a single command the clippings file gets replaced and it syncs.

Docker image not working on Apple M1 chipset

Try to run the setup command as mentioned in the README.md and I get the below error.

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Can you publish an image for arm platform too?

No LICENSE provided

Hi there,

I am interested in using this software and potentially making some changes to it (perhaps in a fork), however, there is license associated with it. Could you please supply a license so users interested in forking know what requirements are in place for them?

Thanks 😊

(Really cool idea btw)

API token is invalid

It tells me my API token is invalid.

image

The integration is added to the database and I provided the correct IDs for both of them

Not syncing all books

The code only works with books that were purchased through Amazon, it doesn't sync PDFs.

Do not sync

I have a problem and I can't resolve it. Apparently it says that the books are already synchronized, but they do not appear at all on the page created in Notion.

Everything was configured exactly as in the video, but I still couldn't get it to work.

Set Up job
Current runner version: '2.309.0'
Operating System
Runner Image
Runner Image Provisioner
GITHUB_TOKEN Permissions
Secret source: Actions
Prepare workflow directory
Prepare all required actions
Getting action download info
Download action repository 'actions/checkout@v3' (SHA:f43a0e5ff2bd294095638e18286ca9a3d1956744)
Download action repository 'addnab/docker-run-action@v3' (SHA:4f65fabd2431ebc8d299f8e5a018d79a769ae185)
Download action repository 'EndBug/add-and-commit@v7' (SHA:8c12ff729a98cfbcd3fe38b49f55eceb98a5ec02)
Download action repository 'ad-m/github-push-action@master' (SHA:29f05e01bb17e6f28228b47437e03a7b69e1f9ef)
Complete job name: sync-highlights

Build addnab/docker-run-action@v318s

Build container for action use: '/home/runner/work/_actions/addnab/docker-run-action/v3/Dockerfile'.

Checkout repository

Run actions/checkout@v3
Syncing repository: mripol/kilde-highlight
Getting Git version info
Temporarily overriding HOME='/home/runner/work/_temp/1a55afff-9978-4de3-b32c-f2af882031c1' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
/usr/bin/git config --global --add safe.directory /home/runner/work/kilde-highlight/kilde-highlight
Deleting the contents of '/home/runner/work/kilde-highlight/kilde-highlight'
Initializing the repository
Disabling automatic garbage collection
Setting up auth
Fetching the repository
Determining the checkout info
Checking out the ref
/usr/bin/git log -1 --format='%H'
'4cd602ea8e4c1463ef5cf30fc9561d07c4b49270'

Sync highlights

Run addnab/docker-run-action@v3
/usr/bin/docker run --name b0ae1e1492cb74cdbaf9ff359f9094dcf_a4279e --label 94351b --workdir /github/workspace --rm -e "INPUT_IMAGE" -e "INPUT_RUN" -e "INPUT_OPTIONS" -e "INPUT_SHELL" -e "INPUT_REGISTRY" -e "INPUT_USERNAME" -e "INPUT_PASSWORD" -e "INPUT_DOCKER_NETWORK" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/kilde-highlight/kilde-highlight":"/github/workspace" 94351b:0ae1e1492cb74cdbaf9ff359f9094dcf
Unable to find image 'ghcr.io/arkalim/kindle-to-notion:master' locally
master: Pulling from arkalim/kindle-to-notion
31e352740f53: Pulling fs layer
2629b68d4311: Pulling fs layer
ddb7cc70f260: Pulling fs layer
18afe6373474: Pulling fs layer
edf0b2b9bb48: Pulling fs layer
745762490991: Pulling fs layer
a84e6dbb39e0: Pulling fs layer
ad5591fb15e2: Pulling fs layer
e14969b0211a: Pulling fs layer
18afe6373474: Waiting
edf0b2b9bb48: Waiting
745762490991: Waiting
a84e6dbb39e0: Waiting
ad5591fb15e2: Waiting
e14969b0211a: Waiting
ddb7cc70f260: Verifying Checksum
ddb7cc70f260: Download complete
31e352740f53: Verifying Checksum
31e352740f53: Download complete
edf0b2b9bb48: Verifying Checksum
edf0b2b9bb48: Download complete
18afe6373474: Verifying Checksum
18afe6373474: Download complete
31e352740f53: Pull complete
745762490991: Verifying Checksum
745762490991: Download complete
a84e6dbb39e0: Verifying Checksum
a84e6dbb39e0: Download complete
2629b68d4311: Verifying Checksum
2629b68d4311: Download complete
e14969b0211a: Verifying Checksum
e14969b0211a: Download complete
ad5591fb15e2: Verifying Checksum
ad5591fb15e2: Download complete
2629b68d4311: Pull complete
ddb7cc70f260: Pull complete
18afe6373474: Pull complete
edf0b2b9bb48: Pull complete
745762490991: Pull complete
a84e6dbb39e0: Pull complete
ad5591fb15e2: Pull complete
e14969b0211a: Pull complete
Digest: sha256:e619f20321eb17d192d20c77045ed3a9ad1e08c508941279b309b8db189a5875
Status: Downloaded newer image for ghcr.io/arkalim/kindle-to-notion:master
📋 Parsing Clippings

➕ Grouping Clippings

💹 Stats for Clippings

🟢 Every book is already synced! "-> apears that the books are already sync"

Commit cache changes

Run EndBug/add-and-commit@v7

Warning: The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Warning: The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Warning: The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Warning: The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Running in /home/runner/work/kilde-highlight/kilde-highlight
Add input parsed as single string, running 1 git add command.

Using 'mripol [email protected]' as author.
Using "Commit cache changes" as commit message.
Internal logs
Working tree clean. Nothing to commit.
Outputs

Push cache changes

Run ad-m/github-push-action@master
Push to branch main
Everything up-to-date

Post Checkout repository

Post job cleanup.
/usr/bin/git version
git version 2.42.0
Temporarily overriding HOME='/home/runner/work/_temp/67449942-d286-45bc-a0c0-6ad70a92ba12' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
/usr/bin/git config --global --add safe.directory /home/runner/work/kilde-highlight/kilde-highlight
/usr/bin/git config --local --name-only --get-regexp core.sshCommand
/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
/usr/bin/git config --local --name-only --get-regexp http.https://github.com/.extraheader
http.https://github.com/.extraheader
/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http.https://github.com/.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"

Complete job

Cleaning up orphan processes

Docker Commands not functioning

image

I am getting all sorts of errors when running the commands specified in the readme. I am running Windows 10 with Docker installed. Does anyone have tips for how to enter the docker commands in powershell without throwing errors?

No sync

Hi, nothing is synchronized. I noticed that I get this error on github: "Warning: The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files."
Screenshot 2023-05-03 alle 23 28 17

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.