gskril / farcaster-indexer Goto Github PK

Save all data from the Farcaster protocol to Postgres

TypeScript 100.00%

farcaster-indexer's Introduction

Farcaster Indexer

This is an indexer that listens for messages from a Farcaster Hub and inserts relevant data into a postgres database.

The most performant way to run this is to co-locate everything (hub, node app, postgres, redis) on the same machine. I recommend Latitude (referral code for $200 of free credits).

How to run

Clone this repo

git clone -b hubs https://github.com/gskril/farcaster-indexer.git

Install dependencies

yarn install

Create a .env file with your hub, database, and redis connection details

cp .env.example .env

Run the latest database migrations

yarn kysely:migrate

Run the indexer

# Recommended to get the full state. You only need to run this once.
# Streaming will start after the backfill is complete.
yarn run backfill

# Ignores backfill and start streaming from the latest recorded event.
# You should run this after one initial backfill.
yarn start

How it works

Backfill and streaming are separate processes.
Every operation is run through BullMQ for better concurrency and error handling.
For backfill, the indexer adds all FIDs (in batches of 100) to a queue and processes them in parallel. The WORKER_CONCURRENCY environment variable controls how many workers are spawned.
Once backfill is complete, the indexer subscribes to a hub's event stream and processes messages as they arrive. BullMQ is used as middleware to ensure that hub events are getting handled fast enough, otherwise the stream will disconnect.

Extras

If you want to add search functionality, you can manually apply the SQL migration at src/db/search-migrations.sql

farcaster-indexer's People

Contributors

Stargazers

Watchers

farcaster-indexer's Issues

Insert links error: bind message has 38010 parameter formats but 0 parameters.

I don't know the reason and how to fix it?

Speed up searchcaster full text search

Yesterday, I asked on Farcaster whether it is possible to speed up query search results for searchcaster
I looked into the code and found that it is using the LIKE syntax for text search: https://github.com/gskril/searchcaster/blob/main/src/pages/api/search.js#L162
I looked through the database migrations to find if a specific full text search index was created: https://github.com/gskril/farcaster-indexer/tree/main/supabase/migrations (it doesn't look like it)

My plan is now to:

successfully seed the indexer
run a test query using full text search (and record the time it took)
work through https://www.postgresql.org/docs/current/datatype-textsearch.html and see if I can improve search result performance
if so, send a PR

Hardware requirements

Hello, please add information what hardware specs are needed to run indexer.

Return all grandchildren and ancestors per cast fetch

Currently, if devs fetch a cast from Searchcaster, it returns children with depth 1. Would it be desirable if it returns all the grandkids per fetch? HN and Reddit does this - when users view a post, all the comments of that post gets displayed. The API looks like this: https://hn.algolia.com/api/v1/items/33757484 (notice that the "children" key recursively contains all the replies). Plus, it would be extra nice if it also returns all the ancestors (all the casts from the root thread to the fetched cast, if there's any).

I'm down to help to make this happen.

Workers are dropping messages during backfill

Backfill with >1 concurrency will seemingly just miss all the links (or some other data type) for a fid(s) but get all casts, reactions, etc from that user.

The user(s) and data type(s) it skips are inconsistent and there are no errors in the logs. It's as if only part of the getFullProfileFromHub() function is running, or Postgres is somehow missing requests.

Verified by running a backfill with WORKER_CONCURRENCY=5, then running it again with WORKER_CONCURRENCY=1 and seeing that some tables had rows with newer updated_at values even though the message timestamp was old.

Thanks to @stephancill for finding the bug by comparing the number of rows in different tables after backfills with different concurrency values.

Backfill error: `invalid byte sequence for encoding "UTF8": 0x00`

Error thrown when backfilling FIDs 308901 - 309000. The job stopped at 71%, so the issue is likely happening somewhere between FID 308970 - 308975.

when running `yarn run seed`: new row violates row-level security policy for table "profile"

yarn run seed
yarn run v1.22.18
warning ../package.json: No license field
$ tsc && node ./dist/helpers/seed.js
Seeding recent registrations from contract logs...
Error inserting registrations {
code: '42501',
details: null,
hint: null,
message: 'new row violates row-level security policy for table "profile"'
}
Seeding profiles from Merkle APIs...

Bug: function public.gen_random_bytes(integer) does not exist

When i run yarn kysely:migrate, it report errors: function public.gen_random_bytes(integer) does not exist.
How can i fix it?