Giter Club home page Giter Club logo

search-indexer's Introduction

Search Indexer

These are a group of Lambda functions that are designed to be triggered by webhooks on our self-hosted Ghost instances. These functions are created using the Serverless framework and are used to index articles from various news sources into a universal search bar.

Here are the current Ghost events / webhooks, and their endpoints:

Ghost Event Endpoint
Post published .../stage/add-index
Post unpublished .../stage/delete-index
Post deleted .../stage/deleted-index
Published post updated .../stage/update-index

The Lambda function on each endpoint receives the blog post from the Ghost webhook and updates the correct Algolia index based on the origin of the Ghost webhook.

Here are the currently configured origins and Algolia indices:

Ghost Origin Algolia index
http://localhost:2368 news-dev
https://www.freecodecamp.org/news/ news
https://www.freecodecamp.org/espanol/news/ news-es
https://chinese.freecodecamp.org/news/ news-zh
https://www.freecodecamp.org/portuguese/news/ news-pt-br
https://www.freecodecamp.org/italian/news/ news-it
https://www.freecodecamp.org/japanese/news/ news-ja
https://www.freecodecamp.org/arabic/news/ news-ar
https://www.freecodecamp.org/bengali/news/ news-bn
https://www.freecodecamp.org/urdu/news/ news-ur
https://www.freecodecamp.org/swahili/news/ news-sw
https://www.freecodecamp.org/korean/news/ news-ko
https://www.freecodecamp.org/turkish/news/ news-tr
https://www.freecodecamp.org/french/news/ news-fr
https://www.freecodecamp.org/hindi/news/ news-hi
https://www.freecodecamp.org/german/news/ news-de
https://www.freecodecamp.org/vietnamese/news/ news-vi
https://www.freecodecamp.org/indonesian/news/ news-id
https://www.freecodecamp.org/haitian/news/ news-ht

Prerequisites:

  • AWS CLI and an AWS account with access to Secrets Manager
  • An Algolia account access to the pre-configured indices above (see table)

How to prepare your local machine

  1. Create an Algolia account with access to the freeCodeCamp News indices
  2. Go to LastPass, search for the shared [keys] [AWS] [Lambda] - News Indexer note, and add the [news-indexer-lambda] credentials to ~/.aws/credentials
  3. Clone this repo and install the necessary packages with npm ci
  4. Copy sample.env.yml to a new file named .env.yml
  5. In .env.yml set AWS_PROFILE to 'news-indexer-lambda'
  6. Start developing

How to develop and test locally

  1. Make your changes and run npm start
  2. Open your local Ghost Integrations dashboard at http://localhost:2368/ghost/#/settings/integrations
  3. Click "Add custom integration" to create a new custom integration
  4. Check the console output and add the Lambda function URLs (ex: https://localhost:3000/dev/add-index) to the correct Ghost events. Check the Ghost events and endpoints table near the top of this document for more information
  5. Publish, update, and delete articles on Ghost locally. You can view your changes on Algolia on the dev application in the news-dev index
  6. If you make changes while the server is running, kill the server with Ctrl / Cmd + C. Then restart the server with npm start to test your changes

How to deploy changes

  1. After a PR with changes is accepted and merged, deploy changes with npm run deploy-dev or npm run deploy

How to change Algolia API keys on AWS

Generate a new key on Algolia

  1. Sign into Algolia with the [email protected] account
  2. Select the dev or Production application from the dropdown at the top of the page
  3. Click "API Keys", then click the "All API Keys" tab near the top of the screen
  4. Click the "New API Key" button, and generate a new key with the following ACLs (action control lists): search, addObject, deleteObject, browse, listIndexes, deleteIndex
  5. Click the "Create" button
  6. Copy the new Algolia API key

Update the Algolia key entry in AWS Secrets Manager

  1. Create an IAM account under the freeCodeCamp AWS team with access to Secrets Manager
  2. Sign in and go to https://us-east-2.console.aws.amazon.com/secretsmanager/
  3. Click "serverless-indexer"
  4. Click "Retrieve secret value" and click the "Edit" button
  5. Paste the new Algolia API key in as the value of ALGOLIA_ADMIN_KEY in either the dev or prod object
  6. Click the "Save" button

How to remove a service

WARNING: Removing a dev or prod service is only necessary in a few cases, such as migrating to a new region.

By removing a service, the base URL for the Lambda functions will be lost. This means that all the webhooks in each Ghost instance will need to be updated once the service is redeployed.

To deploy updates, it's not necessary to remove a service first. Just follow the deployment instructions here.

Use the following commands with caution:

  1. Install Serverless globally with npm install -g serverless
  2. Run serverless remove --stage [dev / prod] --region us-east-2

search-indexer's People

Contributors

renovate[bot] avatar renovate-bot avatar scissorsneedfoodtoo avatar raisedadead avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.