anansi-js / anansi Goto Github PK

Visit docs website

A Node.js web spider for site search. Inspired by the deprecated https://github.com/algolia/docsearch-scraper NOTE: The project is in a very early stage.

Anansi lets you crawl your website, scrape content that matches html selectors you specified in a config file, then index them in a search engine (currently supports Algolia as well as your own custom search plugin) to serve your site search features.

Under the hood, the project uses puppeteer-cluster, which in turn uses puppeteer

npm

npm install --save puppeteer # the project uses puppeteer-cluster and puppeteer under the hood
npm install --save puppeteer-cluster # the project uses puppeteer-cluster and puppeteer under the hood
npm install --save anansi-js

yarn

yarn add puppeteer puppeteer-cluster anansi-js

import { crawlSite } from 'anansi-js';

const letsStartCrawling = async () => {
  await crawlSite({
      configFilePath: 'path/to/your/config.json',
      searchEngineOpts: {
        algolia: {
          apiKey: '<your algolia API key>',
          appId: '<your algolia app ID>',
          indexName: '<your algolia index name>'
        }
      },
      diagnostics: true,
      logLevel: 'debug',
      maxIndexedRecords: 300
    });
}

letsStartCrawling().then(() => {
  process.exit(0);
})

• crawlSite

instantiates a Spider object, initializing it based on your config file and settings, then invoking its crawl method.

all of the scraper settings groups (each group except the default ties to a specific URL pattern)

A group of a scraper settings - mostly hierarchy and metadata selectors, grouped by a specific URL pattern

hierarchy selectors. Essentially a mapping from html selectors to indexed hierarchy levels

For detailed information about the way the spider works, visit: https://github.com/anansi-js/anansi/blob/main/docs/under-the-hood.md