Giter Club home page Giter Club logo

flexsearch-server's Introduction


Search Library

High-performance FlexSearch Server for Node.js

A full documentation of FlexSearch is available here: https://github.com/nextapps-de/flexsearch

Cluster (Vertical Scaling)

FlexSearch Server actually use Node.js Clusters and a Connection Pool to balance all incoming load through the workers.


The performance gain of workers depends on the complexity of the index and how much time a query takes.

For most situations disabled workers performs a lot better, e.g.:

Single Thread
(Request per seconds)
Cluster
(Request per seconds)
Query Time < 10 ms 6122 1215
Query Time > 1000 ms 114 1041

Note: A query time of 10 ms is already based on a large index.

Sharding / Replication (Horizontal Scaling)

Basically the current codebase has all requirements to provide also scaling in horizontal direction. If this project gains popularity I will made an implementation and also provide some docker bindings for an out-of-the-box installation.

Installation

npm install flexsearch-server

Run setup once when not installing via "npm install":

npm run setup

Run as a single web server:

npm start

Run as a server cluster:

npm run cluster

Run with a specified environment:

npm start production
npm run cluster test

The server is listening at:

Environment Server Address
development http://localhost
production

http://localhost:6780
https://localhost:6780
test http://localhost

The default port for production is 6780 (over HTTP) and 6781 (over HTTPS), you can change the default ports in the configs.

When NODE_ENV is set, all manual passed environments will be overridden.

API Overview

RESTful

Request Type Endpoint Description
GET / Returns info about the index
POST /add/:id/:content Add content to the index
POST /update/:id/:content Update content of the index
GET /search/:query Search for query
POST /remove/:id Remove id from the index

JSON

Request Type Endpoint Body Description
POST /add JSON Add multiple contents to the index
POST /update JSON Update multiple contents of the index
GET /search JSON Search for query (custom search)
POST /remove JSON Remove multiple ids from the index

Server Environment Variables

Variable Values Default Description
PORT integer 80 (development)
6780 (production)
Server listening port
PORT_SSL integer 443 (development)
6781 (production)
Server listening port
SSL true
false
false Starts a HTTPS server (additionally to HTTP)
FORCE_SSL true
false
false Accept HTTPS connections only
COMPRESS true
false
true Enable/Disable response compression (gzip)
WORKER integer
"auto"
"auto" Sets the number of workers or automatically determine available cpus
AUTOSAVE integer
false
10000 Enables persistent handler. Set the delay in milliseconds or disable by setting to false

Configuration

When you do not pass any custom options, the environment will be inherited from NODE_ENV by default.

Override the current environment (NODE_ENV) by adding a field in the root of package.json:

  "flexsearch": "production" // development, test

Or add a config payload in the root of package.json:

The field "client" holds the flexsearch options

  "flexsearch": {
    "server": {
      "debug": true,
      "port": 80,
      "port_ssl": 443,
      "force_ssl": false,
      "https": true,
      "compress": true,
      "autosave": 2000,
      "worker": "auto"
    },
    "client": {
      "async": false,
      "cache": 0,
      "worker": 4
    }
  }

Or provide a config json file for each environment respectively and pass file locations:

You can also provide a config json file to override the default settings globally.

  "flexsearch": {
    "development": "config/development.json",
    "production": "config/production.json",
    "test": "config/test",
    "default": "config/default.json"
}

Or just provide a flexsearch.json in the root of your project.

Persistence

Turn on persistence by setting a numeric value as delay on config.autosave. Flexsearch will automatically save the index to the local filesystem and also loads it when starting.

Disable persistence by setting this field to false.

Setup SSL for HTTPS Connections

Inside the folder cert add the certificate as a .crt file and the key as a .pem file for each environment respectively.

Development Environment:

cert/development.crt
cert/development.pem

Production Environment:

cert/production.crt
cert/production.pem

Important Notes

You can increase workers without any drawbacks, but when you decrease worker count, the old index doesnt restore completely from the filesystem. This will be fixed in an upcoming version. As long please do not decrease worker count when using persistent index.


Copyright 2019 Nextapps GmbH
Released under the Apache 2.0 License

flexsearch-server's People

Contributors

ts-thomas avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

flexsearch-server's Issues

Unable to use HTTPS locally

Firstly — great work. Thanks for building this.

I'm attempting to run flexsearch-server locally using HTTPS.

I've created certificates, following the instructions at letsencrypt.org/docs/certificates-for-localhost.

I've renamed the files from localhost.crt to development.crt and from localhost.key to development.pem and copied them to both /cert:

/cert/development.crt
/cert/development.pem

and /certs:

/certs/development.crt
/certs/development.pem

(The console message refers to /certs and the documentation refers to /cert.)

However, I continue to get the following error when running npm run development:

Could not find certificates located at: /certs. Therefore HTTPS server can't initialize.

Any ideas? FlexSearch Server runs fine from localhost over HTTP.

Any plans to support multiple indexes?

Hello. Is there any plans to support multiple indexes? This could be handy for searching multiple different types of documents with different settings, for example, users and logs.

fs.exists(path, callback) Deprecated

The read_from_file function in helper.js uses a deprecated fs.exists() method before attempting to readFile().

Might be better do something like...

read_from_file: () => {
    fs.open(filename, 'r', (err, fd) => {
        if (err) {
            if (err.code === 'ENOENT') {
                console.error(filename + 'does not exist')
                return
            }
        }
        throw err
    }
    fs.readFile(fd) ...
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.