It's a NodeJS implementation of Logstash.
node-logstash is a tool to collect logs on servers. It allows sending its logs to a central server and to ElasticSearch for indexing.
On top of the elastic search, you can use a specialized interface like kibana to dive into your logs.
When I tried logstash, I had some problems. This project mainly addresses those problems. This version should have:
- lower memory footprint
- lower CPU footprint
- faster startup delay
Moreover, it's written in NodeJS, which is a perfect language for programs with many IO.
node-logstash is compatible with logstash. You can replace a node-logstash node with a logstash one. The data are formatted in the same way to be compatible with logstash UIs.
The architecture is identical to logstash architecture. You have to instantiate plugins with the node-logstash core. There are three type of modules:
- inputs plugins: where data come into node-logstash. Examples: file, zeromq transport layer
- filter plugins: extract fields from logs, like timestamps. Example: regex plugin
- outputs plugins: where data leave from node-logstash: Examples: ElasticSearch , zeromq transport layer.
A typical node-logstash deployment contains agents to crawl logs and a log server.
On agent, node-logstash is configured with inputs plugins to get logs from your software stack, and one output plugin to send logs to log server (eg. zeromq output plugin).
On log server, logs come through a zeromq input plugin, are processed (fields and timestamps extraction), and send to ElasticSearch.
Please open an issue.
October 25th, 2015.
When I started node-logstash, the ecosystem around logstash and ElasticSearch were almost non-existent. In 2015, the situation is not the same :
- Great ecosystem around ElasticSearch and logstash, FileBeat project
- Logstash is now the only way to push events to ElasticSearch (deprecation of rivers)
So, what is the future of node-logstash?
- as a tool to collect logs on files and send them through the network, node-logstash is still useful with lower size, instant start, lower CPU / Memory footprint (in my tests with logstash 1.5.0). The comparison is different with Lumberjack and FileBeat.
- as log processing tool, it has the same advantages, but the plugin ecosystem is smaller than Logstash.
- as an injection tool in ElasticSearch: ZeroMQ river will soon be unusable (deprecation of rivers). You have to use bulk api to inject data. It should be less efficient than starting an embedded ElasticSearch node, as in the original Logstash.
Node-logstash is production ready and used in production. Installation is a classical node project installation, with some scripts for native packaging.
Maintainers: currently I, @bpaquet, am the only maintainer. I will keep dependencies up to date, update the core to follow node version, but I do not have time to add features to the core. See Contributing below.
Weaknesses :
- tests are difficult to maintain, even if they are many and the code coverage is good. Replacing vows by mocha is a good way to improve that, but it's a big rework.
Which Pull Requests (PR) will be merged?
Conditions to have a PR merged :
- respect jslint
- provide documentation in /docs
- do not modify core. Modifications allowed :
- add a plugin in
Readme.md. - add optional dependencies in
package.json
- add a plugin in
- If you provide unit tests, you can write in plugin documentation that the plugin is a plugin core.
- If you do not provide unit tests, please indicate in the documentation: "Status: contributed plugin, maintained by @xxxx. Production ready.", and indicate your Github login.
You are encouraged to ask to merge plugins without tests, which are not production ready.
After install, just add your config files to /etc/node-logstash/plugins.conf.d, and restart node-logstash service node-logstash restart.
To see what options are passed to node-logstash, see here.
To change log level, do node-logstash config:set LOG_LEVEL=debug, and restart node-logstash.
- Install NodeJS, version >= 0.12
- Install build tools
- Debian based system:
apt-get install build-essential - Centos system:
yum install gcc gcc-c++ make
- Debian based system:
- Install zmq dev libraries: This is required to build the node zeromq module.
- Debian based system:
apt-get install libzmq1. Under recent releases, this package is present in default repositories. On ubuntu lucid, use this ppa. On debian squeeze, use backports. - Centos 6:
yum install zeromq zeromq-devel. Before, you have to add the rpm zeromq repo :curl http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/home:fengshuo:zeromq.repo > /etc/yum.repos.d/zeromq.repo
- Debian based system:
- Clone repository:
git clone git://github.com/bpaquet/node-logstash.git && cd node-logstash - Install dependencies:
npm install.
The executable is bin/node-logstash-agent
There are two formats for configuration:
- The legacy format uses URLs.
- The other one is identical to the logstash config format.
Note: if you are using multiple config files, you can mix formats.
A plugin is instantiated by an URL. Example: input://file:///tmp/toto.log. This URL
instantiates an input file plugin which monitors the file /tmp/toto.log.
The URLs can be specified:
- directly on the command line
- in a file (use the
--config_fileswitch) - in all files in a directory (use the
--config_dirswitch)
Example for an input file
input {
file {
path => '/tmp/toto.log'
}
}
You can use if to have an event dependent configuration. See here for details.
As for URLs, config can be specified
- directly on the command line
- in a file (use the
--config_fileswitch) - in all files in a directory (use the
--config_dirswitch)
Note: the implementation is young, all bugs reports are welcome. Note: both formats can be mixed.
--log_levelto change the log level (emergency, alert, critical, error, warning, notice, info, debug)--log_fileto redirect logs to a log file.--patterns_directoriesto add some directories (separated by , ), for loading config for regex plugin and grok plugins. Grok patterns files must be located under agroksubdirectory for each specified directory.--db_fileto specify the file to use a database for file inputs (see below)--http_max_socketsto specify the max sockets of http.globalAgent.maxSockets. Default to 100.--alarm_fileto specify a file which will be created if node-logstash goes in alarm mode (see below).
Config file for an agent:
input {
file {
path => "/var/log/nginx/access.log"
}
}
output {
zeromq {
address => ["tcp://log_server:5555"]
}
}
Config file for log server:
input {
zeromq {
address => ["tcp://0.0.0.0:5555"]
}
}
filter {
regex {
pattern => http_combined
}
}
output {
elasticsearch {
host => localhost
port => 9200
}
}
You can add easily add your plugins :
Manually :
- create a directory layout on the path of your choice :
/var/my_plugins/inputs,/var/my_plugins/outputs,/var/my_plugins/filters - set the NODE_PATH variable to
NODE_PATH=/var/my_plugins:/node_logstash_path/lib - add your plugins in
inputs,outputsorfiltersdirectory. In the plugin code, you can reference base plugins withvar base_filter = require('lib/base_filter'); - reference your plugin as usual.
With native packaging
The plugins must be deployed in /var/db/node-logstash/custom_plugins. All subdirectories already exist. The NODE_PATH is already set.
- USR1: stopping or starting all inputs plugins. Can be used to close input when output targets are failing
- USR2: see below file output plugin
Common concepts / parameters :
- Regex
- Grok
- Mutate Replace
- Grep
- Reverse DNS
- Compute field
- Compute date field
- Split
- Truncate
- Rename
- Multiline
- Json fields
- Geoip
- Eval
- Bunyan
- HTTP Status Classifier
- Remove field when equal
Common concepts / parameters :
Common concepts / parameters :
Copyright 2012 - 2014 Bertrand Paquet
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent