平台搭建参考文章

https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7

https://www.digitalocean.com/community/tutorials/how-to-gather-infrastructure-metrics-with-topbeat-and-elk-on-centos-7

https://www.digitalocean.com/community/tutorials/adding-logstash-filters-to-improve-centralized-logging

https://www.digitalocean.com/community/tutorials/how-to-use-kibana-dashboards-and-visualizations

https://www.digitalocean.com/community/tutorials/how-to-map-user-location-with-geoip-and-elk-elasticsearch-logstash-and-kibana

yml语法校验

http://yaml-online-parser.appspot.com/

http://www.yamllint.com/

linux tutorials

https://www.digitalocean.com/community/tutorials

http://www.unixmen.com/

http://linoxide.com/

tomcat日志分析

https://aggarwalarpit.wordpress.com/2015/12/03/configuring-elk-stack-to-analyse-apache-tomcat-logs/

https://www.systemcodegeeks.com/web-servers/apache/configuring-elk-stack-analyse-apache-tomcat-logs/

http://stackoverflow.com/questions/25429377/how-can-i-integrate-tomcat6s-catalina-out-file-with-logstash-elasticsearch

https://blog.codecentric.de/en/2014/10/log-management-spring-boot-applications-logstash-elastichsearch-kibana/

https://blog.lanyonm.org/articles/2014/01/12/logstash-multiline-tomcat-log-parsing.html

https://spredzy.wordpress.com/2013/03/02/monitor-your-cluster-of-tomcat-applications-with-logstash-and-kibana/

log4j日志分析

https://qbox.io/blog

https://github.com/logstash/log4j-jsonevent-layout

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html

https://blog.lanyonm.org/articles/2015/12/29/log-aggregation-log4j-spring-logstash.html

http://www.tianmaying.com/tutorial/elastic-logstash-kibana

grok

http://grokdebug.herokuapp.com/

http://grokconstructor.appspot.com/do/match

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

logstash-input-jdbc

https://www.elastic.co/blog/logstash-jdbc-input-plugin

Elasticsearch安装

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

/etc/yum.repos.d/elasticsearch.repo

[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=https://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

yum install elasticsearch
chkconfig --add elasticsearch

centos 7

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Shield 2.0安装

./bin/plugin install license
./bin/plugin install shield
./bin/elasticsearch

创建用户es_admin/es_admin 角色为admin

./bin/shield/esusers useradd es_admin -p es_admin -r admin
esusers useradd <username> [-p <password>] [-r <roles>]
-h,--help                    Shows this message

-p,--password <password>     The user password

-r,--roles <roles>           Comma-separated list of the roles of the user

curl -u es_admin -XGET 'http://115.28.163.63:9200/'

Logstash安装

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

/etc/yum.repos.d/logstash.repo

[logstash-2.3]
name=Logstash repository for 2.3.x packages
baseurl=https://packages.elastic.co/logstash/2.3/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Kibana安装

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

/etc/yum.repos.d/kibana.repo

[kibana-4.5]
name=Kibana repository for 4.5.x packages
baseurl=http://packages.elastic.co/kibana/4.5/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

yum install kibana
chkconfig --add kibana

Winlogbeat安装

https://download.elastic.co/beats/winlogbeat/winlogbeat-1.2.3-windows.zip

PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-winlogbeat.ps1

winlogbeat.yml

winlogbeat:
  registry_file: C:/ProgramData/winlogbeat/.winlogbeat.yml

  event_logs:
    - name: Application
    - name: Security
    - name: System

output:
  elasticsearch:
    hosts:
      - 192.168.0.228:9200

logging:
  to_files: true
  files:
    path: C:/ProgramData/winlogbeat/Logs
  level: info

Test

  PS C:\Program Files\Winlogbeat> .\winlogbeat.exe -c .\winlogbeat.yml -configtest -e

curl -XPUT 'http://192.168.0.228:9200/_template/winlogbeat' -d@/etc/winlogbeat/winlogbeat.template.json
curl -XDELETE 'http://192.168.0.228:9200/winlogbeat-*'

Packetbeat安装

sudo yum install libpcap
curl -L -O https://download.elastic.co/beats/packetbeat/packetbeat-1.2.3-x86_64.rpm
sudo rpm -vi packetbeat-1.2.3-x86_64.rpm

/etc/packetbeat/packetbeat.yml

curl -XPUT 'http://115.28.163.63:9200/_template/packetbeat' -d@/etc/packetbeat/packetbeat.template.json
curl -XDELETE 'http://115.28.163.63:9200/packetbeat-*'

sudo /etc/init.d/packetbeat start
curl -XGET 'http://115.28.163.63:9200/packetbeat-*/_search?pretty'

Filebeat安装

curl -L -O https://download.elastic.co/beats/filebeat/filebeat-1.2.3-x86_64.rpm
sudo rpm -vi filebeat-1.2.3-x86_64.rpm

/etc/filebeat/filebeat.yml

curl -XPUT 'http://115.28.163.63:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json
curl -XDELETE 'http://115.28.163.63:9200/filebeat-*'
sudo /etc/init.d/filebeat start

Topbeat安装

curl -L -O https://download.elastic.co/beats/topbeat/topbeat-1.2.3-x86_64.rpm
sudo rpm -vi topbeat-1.2.3-x86_64.rpm

/etc/topbeat/topbeat.yml

curl -XPUT 'http://115.28.163.63:9200/_template/topbeat' -d@/etc/topbeat/topbeat.template.json
curl -XDELETE 'http://115.28.163.63:9200/topbeat-*'
sudo /etc/init.d/topbeat start
curl -XGET 'http://115.28.163.63:9200/topbeat-*/_search?pretty'

I'm using elasticsearch 1.5

and it is working perfectly the most part of the time, but everyday at the same time it becomes crazy, CPU % goes to ~70% when the average is around 3-5% there are SUPER servers with 32GB reserved for lucene, swap it is lock and clearing the cache doesn't solve the problem (it doesn't take down the heap mem)

Settings:

3 servers (nodes) 32 cores and 128GB RAM each
2 buckets (indices) one with ~18 million documents (this one doesn't receive updates pretty often just indexing new docs) the other one have around 7-8 million documents but we are constantly bombarding it with updates search delete and indexing as well

The best distribution for our structure, was to have only 1 shard per node with not replicas, we can afford to have a % of the data off for few seconds, that will be back as soon as the server get online again, and this process is fast enough since it doesn't need to relocate anything. previously we used to have 3 shards with 1 replica, but the issue mentioned above occurs as well, so is easy to figure it out that the problem is not related with the distribution.

Things that I already tried,

Merging, i try to use the Optimize API trying to give less load to the schedule merge, but actually the merging process takes a lot of R/W of the disk but it doesn't affect substantially the mem or the CPU load.

Flushing, I tried to flush with long and shot intervals, and the results were the same nothing

changed, since flushing affects directly the merging process and as mentioned above, merging process doesn't takes that much of the CPU or mem usage.

managing the cache, clearing it manually but it doesn't seems to take the cpu load to normal state not even for a moment.

Here is the most of the elasticsearch.yml configs

Force all memory to be locked, forcing the JVM to never swap

bootstrap.mlockall: true

Threadpool Settings

Search pool

threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: 200

Bulk pool

threadpool.bulk.type: fixed
threadpool.bulk.size: 60
threadpool.bulk.queue_size: 3000

Index pool

threadpool.index.type: fixed
threadpool.index.size: 20
threadpool.index.queue_size: 1000

Indices settings

indices.memory.index_buffer_size: 30%
indices.memory.min_shard_index_buffer_size: 12mb
indices.memory.min_index_buffer_size: 96mb

Cache Sizes

indices.fielddata.cache.size: 30%
#indices.fielddata.cache.expire: 6h #will be depreciated & Dev recomend not to use it
indices.cache.filter.size: 30%
#indices.cache.filter.expire: 6h #will be depreciated & Dev recomend not to use it

Indexing Settings for Writes

index.refresh_interval: 30s
#index.translog.flush_threshold_ops: 50000
#index.translog.flush_threshold_size: 1024mb
index.translog.flush_threshold_period: 5m
index.merge.scheduler.max_thread_count: 1

here is the stats when the server is in a normal state:
node_stats_normal.txt

Node stats during the problem.
node_stats.txt

I will appreciate any help or discussion that can point me in the right direction to get rid of this behavior

thanks in advance..

Regards,

Daniel

Originally posted by @ACV2 in elastic/elasticsearch#4288 (comment)