Breaking changes
and specifically rest api changes

I'd love to see this beat be docker-able like:
https://github.com/Ingensi/dockbeat

Kindly create a installation guide mentioning various steps in order to install elasticbeat on Windows.

Current state of the beat requires one socket per cluster to be configured in configuration file. Otherwise, results in back-end ES will be multiplied. Another important consequence arise when configured node is not available or reachable. Data request will fail and no data for given period will be pushed and indexed.

Solution is to implement function which will request all available nodes for each cluster and saves them. Then, when one request fails, we can try another socket from cluster.
While beat is running, this list of IP/cluster will be also updated every defined-period of time to keep in current t the state of the cluster. Let's say every 60 seconds (possibly more).

Any suggestions are welcome.

Ref - elastic/beats#864

On some servers elasticsearch is not able to read cpu-usage, wich results in -1 as cpu_percent-value.
The value ist stored as uint64 and should be changed to a signed integer. Otherwise you will run into errors while processing the json response.

Hi there,

First I want to thank you for this wonderful work of art. I started playing with this beat. I could collect all the metrics of elasticsearch regarding the cluster, nodes and health. Now I wanted some specify metrics that is metrics of a specify topic.

I was wondering if there is a mean to get those metrics with this beat.

Thanks.

When downloading repository from Github site, the repository contains all files except one file i.e. elasticbeat-*.json present inside elasticbeat/kibana/dashboards/index-pattern.

Possible Problem:- In Windows you cannot use * in file name.

Possible Solution:- kindly rename the filename to elasticbeat.json.

PS: I am unsure whether the suggested solution will work but just an idea.

As per the installation guide of running in Kibana, following steps have been mentioned:

cd ~/workspace/go/src/github.com/radoondas/elasticbeat/kibana

get the content of the file import_dashboards.sh and save on the disk

chmod u+x import_dashboards.sh
./import_dashboards.sh -url http://localhost:9200

However, the above path does not exist & hence import_dashboards cannot be fetched and exported. Please correct this behaviour for easy to follow guide.

Hi @radoondas
can you please make one more change?
Can you please rename the file name under kibana/dashboards/dashboard from Cluster-view.json to Elasticbeat-Dashboard.json.
Also kindly rename the title under the same file from Cluster-view to Elasticbeat Dashboard.

Reason: If anyone loads the sample dashboard, then he can easily find the dashboard in Kibana by name of Elasticbeat Dashboard.

Implement Tests for elasticbeat.

Hi

I am unable to import either the Search or Visualization or Dashboard in Kibana as present inside the kibana/dashboard directory.

Upon opening a saved search json file (cluster-stats) having its code as:-
{
"title": "Cluster stats",
"description": "",
"hits": 0,
"columns": [
"type"
],
"sort": [
"@timestamp",
"desc"
],
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{"index":"elasticbeat-","query":{"query_string":{"query":"type:cluster_stats","analyze_wildcard":true}},"filter":[],"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"":{}},"require_field_match":false,"fragment_size":2147483647}}"
}
}

Upon verifying with an existing saved search found few lines to be missing in the file due to which it isn't importing. Added _id,_type & _source which was then able to import into Kibana properly.
[
{
"_id": "Cluser-stat",
"_type": "search",
"_source": {
"title": "Cluster stats",
"description": "",
"hits": 0,
"columns": [
"type"
],
"sort": [
"@timestamp",
"desc"
],
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{"index":"elasticbeat-","query":{"query_string":{"query":"type:cluster_stats","analyze_wildcard":true}},"filter":[],"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"":{}},"require_field_match":false,"fragment_size":2147483647}}"
}
}
}
]

Similarly faced problems for Visualizations & Dashboard which too were able to solve & import into Kibana in that way.

Update Go to 1.7.1

I am facing difficulty while installing elasticbeat. I am using Ubuntu.
I am mentioning the steps along with error while installing elasticbeat:-

1. Installed Go using following command:-
   sudo apt-get install golang

2. Verified Go language is installed:-
   go version
   Output:- go version go1.2.1 linux/amd64

3. Added following variables to ~/.bashrc file:-
   export GOROOT="/usr/lib/go"
   export GOPATH="$HOME/go"
   export PATH="$GOROOT/bin:$PATH"

4. Navigated to workspace & created directory:-
   cd $GOPATH
   mkdir -p src/github.com/radoondas

5. Navigated to the directory and cloned elasticbeat repository using following command:-
   cd $GOPATH/src/github.com/radoondas/elasticbeat
   git clone https://github.com/radoondas/elasticbeat.git

6. Then running make command gives following error:-
   yuvraj@ubuntu:~/go/src/github.com/radoondas/elasticbeat$ make
   go build

   main.go:6:2: cannot find package "github.com/elastic/beats/libbeat/beat" in any of:
   /usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/beat (from $GOROOT)
   /home/yuvraj/go/src/github.com/elastic/beats/libbeat/beat (from $GOPATH)
   beater/elasticbeat.go:9:2: cannot find package "github.com/elastic/beats/libbeat/cfgfile" in any of:
   /usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/cfgfile (from $GOROOT)
   /home/yuvraj/go/src/github.com/elastic/beats/libbeat/cfgfile (from $GOPATH)
   beater/elasticbeat.go:10:2: cannot find package "github.com/elastic/beats/libbeat/common" in any of:
   /usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/common (from $GOROOT)
   /home/yuvraj/go/src/github.com/elastic/beats/libbeat/common (from $GOPATH)
   beater/elasticbeat.go:11:2: cannot find package "github.com/elastic/beats/libbeat/logp" in any of:
   /usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/logp (from $GOROOT)
   /home/yuvraj/go/src/github.com/elastic/beats/libbeat/logp (from $GOPATH)
   beater/elasticbeat.go:12:2: cannot find package "github.com/elastic/beats/libbeat/publisher" in any of:
   /usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/publisher (from $GOROOT)
   /home/yuvraj/go/src/github.com/elastic/beats/libbeat/publisher (from $GOPATH)
   make: *** [build] Error 1

The above error was due to libbeat folder not found.

Solved above error by cloning beats director of Elastic using following steps:-

1. Created a directory
   mkdir -p $GOPATH/src/github.com/elastic
2. Navigated to the directory and cloned Elastic beats repository using following command:-
   cd $GOPATH/src/github.com/elastic
   git clone https://github.com/elastic/beats.git
3. Copied the folder in the location it was expecting.
   sudo cp -r github.com/ /usr/lib/go/src/pkg/

After doing the following & again running make command from
yuvraj@ubuntu:~/go/src/github.com/radoondas/elasticbeat$ make
gives following error:-

/usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/outputs/kafka/client.go:10:2: cannot find package "github.com/Shopify/sarama" in any of:
/usr/lib/go/src/pkg/github.com/Shopify/sarama (from $GOROOT)
/home/yuvraj/go/src/github.com/Shopify/sarama (from $GOPATH)
/usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:16:2: cannot find package "github.com/dustin/go-humanize" in any of:
/usr/lib/go/src/pkg/github.com/dustin/go-humanize (from $GOROOT)
/home/yuvraj/go/src/github.com/dustin/go-humanize (from $GOPATH)
/usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/outputs/redis/client.go:10:2: cannot find package "github.com/garyburd/redigo/redis" in any of:
/usr/lib/go/src/pkg/github.com/garyburd/redigo/redis (from $GOROOT)
/home/yuvraj/go/src/github.com/garyburd/redigo/redis (from $GOPATH)
/usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/common/geolite.go:9:2: cannot find package "github.com/nranchev/go-libGeoIP" in any of:
/usr/lib/go/src/pkg/github.com/nranchev/go-libGeoIP (from $GOROOT)
/home/yuvraj/go/src/github.com/nranchev/go-libGeoIP (from $GOPATH)
/usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/beat/beat.go:51:2: cannot find package "github.com/satori/go.uuid" in any of:
/usr/lib/go/src/pkg/github.com/satori/go.uuid (from $GOROOT)
/home/yuvraj/go/src/github.com/satori/go.uuid (from $GOPATH)
/usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/common/config.go:4:2: cannot find package "github.com/urso/ucfg" in any of:
/usr/lib/go/src/pkg/github.com/urso/ucfg (from $GOROOT)
/home/yuvraj/go/src/github.com/urso/ucfg (from $GOPATH)
/usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/common/config.go:5:2: cannot find package "github.com/urso/ucfg/yaml" in any of:
/usr/lib/go/src/pkg/github.com/urso/ucfg/yaml (from $GOROOT)
/home/yuvraj/go/src/github.com/urso/ucfg/yaml (from $GOPATH)
/usr/lib/go/src/pkg/github.com/elastic/beats/libbeat/outputs/transport/proxy.go:8:2: cannot find package "golang.org/x/net/proxy" in any of:
/usr/lib/go/src/pkg/golang.org/x/net/proxy (from $GOROOT)
/home/yuvraj/go/src/golang.org/x/net/proxy (from $GOPATH)
make: *** [build] Error 1

Can you please help in solving the error?

I see that we are missing critical fields like

1. Index Size in bytes
2. Cluster Size in bytes
3. Node Size in bytes
4. Id Cache size in bytes
5. Filter cache size in bytes.

It would be nice to have these fields added.

Hello, cannot get this to build with "go version go1.8.1 darwin/amd64".

make
go build
#>
./main.go:13: cannot use beater.New() (type *beater.Elasticbeat) as type "github.com/elastic/beats/libbeat/beat".Creator in argument to "github.com/elastic/beats/libbeat/beat".Run
make: *** [build] Error 2

Any ideas?

Thanks, Lynn

Hello @radoondas,

First thanks for your work on this project! I'm staring to use it and it will give us much more control/visibility over the performance of our ES clusters.

I would like to suggest the addition of the indices [1] metrics when collecting nodes stats.

These metrics are important as they show how long your nodes are taking to index documents, index errors, refresh times, query cache hit/miss ratios and many other interesting metrics.

Would you be willing to integrate such a modification? If yes, I could try to work on a PR so we can discuss, review and eventually merge.

Let me know what do you think.

[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html#node-indices-stats

Thanks,

Update GO to 1.6.3

Update GO to version 1.7.3

Hello @radoondas,

I'm running a ES cluster behind a reverse proxy and this proxy is set to talk HTTPS.

When I set the config option output.elasticsearch.hosts to ["https://server:443"] despite elasticbeat printing this to the logs:

2017/04/05 18:32:39.607965 client.go:452: DBG  ES Ping(url=https://server:443, timeout=1m30s)
2017/04/05 18:32:39.735167 client.go:461: DBG  Ping status code: 200

I see the following erros on the log: I'm running with -e -d "*".

2017/04/05 18:33:41.821657 output.go:158: INFO Trying to load template for client: &{{https://server:443   %!s(*http.Client=&{0xc420200000 <nil> <nil> 90000000000}) %!s(bool=true) %!s(func() error=0x73be20)} infra-elk-es map[] {{ <nil> %!s(bool=false) %!s(int=0) %!s(int=0) %!s(int=0)}  %!s(elasticsearch.state=0) }}
2017/04/05 18:33:41.821667 client.go:480: DBG  HEAD https://server:443/_template/elasticbeat <nil>
2017/04/05 18:33:41.829050 output.go:165: INFO Existing template will be overwritten, as overwrite is enabled.
2017/04/05 18:33:41.847960 client.go:416: INFO Elasticsearch template with name 'elasticbeat' loaded
2017/04/05 18:33:41.847985 single.go:79: INFO Error publishing events (retrying): not connected
2017/04/05 18:33:41.847989 single.go:143: INFO send fail

If I change the value to ["http://server:80"] all work as expected and elasticbeat is able to publish all metrics.

Also in the logs elasticbeat print the raw documents that it's sendingo to ES and looking at these documento we can see that it is capable of reading data from a HTTPS ES but not writing.

2017/04/05 18:41:28.320726 client.go:171: DBG  Publish: {
  "@timestamp": "2017-04-05T18:41:28.320Z",
  "beat": {
    "hostname": "dea0d5f0672d",
    "name": "elasticbeat"
  },
  "cluster_health": {
    "active_primary_shards": 21,
    "active_shards": 21,
    "active_shards_percent_as_number": 50,
    "cluster_name": "elasticsearch",
    "delayed_unassigned_shards": 0,
    "intializing_shards": 0,
    "number_of_data_nodes": 1,
    "number_of_in_flight_fetch": 0,
    "number_of_nodes": 1,
    "number_of_pending_tasks": 0,
    "relocating_shards": 0,
    "status": "yellow",
    "status_num": 1,
    "task_max_waiting_in_queue_millis": 0,
    "timed_out": false,
    "unassigned_shards": 21
  },
  "type": "cluster_health",
  "url": "https://server:443"
}

I also tested the ability do write data to this ES using HTTPS and all work as expected. I used a simple curl call do remove an index.

Even setting output.elasticsearch.protocol to https does not work.

Thanks,