Giter Club home page Giter Club logo

jhipster-console's Introduction

JHipster Console (deprecated)

Notice

As of JHipster version 7, the JHipster Console is unsupported.

Azure DevOps Build Status Build Status Join the chat at https://gitter.im/jhipster/jhipster-console

This is the JHipster Console, based on the ELK Stack and Zipkin. It provides a default configuration to get started with logs, metrics and traces monitoring with ELK and Zipkin as well as some nice dashboards.

Please refer to the JHipster Monitoring Documentation for instructions on how to set up the JHipster Console.

Quick Start

Start everything by running docker-compose up -d in the bootstrap/ directory, then access Kibana at localhost:5601 and Zipkin at localhost:9411.

Contributing

PRs are welcome ! When contributing, make sure you force build and restart everything with docker-compose down && docker-compose build --no-cache && docker-compose up.

Images hosted on Docker Hub

  • jhipster/jhipster-console Automated build Docker Pulls
  • jhipster/jhipster-elasticsearch Automated build Docker Pulls
  • jhipster/jhipster-logstash Automated build Docker Pulls
  • jhipster/jhipster-zipkin Automated build Docker Pulls
  • jhipster/jhipster-alerter Automated build Docker Pulls
  • jhipster/jhipster-curator Automated build Docker Pulls
  • jhipster/jhipster-import-dashboards Automated build Docker Pulls

jhipster-console's People

Contributors

anjia0532 avatar cbornet avatar coderla avatar csaban avatar deepu105 avatar eumji025 avatar gitter-badger avatar gmarziou avatar jdubois avatar klainte avatar lmbai avatar mptardy avatar mraible avatar pascalgrimaud avatar pierrebesson avatar shaolans avatar sudharakap avatar thomastardy avatar yavuz-gokirmak avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jhipster-console's Issues

[FEATURE] Log forwarding from a file

Some people are concerned with lower performance when using a socket connection from their app and would rather have a separate agent read from log files.

We could use either logstash or filebeat for this task.

jhipster-import-dashboards's wait-for-elasticsearch.sh: Permission denied

git clone the latest jhipster-console and build it from docker build jhipster-import-dashboards -t jhipster-import-dashboards ,but this container it's not work.

2017/8/4 下午2:27:41/bin/sh: ./wait-for-elasticsearch.sh: Permission denied
2017/8/4 下午2:27:57/bin/sh: ./wait-for-elasticsearch.sh: Permission denied

the create-index-patterns.sh and wait-for-elasticsearch.sh ownner is root.

bash-4.2$ ls -lah
total 13M
drwxr-x--- 1 root metricbeat 4.0K Aug  3 14:07 .
drwxr-x--- 1 root metricbeat 4.0K Aug  3 14:07 ..
-rw-rw-r-- 1 root root        333 Aug  3 09:40 create-index-patterns.sh
-rwxr-x--- 1 root metricbeat  13M Jul 18 20:40 import_dashboards
-rwxr-x--- 1 root metricbeat  15K Jul 18 20:39 migrate_beat_config_1_x_to_5_0.py
-rw-rw-r-- 1 root root        242 Aug  3 09:40 wait-for-elasticsearch.sh
bash-4.2$

Update the official documentation regarding Elastalert

According to the official Jhipster website, there is a Kibana dashboard to display Elastalert alerrs
https://jhipster.github.io/monitoring/#elastalert

Go to http://localhost:5601/app/kibana#/dashboard/alerting-dashboard to see the history of all your alerts.

I tried but there is no dashboard anymore, confirmed by the following commit:
33d5408

I tried to install the plugin by my self using BitSensor documentation but failed also. I think BitSensor developped this dashboard for internal use mainly not for OSS.

I think The official doc should present Elastalert rules but no the rest is misleading.

Prefix metric field names

To have "metric_type": "GAUGE" instead of "type": "GAUGE" because type is already used ("type": "syslog")

jhipster-console cannot resolve links on Docker Cloud

@PierreBesson same issue that makes us change the Dockerfile to use the latest version from alpine. But researching jhipster-console, it uses kibana, and kibana uses directly debian.

Any ideas why jhipster-console container is not able to resolve the hostnames on Docker Cloud Stack configuration?

Cannot launch JHipster Console in Windows

I have this error for the JHipster-Console container when launching 'docker-compose up' on Windows.

/docker-entrypoint.sh: /tmp/entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/docker-entrypoint.sh: line 25: /tmp/entrypoint.sh: Success
/docker-entrypoint.sh: /tmp/entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/docker-entrypoint.sh: line 25: /tmp/entrypoint.sh: Success
/docker-entrypoint.sh: /tmp/entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/docker-entrypoint.sh: line 25: /tmp/entrypoint.sh: Success

@pascalgrimaud I need your help here.

Add logstash-* index

And set it as default so that there is no need to create it when starting Kibana the first time
Also remove filebeat-* index

elastalert-create-index fails because config.yaml not present

In start-elastalert.sh the celastalert-create-index is executed in the wrong directory. Therefore the elastalert config.yaml cannot be found an the following error occurs.

hipster-console_1 | Creating Elastalert index in Elasticsearch... jhipster-console_1 | Enter elasticsearch host: Traceback (most recent call last): jhipster-console_1 | File "/usr/local/bin/elastalert-create-index", line 9, in <module> jhipster-console_1 | load_entry_point('elastalert', 'console_scripts', 'elastalert-create-index')() jhipster-console_1 | File "/opt/elastalert/elastalert/create_index.py", line 52, in main jhipster-console_1 | host = args.host if args.host else raw_input('Enter elasticsearch host: ') jhipster-console_1 | EOFError: EOF when reading a line jhipster-console_1 | Starting Kibana jhipster-console_1 | INFO:elastalert:Starting up jhipster-console_1 | WARNING:elasticsearch:GET http://elk-elasticsearch:9200/elastalert_status/elastalert/_search?size=1000 [status:404 request:0.030s] jhipster-console_1 | WARNING:elasticsearch:GET http://elk-elasticsearch:9200/elastalert_status/elastalert_status/_search?_source_include=endtime%2Crule_name&size=1 [status:404 request:0.019s] jhipster-console_1 | ERROR:root:Error querying for last run: TransportError(404, u'index_not_found_exception', u'no such index')

This can be reproduced when enabling alerting (ENABLE_ALERTING=true)

Visualizations fail with monolith app

Because they look for the field instance_id that is only sent with micro-services
Shouldn't we base on app_name.raw instead ? This would make the visualizations available for monolith apps and even multi-monolith (app1 monolith and app2 monolith monitored by the same ELK).

jhipster-elasticsearch Ulimit.soft must be an integer

OS:Window 10
ERROR: for jhipster-elasticsearch Ulimit.soft must be an integer
Traceback (most recent call last):
File "docker-compose", line 3, in
File "compose\cli\main.py", line 67, in main
File "compose\cli\main.py", line 117, in perform_command
File "compose\cli\main.py", line 922, in up
File "compose\project.py", line 416, in up
File "compose\parallel.py", line 69, in parallel_execute
ValueError: Ulimit.soft must be an integer
Failed to execute script docker-compose

Webhooks on certain conditions

Is that possible to have a way to configure webhooks for certain conditions to help scale the infrastructure? CPU levels, network bandwidth usage, number of requests, jvm heap size...

No log for the logstash-* pattern

Starting the console with the default config, I can access the Kibana dashboard but it doesn't seem to find anything under the logstash-* pattern.

I've verified that jhipster-cloud is sending the log through the appender to the console IP on UDP port 5000. (tcpdump on the sending machine)
And I can see that that the docker machine has started the logstash instance and is listening to UDP 5000 (with netstat)

Is there anything I should do to get the default config working?

When I go to Kibana's settings tab I have this message Unable to fetch mapping. Do you have indices matching the pattern?

And on the Discover tab here's what I have if it can help:
screen shot 2016-04-21 at 18 32 40

On the log side, I get a corresponding 404:
jhipster-console_1 | {"type":"response","@timestamp":"2016-04-21T10:30:53+00:00","tags":[],"pid":1,"method":"get","statusCode":404,"req":{"url":"/elasticsearch/logstash-*/_mapping/field/*?_=1461234653807&ignore_unavailable=false&allow_no_indices=false&include_defaults=true","method":"get","headers":{"host":"192.168.0.74:5601","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://192.168.0.74:5601/app/kibana","kbn-version":"4.5.0","connection":"keep-alive"},"remoteAddress":"10.0.2.2","userAgent":"10.0.2.2","referer":"http://192.168.0.74:5601/app/kibana"},"res":{"statusCode":404,"responseTime":7,"contentLength":9},"message":"GET /elasticsearch/logstash-*/_mapping/field/*?_=1461234653807&ignore_unavailable=false&allow_no_indices=false&include_defaults=true 404 7ms - 9.0B"}

docker-compose up fail to start logstash

During the startup process after docker-compose up from the project's root, the only log for logstash is the following:

elk-logstash_1 | {:timestamp=>"2016-04-21T03:54:54.466000+0000", :message=>"translation missing: en.logstash.runner.configuration.file-not-found", :level=>:error} jhipsterconsole_elk-logstash_1 exited with code 1

and consequently, only jhipster-console and elk-elasticsearch are started.

Graph ratios

IMO it is better to graph ratios than raw values, esp. in the context of micro-services/multiple apps.
For intance I would like to graph JVM memory usage ratio = JVM memory used/ JVM memory max.
I couldn't do it with Kibana. With timelion I can do it with .es(jvm.memory.total.used, avg:metric_value).divide(.es(jvm.memory.total.max, avg:metric_value)) but I didn't manage to split between instance_name like we do with Kibana.
Anyone has an idea ?

Upgrading ELK stack to 5.3 and use official images

I couldn't manage to configure elasticsearch to run clustered on my prod environment. I am passing these env vars to elastic container, may be they are wrong..

        environment:
          - cluster.name=es-cluster
          - transport.host=0.0.0.0
          - discovery.zen.minimum_master_nodes=1

I tried new supported images docker.elastic.co/elasticsearch/elasticsearch:5.3.0 and clustering is working with a couple of environment variable in docker-compose

        environment:
          - cluster.name=es-cluster
          - "discovery.zen.ping.unicast.hosts=dh-elk-02"

Trying to understand the problem in images which we are using in jhipster. Should I focus to solve the problem in elasticsearch:5.1.2-alpine image (deprecated) or we should upgrade images to supported ones.

I can try to upgrade and get issue #49 also which is related..

InstanceId displayed in registry and kibana are different

The instanceId displayed in the registry is not the same in jhipster-console.
I'm not sure if it's a normal behavior.

So to confirm that, in LoggingConfiguration.java, I used this code:

    public void addLogstashAppender(LoggerContext context) {
        log.info("Initializing Logstash logging");

        LogstashSocketAppender logstashAppender = new LogstashSocketAppender();
        logstashAppender.setName("LOGSTASH");
        logstashAppender.setContext(context);
        String customFields = "{\"app_name\":\"" + appName + "\",\"app_port\":\"" + serverPort + "\"," +
            "\"instance_id\":\"" + instanceId + "\"}";
        log.debug(">>>>>>>>>>>>>> {}", instanceId);

I got:

x.x.x.x.x.config.LoggingConfiguration    : >>>>>>>>>>>>>> referentiel:93d13e7ea190e1e6cf0e017880728550

In the main method, I added:

log.debug("----->>>>>>>>>>>>>>> {}", env.getProperty("eureka.instance.instanceId"));

I got:

----------------------------------------------------------
	Config Server: 	Connected to the Registry config server! (local)
----------------------------------------------------------
2017-04-11 12:01:01.400 DEBUG 2319 --- [  restartedMain] x.x.x.referentiel.ReferentielApp  : ----->>>>>>>>>>>>>>> referentiel:1edc5154c85e71730ce6acc6bd296a70

In jhipster-console, I have: 93d13e7ea190e1e6cf0e017880728550
And in registry: 1edc5154c85e71730ce6acc6bd296a70

Not forwarding logs after a while

Hi, I encountered to a wired issue.

After all the microservices are up I can see the logs very well in dashboard, all metrics logs and application logs.

But after one day or less, I cannot see the metrics/application logs of some microservices, the interested part is gateway always is there, I never saw it happened for gateway, it will happen only for microservices.

Note: I can see all the logs in file logs and container logs but dashboard

Configure alerting with a webhook reporting example

OK, I just discovered http://webhook.site/ and the great thing about this is that you just have to pass a UUID in the URL to have your webhook testing setup just for you.
This means we can have real alerting working out of the box ! So we will have something to show off for demos.

I'm currently doing it for Prometheus alerting, for now I'm only adding one alert on user registration. Then I will do it for Elastalert.

Also interesting to mention that Grafana will have support for Alerting as well (http://docs.grafana.org/guides/whats-new-in-v4/). Although we decided before on not using it and sticking with Kibana (to have logs and metrics in one place) it might be interesting to investigate how Timeseries support for Elasticsearch has progressed.

Ulimit.soft must be an integer Failed to execute script docker-compose

CMD> docker-compose up -d
Starting javiercuicapuza_jhipster-logstash_1
Starting javiercuicapuza_jhipster-console_1
Starting javiercuicapuza_jhipster-alerter_1

ERROR: for jhipster-elasticsearch Ulimit.soft must be an integer
Traceback (most recent call last):
File "docker-compose", line 3, in
File "compose\cli\main.py", line 64, in main
File "compose\cli\main.py", line 116, in perform_command
File "compose\cli\main.py", line 876, in up
File "compose\project.py", line 416, in up
File "compose\parallel.py", line 66, in parallel_execute
ValueError: Ulimit.soft must be an integer
Failed to execute script docker-compose

ZipKin Dependency Issue

Hi,

I tried to clone the project and run the docker-compose up -d command but I am receiving the following error:

fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

ERROR: unsatisfiable constraints:
coreutils (missing):
required by: world[coreutils]

ERROR: Service 'jhipster-zipkin' failed to build: The command '/bin/sh -c apk add --no-cache coreutils && curl -SL $ZIPKIN_REPO/io/zipkin/dependencies/zipkin-dependencies/$ZIPKIN_DEPENDENCIES_VERSION/zipkin-dependencies-$ZIPKIN_DEPENDENCIES_VERSION.jar > zipkin-dependencies.jar' returned a non-zero code: 1

I can confirm I am able to download the gz files. Is there perhaps anything I am missing? It might be related to gliderlabs/docker-alpine#55 perhaps.

Regards

How to handle logstash.conf deploying on Docker Cloud

The issue is: when you run docker-compose it is easy, the logstash.conf file is there, locally on your computer and you map this local directory to your container.

When you try to run it directly on Docker Cloud through a slack file, what do to? It is not there.

Simplify distribution by creating jhipster-logstash, jhipster-elasticsearch and jhipster-alerting official images

Create the following automated build at docker hub in the JHipster organisation:

  • jhipster-elasticsearch (simple wrapper over a specific version of the official image)
  • jhipster-logstash (same but with a logstash.conf file already included so it can be deployed easily see #34 )
  • jhipster-alerting or jhipster-console-alerting or jhipster-elastalert

I would like to introduce this change as a way to simplify development and maintenance of the console. People won't have to know which version of logstash, elasticsearch to use with the console, they can just use the same version for everything.

Basically, we would need to wrap all official images in our own, I can find many reasons why this would be good :

  • Only one version to know to install the stack, so this will simplify the docs.
  • You can omit the version number and it will latest and it will work (this is not possible currently)
  • It would be easier to keep track of which versions of those images works well together (if wanting to use older versions).
  • (optional) We will be able to move to alpine even before the official images. This is not so hard to do (see kost/docker-alpine/blob/master/alpine-elk/Dockerfile)
  • It would be easier for people to customize the console after doing a git clone. So it would be easier for them to contribute.

The drawbacks:

  • We multiply the number of images
  • our images are not "official" anymore (not really a problem as we are open source and you can just fork us)

jhipster-console not working "out-of-the-box"

Hi,

I tried to recreate jhipster-console installation and I'm not able to get it up and running as described in the documentation: jhipster monitoring

So before starting I removed all the containers and images related to jhipster from my system including elastic, kibana, zipkin, etc. usin docker rm and docker rmi commands.
Then I downloaded latest bootstrap docker-compose.yml using instructions from the documentation page:
curl -O https://raw.githubusercontent.com/jhipster/jhipster-console/master/bootstrap/docker-compose.yml
I changed logstash port (since I already use 5000 on my docker network) in the docker-compose.yml and limited port binding only to localhost:

# Production ready docker configuration
version: '2'
services:
    jhipster-elasticsearch:
        image: jhipster/jhipster-elasticsearch
        ports:
            - 127.0.0.1:9200:9200
            - 127.0.0.1:9300:9300
        # Uncomment this as well as the volume section down below
        # to have elasticsearch data persisted to a volume
        # you will need to create a named volume with `docker volume create log-data`
        volumes:
           - log-data:/usr/share/elasticsearch/data

    jhipster-logstash:
        image: jhipster/jhipster-logstash
        environment:
            - INPUT_TCP_PORT=6000
            - INPUT_UDP_PORT=6000
            - ELASTICSEARCH_HOST=jhipster-elasticsearch
            - ELASTICSEARCH_PORT=9200
            - LOGSTASH_DEBUG=false
        ports:
            - 127.0.0.1:6000:6000
            - 127.0.0.1:6000:6000/udp
        # Uncomment this section and add a logstash.conf file in log-conf/
        # to have logstash config loaded from a volume
        #volumes:
        #    - ./log-conf/:/usr/share/logstash/pipeline/

    jhipster-console:
        image: jhipster/jhipster-console
        ports:
            - 127.0.0.1:5601:5601
        environment:
            - ELASTICSEARCH_URL=http://jhipster-elasticsearch:9200

    jhipster-import-dashboards:
        image: jhipster/jhipster-import-dashboards
        environment:
            - ELASTICSEARCH_URL=http://jhipster-elasticsearch:9200
        depends_on:
            - jhipster-elasticsearch

    jhipster-curator:
        image: jhipster/jhipster-curator
        environment:
            - ES_HOST=jhipster-elasticsearch
            - UNIT_COUNT=7
            - UNIT=days

#    jhipster-alerter:
#        image: jhipster/jhipster-alerter
#        #volumes:
#        #    - ../jhipster-alerter/rules/:/opt/elastalert/rules/
#        #    - ../alerts/config.yaml:/opt/elastalert/config.yaml

# Uncomment to use the log-data volume
volumes:
    log-data:
        driver: local

networks:
    default:
        external:
            name: docker_default

Before starting services I created a log-data docker volume to persist data: docker volume create log-data.
I started the services with docker-compose up -d with the following output:

[user@server jhipster-console]$ docker-compose up -d
Pulling jhipster-elasticsearch (jhipster/jhipster-elasticsearch:latest)...
latest: Pulling from jhipster/jhipster-elasticsearch
e6e5bfbc38e5: Pull complete
8d46f83cd316: Pull complete
145b1d53df05: Pull complete
715fc3bc39a0: Pull complete
85627ecfd2ec: Pull complete
c7d6223729c4: Pull complete
624375db3ae0: Pull complete
48610d65e184: Pull complete
92837f745764: Pull complete
5a47382f3275: Pull complete
0fa7524124f6: Pull complete
Digest: sha256:6fa90b0e28888e3c430b5f69aba83ec8619d36c75cf0476e5b6a25fcc927adb1
Status: Downloaded newer image for jhipster/jhipster-elasticsearch:latest
Pulling jhipster-logstash (jhipster/jhipster-logstash:latest)...
latest: Pulling from jhipster/jhipster-logstash
e6e5bfbc38e5: Already exists
59d1556260fa: Pull complete
91d5266b6948: Pull complete
50b209a6f738: Pull complete
16e5c03f89fa: Pull complete
20e8ad9090bd: Pull complete
0098debe6d27: Pull complete
42304a1347ba: Pull complete
56f1a64a5224: Pull complete
ed6f19558c13: Pull complete
0ba3a51197f4: Pull complete
2cc131675c2e: Pull complete
5a03ea211320: Pull complete
Digest: sha256:0959747545b42b46d2a4bd61bd07cd8f02ffaf748c494da1c8505d1287ebb909
Status: Downloaded newer image for jhipster/jhipster-logstash:latest
Pulling jhipster-curator (jhipster/jhipster-curator:latest)...
latest: Pulling from jhipster/jhipster-curator
90f4dba627d6: Pull complete
19bc0bb0be9f: Pull complete
1a1ae718c97d: Pull complete
f7057ce97169: Pull complete
5f03793b51da: Pull complete
b00fd3c303f4: Pull complete
1c49ff5a7cfd: Pull complete
10e6b005807f: Pull complete
Digest: sha256:23c8104a929dad88bec89f7a5cf5d0058489f1d717b485b6cb312ae6157f821c
Status: Downloaded newer image for jhipster/jhipster-curator:latest
Pulling jhipster-console (jhipster/jhipster-console:latest)...
latest: Pulling from jhipster/jhipster-console
e6e5bfbc38e5: Already exists
3dae5b623dcc: Pull complete
3bfa3fa49f04: Pull complete
e8252b3f6391: Pull complete
a05e5e7cd335: Pull complete
539914b1aae0: Pull complete
4ee696c5d852: Pull complete
bbba454322a3: Pull complete
3fa7a13ec52e: Pull complete
e17da997b2b5: Pull complete
41641300cc0b: Pull complete
af7b9af71de2: Pull complete
Digest: sha256:20e9198df3e00ff539409b60db5eeb1e000947d226f0b30b9d67462ecb33c509
Status: Downloaded newer image for jhipster/jhipster-console:latest
Pulling jhipster-import-dashboards (jhipster/jhipster-import-dashboards:latest)...
latest: Pulling from jhipster/jhipster-import-dashboards
e6e5bfbc38e5: Already exists
ad0b5633606d: Pull complete
30fedc8fe87d: Pull complete
cf9b242aebe1: Pull complete
9b5b6af4bc79: Pull complete
2d4bcefc8e17: Pull complete
d1c8ab80c686: Pull complete
c9d410b21ef7: Pull complete
f65ecb8348ac: Pull complete
Digest: sha256:bee52f5c2df18f8840835b7e93f80e6aa96357a709aea831e41be648d988ca84
Status: Downloaded newer image for jhipster/jhipster-import-dashboards:latest
Creating jhipsterconsole_jhipster-logstash_1 ... 
Creating jhipsterconsole_jhipster-elasticsearch_1 ... 
Creating jhipsterconsole_jhipster-curator_1 ... 
Creating jhipsterconsole_jhipster-console_1 ... 
Creating jhipsterconsole_jhipster-logstash_1
Creating jhipsterconsole_jhipster-elasticsearch_1
Creating jhipsterconsole_jhipster-curator_1
Creating jhipsterconsole_jhipster-console_1 ... done
Creating jhipsterconsole_jhipster-import-dashboards_1 ... 
Creating jhipsterconsole_jhipster-import-dashboards_1 ... done
[user@server jhipster-console]$ 
[user@server jhipster-console]$ 
[user@server jhipster-console]$ 
[user@server jhipster-console]$ docker-compose ps
                    Name                                  Command               State                                   Ports                                  
--------------------------------------------------------------------------------------------------------------------------------------------------------------
jhipsterconsole_jhipster-console_1             /bin/sh -c /usr/local/bin/ ...   Up      127.0.0.1:5601->5601/tcp                                               
jhipsterconsole_jhipster-curator_1             /usr/sbin/crond -f               Up                                                                             
jhipsterconsole_jhipster-elasticsearch_1       /bin/bash bin/es-docker          Up      127.0.0.1:9200->9200/tcp, 127.0.0.1:9300->9300/tcp                     
jhipsterconsole_jhipster-import-dashboards_1   /bin/sh -c ./wait-for-elas ...   Up                                                                             
jhipsterconsole_jhipster-logstash_1            /usr/local/bin/docker-entr ...   Up      5044/tcp, 127.0.0.1:6000->6000/tcp, 127.0.0.1:6000->6000/udp, 9600/tcp 

Then I accessed the jhipster console and this is the screen I'm getting:
screenshot from 2017-08-22 10-57-40
and after selecting logstash-* from the drop down I get:
screenshot from 2017-08-22 10-57-57

Then I tried to do dashboard import once again:

[user@server jhipster-console]$ docker-compose up jhipster-import-dashboards
jhipsterconsole_jhipster-elasticsearch_1 is up-to-date
jhipsterconsole_jhipster-console_1 is up-to-date
Starting jhipsterconsole_jhipster-import-dashboards_1 ... 
Starting jhipsterconsole_jhipster-import-dashboards_1 ... done
Attaching to jhipsterconsole_jhipster-import-dashboards_1
jhipster-import-dashboards_1  | Waiting for Elasticsearch to startup (max 5min)
jhipster-import-dashboards_1  | {"cluster_name":"docker-cluster","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.0}Importing dashboards
jhipster-import-dashboards_1  | Creating index patterns if not present
jhipster-import-dashboards_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
jhipster-import-dashboards_1  |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{"_index":".kibana","_type":"index-pattern","_id":"logstash-*","_version":7,"result":"updated","100   215  100   159  100    56   5075   1787 --:--:-- --:--:-- --:--:--  5129
jhipster-import-dashboards_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
jhipster-import-dashboards_1  |                                  Dload  Upload   Total   Spent    Left  Speed
100    91  100    60  100    31   7310   3777 --:--:-- --:--:-- --:--:--  8571No handler found for uri [/.kibana/config/] and method [PUT]
jhipster-import-dashboards_1  | Import directory /tmp/index-pattern
jhipster-import-dashboards_1  | Import index to /.kibana/index-pattern/logstash-* from /tmp/index-pattern/logstash.json
jhipster-import-dashboards_1  | 
jhipster-import-dashboards_1  | Import directory /tmp/dashboard
jhipster-import-dashboards_1  | Import dashboard /tmp/dashboard/ehcache-dashboard.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/ehcache-hits.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/ehcache-misses.json
jhipster-import-dashboards_1  | Import dashboard /tmp/dashboard/jvm-dashboard.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/log-count.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/jvm-total-memory.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/jvm-heap-memory.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/jvm-threads.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/jvm-non-heap-memory.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/jvm-memory-average.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/jvm-memory-ratio.json
jhipster-import-dashboards_1  | Import dashboard /tmp/dashboard/logs-dashboard.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/logs-levels.json
jhipster-import-dashboards_1  | Import search /tmp/search/Stacktraces.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/Log-forwarding-instructions.json
jhipster-import-dashboards_1  | Import search /tmp/search/All-logs.json
jhipster-import-dashboards_1  | Import dashboard /tmp/dashboard/metrics-dashboard.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/request-rate.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/request-rate-for-resources.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/metrics-timer.json
jhipster-import-dashboards_1  | Import dashboard /tmp/dashboard/microservices-dashboard.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/instances.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/applications.json
jhipster-import-dashboards_1  | Import dashboard /tmp/dashboard/performance-dashboard.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/request-rate.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/request-rate-for-resources.json
jhipster-import-dashboards_1  | Import visualization /tmp/visualization/request-rate-per-resource-per-app.json
jhipsterconsole_jhipster-import-dashboards_1 exited with code 0
[user@server jhipster-console]$

What I can see out of this last output that kibana is not properly configured:

jhipster-import-dashboards_1  | Creating index patterns if not present
jhipster-import-dashboards_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
jhipster-import-dashboards_1  |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{"_index":".kibana","_type":"index-pattern","_id":"logstash-*","_version":7,"result":"updated","100   215  100   159  100    56   5075   1787 --:--:-- --:--:-- --:--:--  5129
jhipster-import-dashboards_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
jhipster-import-dashboards_1  |                                  Dload  Upload   Total   Spent    Left  Speed
100    91  100    60  100    31   7310   3777 --:--:-- --:--:-- --:--:--  8571No handler found for uri [/.kibana/config/] and method [PUT]
jhipster-import-dashboards_1  | Import directory /tmp/index-pattern
jhipster-import-dashboards_1  | Import index to /.kibana/index-pattern/logstash-* from /tmp/index-pattern/logstash.json

Is there something missing from the instructions on the monitoring page?

logstash service not starting

Hi,

I'm trying to run jhipster-console to be able to attach a monolith application I'm working on to it for logging and monitoring.
I cloned console repository with git clone https://github.com/jhipster/jhipster-console.git and then I started the services with cd jhipster-console && docker-compose up -d and after a minute or so I noticed that the logstash service stoppped:

✔ ~/projects/jhipster-console [master L|✔] 
08:30 $ docker-compose ps
                    Name                                  Command               State                 Ports               
-------------------------------------------------------------------------------------------------------------------------
jhipsterconsole_jhipster-console_1             /bin/sh -c /usr/local/bin/ ...   Up       0.0.0.0:5601->5601/tcp           
jhipsterconsole_jhipster-elasticsearch_1       /bin/bash bin/es-docker          Up       0.0.0.0:9200->9200/tcp, 9300/tcp 
jhipsterconsole_jhipster-import-dashboards_1   /bin/sh -c /usr/share/metr ...   Exit 0                                    
jhipsterconsole_jhipster-logstash_1            /usr/local/bin/docker-entr ...   Exit 1 

Then I went to see the log for the logstash service and this is the output:

✔ ~/projects/jhipster-console [master L|✔] 
08:40 $ docker-compose logs jhipster-logstash 
Attaching to jhipsterconsole_jhipster-logstash_1
jhipster-logstash_1           | Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
jhipster-logstash_1           | [2017-08-17T08:40:37,076][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
jhipster-logstash_1           | ERROR: No configuration file was specified. Perhaps you forgot to provide the '-f yourlogstash.conf' flag?
jhipster-logstash_1           | usage:
jhipster-logstash_1           |   bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
jhipster-logstash_1           |   bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
jhipster-logstash_1           |   bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
jhipster-logstash_1           |   bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
jhipster-logstash_1           |   bin/logstash --help

What did I miss? Is this a bug or am I doing something wrong?

Merge JHipster-Console and Elastalert Containers

I have been evaluating Elastalert as an alerting solution and it is working really well. I have made an alerting dashboard for Kibana that display recent alerts.

alerting-dashboard

So now I want to include it in the main image so that users don't have to deal with two different containers.

[PR]added Elasticsearch Curator

Elasticsearch Curator helps you curate, or manage, your Elasticsearch indices and snapshots by:

Obtaining the full list of indices (or snapshots) from the cluster, as the actionable list
Iterate through a list of user-defined filters to progressively remove indices (or snapshots) from this actionable list as needed.
Perform various actions on the items which remain in the actionable list.
PR#64

zipkin does not work

JHipster Version(s)
JHipster configuration, a .yo-rc.json file generated in the root folder
{
  "generator-jhipster": {
    "jhipsterVersion": "4.4.1",
    "baseName": "xx",
    "packageName": "xx",
    "packageFolder": "xx",
    "serverPort": "8082",
    "authenticationType": "uaa",
    "uaaBaseName": "uaa",
    "hibernateCache": "hazelcast",
    "clusteredHttpSession": false,
    "websocket": false,
    "databaseType": "sql",
    "devDatabaseType": "h2Disk",
    "prodDatabaseType": "mysql",
    "searchEngine": false,
    "messageBroker": false,
    "serviceDiscoveryType": "eureka",
    "buildTool": "maven",
    "enableSocialSignIn": false,
    "jwtSecretKey": "xx",
    "enableTranslation": true,
    "applicationType": "microservice",
    "testFrameworks": [
      "gatling"
    ],
    "jhiPrefix": "jhi",
    "skipClient": true,
    "skipUserManagement": true,
    "nativeLanguage": "zh-cn",
    "languages": [
      "zh-cn"
    ],
    "clientPackageManager": "yarn"
  }
}
Browsers and Operating System

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

git version 2.11.0.windows.3

node: v7.7.2

npm: 4.1.2

yeoman: 1.8.5

yarn: 0.23.4

docker-compose.yml

version: '2'
services:
    
    gateway-app:
        image: gateway
        environment:
            - SPRING_PROFILES_ACTIVE=prod,swagger
            - EUREKA_CLIENT_SERVICE-URL_DEFAULTZONE=http://admin:$${jhipster.registry.password}@jhipster-registry:8761/eureka
            - SPRING_CLOUD_CONFIG_URI=http://admin:$${jhipster.registry.password}@jhipster-registry:8761/config
            - SPRING_DATASOURCE_URL=jdbc:mysql://gateway-mysql:3306/gateway?useUnicode=true&characterEncoding=utf8&useSSL=false
            - SPRING_DATA_CASSANDRA_CONTACTPOINTS=gateway-cassandra
            - JHIPSTER_SLEEP=80
            - JHIPSTER_REGISTRY_PASSWORD=admin
        ports:
            - 8080:8080
    gateway-mysql:
        image: mysql:5.7.13
        environment:
            - MYSQL_USER=root
            - MYSQL_ALLOW_EMPTY_PASSWORD=yes
            - MYSQL_DATABASE=gateway
        command: mysqld --lower_case_table_names=1 --skip-ssl --character_set_server=utf8
    
    xx-app:
        image: xx
        environment:
            - SPRING_PROFILES_ACTIVE=prod,swagger,zipkin
            - EUREKA_CLIENT_SERVICE-URL_DEFAULTZONE=http://admin:$${jhipster.registry.password}@jhipster-registry:8761/eureka
            - SPRING_CLOUD_CONFIG_URI=http://admin:$${jhipster.registry.password}@jhipster-registry:8761/config
            - SPRING_DATASOURCE_URL=jdbc:mysql://xx-mysql:3306/xx?useUnicode=true&characterEncoding=utf8&useSSL=false
            - JHIPSTER_SLEEP=30
            - JHIPSTER_REGISTRY_PASSWORD=admin
    xx-mysql:
        image: mysql:5.7.13
        environment:
            - MYSQL_USER=root
            - MYSQL_ALLOW_EMPTY_PASSWORD=yes
            - MYSQL_DATABASE=xx
        command: mysqld --lower_case_table_names=1 --skip-ssl --character_set_server=utf8
    
    uaa-app:
        image: uaa
        environment:
            - SPRING_PROFILES_ACTIVE=prod,swagger
            - EUREKA_CLIENT_SERVICE-URL_DEFAULTZONE=http://admin:$${jhipster.registry.password}@jhipster-registry:8761/eureka
            - SPRING_CLOUD_CONFIG_URI=http://admin:$${jhipster.registry.password}@jhipster-registry:8761/config
            - SPRING_DATASOURCE_URL=jdbc:mysql://uaa-mysql:3306/uaa?useUnicode=true&characterEncoding=utf8&useSSL=false
            - JHIPSTER_SLEEP=30
            - JHIPSTER_REGISTRY_PASSWORD=admin
    uaa-mysql:
        image: mysql:5.7.13
        environment:
            - MYSQL_USER=root
            - MYSQL_ALLOW_EMPTY_PASSWORD=yes
            - MYSQL_DATABASE=uaa
        command: mysqld --lower_case_table_names=1 --skip-ssl --character_set_server=utf8
    
    
    jhipster-registry:
        image: jhipster/jhipster-registry:v3.0.0
        volumes:
            - ./central-server-config:/central-config
        # By default the JHipster Registry runs with the "prod" and "native"
        # Spring profiles.
        # "native" profile means the filesystem is used to store data, see
        # http://cloud.spring.io/spring-cloud-config/spring-cloud-config.html
        environment:
            - SPRING_PROFILES_ACTIVE=dev,native
            - SECURITY_USER_PASSWORD=admin
            - JHIPSTER_REGISTRY_PASSWORD=admin
            - JHIPSTER_LOGGING_LOGSTASH_ENABLED=true
            - JHIPSTER_LOGGING_LOGSTASH_HOST=jhipster-logstash
            - JHIPSTER_METRICS_LOGS_ENABLED=true
            - JHIPSTER_METRICS_LOGS_REPORTFREQUENCY=60
            # - GIT_URI=https://github.com/jhipster/jhipster-registry/
            # - GIT_SEARCH_PATHS=central-config
        ports:
            - 8761:8761


    jhipster-elasticsearch:
        image: jhipster/jhipster-elasticsearch:v2.0.1
        ports:
            - 9200:9200
            - 9300:9300
        # Uncomment this section to have elasticsearch data persisted to a volume
        #volumes:
        #   - ./log-data:/usr/share/elasticsearch/data
    jhipster-logstash:
        image: jhipster/jhipster-logstash:v2.0.1
        command: logstash -f /conf/logstash.conf
        ports:
            - 5000:5000/udp
        # Uncomment this section to have logstash config loaded from a volume
        #volumes:
        #    - ./log-conf/:/conf
    jhipster-console:
        image: jhipster/jhipster-console:v2.0.1
        ports:
            - 5601:5601
    jhipster-zipkin:
        image: jhipster/jhipster-zipkin:v2.0.1
        ports:
            - 9411:9411
        environment:
            - ES_HOSTS=http://jhipster-elasticsearch:9200
            - ZIPKIN_UI_LOGS_URL=http://localhost:5601/app/kibana#/dashboard/logs-dashboard?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-1h,mode:quick,to:now))&_a=(filters:!(),options:(darkTheme:!f),panels:!((col:1,id:logs-levels,panelIndex:2,row:1,size_x:6,size_y:3,type:visualization),(col:7,columns:!(stack_trace),id:Stacktraces,panelIndex:7,row:1,size_x:4,size_y:3,sort:!('@timestamp',desc),type:search),(col:11,id:Log-forwarding-instructions,panelIndex:8,row:1,size_x:2,size_y:3,type:visualization),(col:1,columns:!(app_name,level,message),id:All-logs,panelIndex:9,row:4,size_x:12,size_y:7,sort:!('@timestamp',asc),type:search)),query:(query_string:(analyze_wildcard:!t,query:'{traceId}')),title:logs-dashboard,uiState:())

application-prod.yml

# ...
spring: 
    zipkin: # Use the "zipkin" Maven profile to have the Spring Cloud Zipkin dependencies
        base-url: http://jhipster-zipkin:9411
        enabled: true
        locator:
            discovery:
                enabled: true
# ...
#build docker image
./mvnw package -Pprod,zipkin docker:build

#run the docker services
docker-compose up -d

snipaste20170516_112436

the jhipster-zipkin env ${ZIPKIN_UI_LOGS_URL} use http://localhost:5601,i changed to http://jhipster-console:5601 but it was still invalid.
snipaste20170516_113741

curl -Xget "http://jhipster-elasticsearch:9200/_search?q=traceId&pretty"
# output
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 10,
    "successful" : 10,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

When I Start Console Show 'start ES failed'

when I start jhipster-console by docker use 'docker-componse up' show this exception:

jhipster-elasticsearch_1  | [2016-12-25T05:55:46,338][INFO ][o.e.b.BootstrapCheck     ] [Z_PTvLl] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
jhipster-elasticsearch_1  | ERROR: bootstrap checks failed

who can tell me reason,thanks~

Alerts Count visualization problem

Hi,
I am experimenting with your console and alterer, good stuff, but I think I found a problem:

On alerting dashboard the Alerts Count visualization does not work:
"A field associated with this object no longer exists in the index pattern."

It seems to be the match_body.app_name which is used in a terms aggregation.
But if I check the elastalert_status mapping in ES for match_body:
"match_body": {
"type": "object",
"enabled": false
}

From ES documentation:
"The enabled flag allows to disable parsing and indexing a named object completely. This is handy when a portion of the JSON document contains arbitrary JSON which should not be indexed, nor added to the mapping."

Meaning that building aggregations on match_body does not work unless you recreate the elastalert_status index and change the enable flag. But I am not sure if it is a good idea because match_body contains the whole original document that matched including the message field that can contain any arbitrary text like exceptions without any limitation. So it could blow the index.

Please, correct me if I misinterpret anything.
Thanks,
Szabolcs

Index patterns fields are not always getting created by Kibana

It causes dashboards to display errors when displayed started.

This can be solved in some cases by refreshing the index pattern so I added a tip to the docs but it is not enough in all cases.

Should we configure logstash differently to solve this problem ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.