flant / loghouse Goto Github PK
View Code? Open in Web Editor NEWReady to use log management solution for Kubernetes storing data in ClickHouse and providing web UI.
License: Apache License 2.0
Ready to use log management solution for Kubernetes storing data in ClickHouse and providing web UI.
License: Apache License 2.0
When user opens dashboard, follow mode should be enabled by default.
If you enter an incorrect request, you get the error Internal server Error and in loghouse pod see logs:
2017-11-04 07:27:14 - LoghouseQuery::BadFormat - logs: Failed to match sequence (EXPRESSION subquery:(SUBQUERY?)) at line 1 char 1.:
I have following in logs when deploy loghouse via helm:
Code: 117. DB::Exception: Unknown field found while parsing JSONEachRow format: _SELINUX_CONTEXT
2017-11-08 05:28:55 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-11-08 05:28:55 +0000 error_class="RuntimeError" error="command returns 29952: bash /usr/local/bin/insert_ch.sh /tmp/fluent-plugin-exec-20171108-7-kfyf2o" plugin_id="object:3fce70caaa48"
2017-11-08 05:28:55 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.12.40/lib/fluent/plugin/out_exec.rb:104:in `write'
2017-11-08 05:28:55 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.12.40/lib/fluent/buffer.rb:354:in `write_chunk'
2017-11-08 05:28:55 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.12.40/lib/fluent/buffer.rb:333:in `pop'
2017-11-08 05:28:55 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.12.40/lib/fluent/output.rb:342:in `try_flush'
2017-11-08 05:28:55 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.12.40/lib/fluent/output.rb:149:in `run'
Code: 117. DB::Exception: Unknown field found while parsing JSONEachRow format: _SELINUX_CONTEXT
2017-11-08 05:28:55 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-11-08 05:28:57 +0000 error_class="RuntimeError" error="command returns 29952: bash /usr/local/bin/insert_ch.sh /tmp/fluent-plugin-exec-20171108-7-fm3rdj" plugin_id="object:3fce70caaa48"
2017-11-08 05:28:55 +0000 [warn]: suppressed same stacktrace
Code: 117. DB::Exception: Unknown field found while parsing JSONEachRow format: _SELINUX_CONTEXT
2017-11-08 05:28:57 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-11-08 05:29:01 +0000 error_class="RuntimeError" error="command returns 29952: bash /usr/local/bin/insert_ch.sh /tmp/fluent-plugin-exec-20171108-7-1tfk6fi" plugin_id="object:3fce70caaa48"
What with it could be connected?
Sometimes we need to search some logs, that does not match some regex, but re2 does not have simple solution out of the box (google/re2#156). New operator !~
would be simple and powerfull solution for this situation.
loghouse/charts/loghouse/values.yaml
Line 86 in beb2603
Probably meant to be 512Mi
Is there manual for using without docker + kubernetes ?
I wan't to use it in kvm so I need some manual =)
I am using Kubernetes 1.8.1 and when I use helm to deploy this, the ingress never gets a public ip endpoint and has no errors to speak of (nothing in events etc) any ideas?
An ability to save a query by user. Admin can save public queries.
Queries can be saved in configmap.
Ctrl-click on label should open a new tab.
Filter should be copied with clicked label added.
the current implementation is only for kubernetes?
or in docker-compose can I also run?
Filter might contain an exact match for label. In this case, this label should be hidden automatically.
I'm on Kubernetes ver. 1.8.3 running on my own server.
I'm getting the following error:
root@db2new:~/kube# helm install -n loghouse loghouse
Error: release loghouse failed: Timeout: request did not complete within allowed duration
root@db2new:~/kube#
Tiller logs:
[storage] 2017/11/27 04:08:34 getting release history for "loghouse"
[tiller] 2017/11/27 04:08:34 uninstall: Release not loaded: loghouse
[tiller] 2017/11/27 04:08:41 preparing install for loghouse
[storage] 2017/11/27 04:08:41 getting release history for "loghouse"
[tiller] 2017/11/27 04:08:41 rendering loghouse chart using values
2017/11/27 04:08:41 info: manifest "loghouse/templates/tabix/tabix.yaml" is empty. Skipping.
2017/11/27 04:08:41 info: manifest "loghouse/templates/clickhouse/clickhouse-pvc.yaml" is empty. Skipping.
2017/11/27 04:08:41 info: manifest "loghouse/templates/tabix/tabix-svc.yaml" is empty. Skipping.
2017/11/27 04:08:41 info: manifest "loghouse/templates/clickhouse/clickhouse-ingress.yaml" is empty. Skipping.
2017/11/27 04:08:41 info: manifest "loghouse/templates/tabix/tabix-ingress.yaml" is empty. Skipping.
[tiller] 2017/11/27 04:08:42 performing install for loghouse
[tiller] 2017/11/27 04:08:42 executing 2 pre-install hooks for loghouse
[tiller] 2017/11/27 04:08:42 hooks complete for pre-install loghouse
[storage] 2017/11/27 04:08:42 getting release history for "loghouse"
[storage] 2017/11/27 04:08:42 creating release "loghouse.v1"
[kube] 2017/11/27 04:08:42 building resources from manifest
[kube] 2017/11/27 04:08:42 creating 19 resource(s)
[tiller] 2017/11/27 04:09:13 warning: Release "loghouse" failed: Timeout: request did not complete within allowed duration
[storage] 2017/11/27 04:09:13 updating release "loghouse.v1"
[tiller] 2017/11/27 04:09:13 failed install perform step: release loghouse failed: Timeout: request did not complete within allowed duration
It turns out first installation to a fresh k8s node always succeeds, but all subsequent ones (after helm del --purge loghouse
) fail with timeout.
After the filter is changed and some columns are hidden (manually), user should have an ability to restore columns to their “auto” state, i.e. display all the columns with an exception for automatically hidden.
Errot, while install in kubernetes 1.8.
Note: CronJob resource in batch/v2alpha1 API group has been deprecated starting from cluster version 1.8. You should switch to using batch/v1beta1, instead, which is enabled by default in the API server. Further in this document, we will be using batch/v1beta1 in all the examples.
Please, make changes for current kubernetes version.
There are Docker logs, kernel logs, and others having no namespace and no pod_name label, so it’s difficult to distinguish user permissions for these records and to filter them. We need to find a solution.
[kube] root@kube ~ # vi loghouse/values.yaml
[kube] root@kube ~ # helm upgrade --namespace loghouse loghouse loghouse/
Error: UPGRADE FAILED: jobs.batch "loghouse-init-tables" already exists
[kube] root@kube ~ # kubectl -n loghouse delete jobs/loghouse-init-tables
job "loghouse-init-tables" deleted
[kube] root@kube ~ # kubectl -n loghouse delete jobs/loghouse-init-tables
job "loghouse-init-tables" deleted
[kube] root@kube ~ # helm upgrade --namespace loghouse loghouse loghouse/
Release "loghouse" has been upgraded. Happy Helming!
LAST DEPLOYED: Fri Dec 8 14:18:30 2017
Log records are currently displayed as a list of strings. Table view having labels as columns should be useful too.
If you do not add the user to the configmap loghouse, then when you try to login, we get Internal Server Error with an error:
95.213.149.180, 95.213.149.180 - - [04/Nov/2017:07:14:51 +0000] "GET /query HTTP/1.0" 500 30 0.0010 2017-11-04 07:14:52 - RuntimeError - no user permissions configured for user
Sometimes need to use global negation in queries like not (expr1 and expr2)
The Google StackDriver interface allows distinguishing normal log lines from errors visually with ease. Please improve UI to allow the same.
Fluentd sends records every second. When loghouse gets records for one particular second fluentd can be still sending them.
We can resolve this by requesting records for 2 seconds (more precisely, for 2 fluentd sending periods)/
Usually we know only approximate time when something happened so we just want to enter that time and see logs around it.
From-to approach should only be left on downloading logs for needed period.
To resolve outgaes in okmeter it is necessary to change nginx log format of tabix.ui.
We need to add $request_time and $upstream_response_time fields to log format.
Message from okmeter:
log_format "combined" missing variables: "$request_time" "$upstream_response_time"
Backend is a small Ruby app.
It is better to follow Kubernetes community and use Golang.
Since rotation deletes old log records, making their upload to S3 available on schedule will be great.
Прислал МР который добавляет авто-генерацию хеша пароля для ClickHouse
Query language improvements:
SPA (single-page application) has no page switching and it’s more responsive (all the data is loaded via AJAX). Query saving should be in a separate view or dialog (e.g. graph editor in Grafana).
Angular is preferable since it’s used in Grafana and kubernetes-dashboard. Angular+React (like in grafana) may be another option.
I would like to have an ability to specify what keys should be shown in query template.
With this feature I will be able to specify list of hidden keys only one time - when I'm creating query template. And every time when I'll open this template, I will only see keys that I am needed.
Wrong query language definitions (https://github.com/flant/loghouse/blob/master/docs/ru/query-language.md)
This query
SELECT string_fields.names,
string_fields.values
FROM
logs.logsWHERE
date=today() AND string_fields.values = '1'Order by timestamp
LIMIT 100
throw
Code: 43, e.displayText() = DB::Exception: Illegal types of arguments (Array(String), String) of function equals, e.what() = DB::Exception
Images:
flant/loghouse-clickhouse:0.0.2
flant/loghouse-fluentd:0.0.2
flant/loghouse-tabix:0.0.2
When user saves a query, displayed labels should be also saved.
I'm trying to install loghouse to Google Cloud-bases Kubernetes cluster (node pool version is 1.8.3-gke.0).
I'm getting the following error:
➜ cm-scripts git:(master) ✗ helm install -n loghouse loghouse
Error: release loghouse failed: clusterroles.rbac.authorization.k8s.io "fluentd" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]}] user=&{system:serviceaccount:kube-system:default 3d69e5b6-d03a-11e7-9bc4-42010af00233 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
➜ cm-scripts git:(master) ✗
There should be the only select filled with the labels currently displayed. Hidden labels which are available should be displayed in autocomplete list. If all labels are displayed, autocomplete list will be empty. “Clear” button resetting labels to their default state should be available.
Currently this default state is simply “display everything” but in future it should be improved via special algorithm hiding labels depending on a filter query.
In our setup, we have several nodes with ~8Gb RAM. Clickhouse eats all of memory and not give it back. So clickhouse memory usage grows in expluatation period. And when it's more than node can give, node crashes with OutOfMemory error.
Is there some option to reduse memory usage of clickhouse service
loghouse/images/fluentd/insert_ch.sh
Line 4 in beb2603
INSERT INTO logs${TABLE}
не совпадает с настройками clickhouse по таблице, и пытается записать в не существующую таблицу.
Old logs should be cleaned from time to time. 3 options will be introduced for that purpose:
No clear decision yet.
Options available:
Hi,
I try to deploy Clickhouse in my Kubernetes cluster but Clickhouse pod crash at startup.
Here is pod's logs :
╰─# k -n loghouse logs -f clickhouse-server-79dc494dfc-sz268
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
2017.11.02 03:01:08.091964 [ 1 ] <Warning> Application: Logging to console
2017.11.02 03:01:08.103679 [ 1 ] <Warning> ConfigProcessor: Include not found: networks
2017.11.02 03:01:10.103799 [ 2 ] <Warning> ConfigProcessor: Include not found: clickhouse_remote_servers
2017.11.02 03:01:10.103848 [ 2 ] <Warning> ConfigProcessor: Include not found: clickhouse_compression
2017.11.02 03:01:10.107723 [ 3 ] <Warning> ConfigProcessor: Include not found: networks
I used the Helm chart :
name: loghouse
version: 0.0.1
And don't made any changes in templates/clickhouse/clickhouse-configmap.yml
Kubernetes v1.8.2
Calico CNI
IPv6 only (I made the changes to make it work about Service ClusterIP)
Debian 9.2
Hello!
I try build loghouse in RedHat-based, but get some error.
I view https://github.com/flant/loghouse/blob/master/Dockerfile and have a few question.
Is ruby:2.3.4 minimal version?
You are add 2 file and run 2 command
ADD Gemfile /app/Gemfile
ADD Gemfile.lock /app/Gemfile.lock
RUN bundle config git.allow_insecure true
RUN bundle install
how simple build without docker?
Thank you
Account:
https://habrahabr.ru/users/chemtech/
Email:
patsev(dot)anton(at)gmail(dot)com
May be talk at russian language
Download button must activate a background server process of archiving of resulting records, so user will be able to see a progress bar of that process. As archiving process is completed, a secret temporary link should become available for user. User can download a file by clicking on it or share it. Link should expire and archive should be deleted from the server in 2 days.
Details:
All clusters are different: bare metal, GCE, AWS, Azure… There should be several installation configurations:
Fluentd is feature rich while loghouse uses it as a log forwarder only. Small Go program will do this job better.
Low resource requirements and at the same time extraordinary performance are killer features that not described in the documentation at all!
Project needs its own logo and a favicon.ico.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.