microsoft / k2bridge Goto Github PK
View Code? Open in Web Editor NEWEnables Kibana to query Azure Data Explorer (ADX / Kusto)
License: Other
Enables Kibana to query Azure Data Explorer (ADX / Kusto)
License: Other
K2Bridge currently only supports Kibana 6. We want to support Kibana 7 (and Elasticsearch 7).
Identified gaps:
spike/kibana7
courier:batchSearches
option under Kibana Advanced settings, but that results in frequent UI errors related to the "timed_out" property. The cause for this seems to be that K2Bridge only returns results for the first request in an ndjson document.In addition:
spike/kibana7
Provide a solution to split an existing primary aggregation using another one (for instance, following terms aggregation is split again using month date histogram, so each carrier bucket provide a count metric for each month).
"aggs": {
"3": {
"terms": {
"field": "Carrier",
"order": {
"_key": "desc"
},
"size": 5
},
"aggs": {
"4": {
"date_histogram": {
"field": "timestamp",
"calendar_interval": "1M",
"time_zone": "Europe/Paris",
"min_doc_count": 1
}
}
}
}
},
The concept is to apply a new query on each individual bucket produced by Terms aggregation. Need some additional investigation but we can probably reuse the work done on TopHits with BuildPartitionQuery.
We're having the following exception. latest K2
System.InvalidCastException: Unable to cast object of type 'System.DBNull' to type 'System.SByte'.
at K2Bridge.KustoDAL.HitsMapper.<>c.<.cctor>b__8_0(Object value) in /app/K2Bridge/KustoDAL/HitsMapper.cs:line 28
at K2Bridge.KustoDAL.HitsMapper.ReadHit(DataRow row, QueryData query, LuceneHighlighter highlighter) in /app/K2Bridge/KustoDAL/HitsMapper.cs:line 73
at K2Bridge.KustoDAL.HitsMapper.<>c__DisplayClass3_0.<MapAndAnalyzeRows>b__0(DataRow row) in /app/K2Bridge/KustoDAL/HitsMapper.cs:line 53
at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
at K2Bridge.KustoDAL.KustoResponseParser.ReadResponse(QueryData query, IDataReader reader, TimeSpan timeTaken) in /app/K2Bridge/KustoDAL/KustoResponseParser.cs:line 181
at K2Bridge.KustoDAL.KustoResponseParser.Parse(IDataReader reader, QueryData queryData, TimeSpan timeTaken) in /app/K2Bridge/KustoDAL/KustoResponseParser.cs:line 102
Data may be sensitive, will send you by email.
A multi-value
metrics aggregation that calculates one or more percentile ranks over numeric values extracted from the aggregated documents. These values can be generated by a provided script or extracted from specific numeric or histogram fields in the documents.
Please see Percentiles are (usually) approximate and Compression for advice regarding approximation and memory use of the percentile ranks aggregation
Percentile rank show the percentage of observed values which are below certain value. For example, if a value is greater than or equal to 95% of the observed values it is said to be at the 95th percentile rank.
In the installation doc (https://github.com/microsoft/K2Bridge/blob/master/docs/installation.md), both of the helm commands use the switch -n k2bridge. The doc did NOT mention that such namespace should exist. One either has to create such namespace before running the helm commands or face failure.
Two questions:
An internal server error occurred
Error: An internal server error occurred
at http://20.85.26.253:5601/bundles/commons.bundle.js:1:1203245
at processQueue (http://20.85.26.253:5601/built_assets/dlls/vendors.bundle.dll.js:307:199687)
at http://20.85.26.253:5601/built_assets/dlls/vendors.bundle.dll.js:307:200650
at Scope.$digest (http://20.85.26.253:5601/built_assets/dlls/vendors.bundle.dll.js:307:210412)
at Scope.$apply (http://20.85.26.253:5601/built_assets/dlls/vendors.bundle.dll.js:307:213219)
at done (http://20.85.26.253:5601/built_assets/dlls/vendors.bundle.dll.js:307:132717)
at completeRequest (http://20.85.26.253:5601/built_assets/dlls/vendors.bundle.dll.js:307:136329)
at XMLHttpRequest.requestLoaded (http://20.85.26.253:5601/built_assets/dlls/vendors.bundle.dll.js:307:135225)
This bug is about the visualization part.
Reproduction:
see complete thread here:
#114 (comment)
Above TLS in ingress.yaml add input for ingressClassName: ${{ .Values.ingress.className }}
so ingress controller will pick up ingress and assign LB IP to it.
In pull request #134, metadata table had been introduced to fix missing buckets. I suggest to use the same table to move remaining metadata used in column names.
This suggestion will be applied on:
Is it possible to use K2Bridge to connect data from an existing Azure table (from a storage account) to an existing Kibana environment?
From what I have read so far and found online, it requires a new Azure Data Explorer cluster and database for K2Bridge work and as well a new Kibana environment.
Has anyone done the same setup I'm aiming to do?
Also, in the README.md it refers to supporting Kibana 7, does it support Kibana 8?
HI. Getting the syntax error here:
"query": {
"match": {
"CallerFilePath": {
"query": "D:\\BuildAgent\\work\\abc\\src\\abc\\Controllers\\ErrorLogApiController.cs",
"type": "phrase"
}
}
}
}
Discover: Syntax error: Query could not be parsed: I could not parse that, sorry.. Query: 'let fromUnixTimeMilli = (t:long) {datetime(1970 - 01 - 01) + t * 1millisec}; let _data = database("log").Cashhardware | where (CallerFilePath == "D:\BuildAgent\work\abc\src\abc\Controllers\ErrorLogApiController.cs") and (Timestamp >= fromUnixTimeMilli(1584037503141) and Timestamp <= fromUnixTimeMilli(1584642303141)); (_data | summarize count() by Timestamp = bin(Timestamp, 3h) | order by Timestamp asc | as aggs); (_data | order by Timestamp desc | limit 500 | as hits)'
It seems that it doesn't escape slashes in "D:\BuildAgent" filter.
Reference documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html
"An aggregation that returns interesting or unusual occurrences of terms in a set." In Kibana, it can only be applied to text fields.
According to the documentation, this aggregation is primarily used with a "parent-level aggregation to segment the data ready for analysis". In other terms, it is typically a sub-aggregation of another bucket aggregation, like Terms or a Histogram.
This makes this issue dependant on support for sub-bucket aggregations (#145).
Before implementing, we need to identify how to perform a similar query in Kusto. According to the doc, Significant Terms "are the terms that have undergone a significant change in popularity measured between a foreground and background set. [...] In the simplest case, the foreground set of interest is the search results matched by a query and the background set used for statistical comparisons is the index or indices from which the results were gathered."
Sample request:
"aggs": {
"2": {
"histogram": {
"field": "AvgTicketPrice",
"interval": 100,
"min_doc_count": 1
},
"aggs": {
"3": {
"significant_terms": {
"field": "DestCountry",
"size": 3
}
}
}
}
Response: (extract)
"aggregations": {
"2": {
"buckets": [
{
"3": {
"doc_count": 749,
"bg_count": 13059,
"buckets": [
{
"key": "IT",
"doc_count": 243,
"score": 0.2552994319244096,
"bg_count": 2371
},
{
"key": "US",
"doc_count": 172,
"score": 0.11694192970564077,
"bg_count": 1987
},
{
"key": "CH",
"doc_count": 79,
"score": 0.10476945913799716,
"bg_count": 691
}
]
},
"key": 100,
"doc_count": 749
},
{
"3": {
"doc_count": 1067,
"bg_count": 13059,
"buckets": [
{
"key": "IT",
"doc_count": 241,
"score": 0.0551183926043845,
"bg_count": 2371
},
{
"key": "CH",
"doc_count": 83,
"score": 0.03656787843506986,
"bg_count": 691
},
{
"key": "US",
"doc_count": 185,
"score": 0.02418926301801488,
"bg_count": 1987
}
]
},
"key": 200,
"doc_count": 1067
},
Searches such as field:>50 results in a generated KQL query:
(field == ">50")
In the Helm chart, configuration values such as aadClientId (and others) are set twice:
In deployment.yaml:
- name: aadClientId
value: "{{ .Values.settings.aadClientId }}"
In configmap.yaml:
aadClientId: "{{ .Values.settings.aadClientId }}"
I am using K2bridge with Client ID and Client Secret using collectTelemetry true and instrumentation key being set. The k2bridge is deployed in a nodepool with subnet from a private VNET. I need to understand if there are permissions needs to be given in order to collect telemetry.
We need to convert the Visitor class method to be async to support:
K2Bridge/Visitors/RangeClauseVisitor.cs
var t = ClauseFieldTypeProcessor.GetType(schemaRetriever, rangeClause.FieldName).Result;
text range searches such as : title:{Aida TO Carmen}
are not supported currently
Hi tried to follow the instructions in here
Line 96 in 683fb42
I run the following command with variables defined:
michael [ ~/K2Bridge ]$ helm install k2bridge charts/k2bridge --namespace k2bridge --set k2bridge.adx.url=$ADX_URL --set k2bridge.adx.database=$ADX_DATABASE --set k2bridge.adx.clientId=$ADX_CLIENT_ID --set k2bridge.adx.clientSecret=$ADX_CLIENT_SECRET --set k2bridge.adx.tenantId=$ADX_TENANT_ID
receive the following error message
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "k2bridgees-master-pdb" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
ensure CRDs are installed first
tried to follow the latest NOTES.txt comment and set values.yaml with maxUnavailable: 1 but it still doesn't work
running on AKS k8s version 1.29.7
Exists filter does not work with fields with special symbols in names.
We have index with column Payload.@l.
It works good, if we use it to filter rows by value, for example:
{
"query": {
"match_phrase": {
"Payload.@l": "Error"
}
}
}
But we get an error, if we try to filter rows based on this column existence:
{
"exists": {
"field": "Payload.@l"
}
}
Sample error message:
[0]Kusto.Data.Exceptions.SyntaxException: Syntax error: A recognition error occurred.. Query: 'let _data = database("log").InfraKustokibana | where (['Env'] == "dev-sre") and (['Timestamp'] >= todatetime("2022-01-17T08:13:05.1150000Z") and ['Timestamp'] <= todatetime("2022-01-19T08:13:05.1150000Z")) and (['Payload'].['@l'] == "Error") and (isnotnull(Payload.@l));
let _extdata = _data
| extend ['2'] = bin(['Timestamp'], 1h);
let _summarizablemetrics = _extdata
| summarize count() by ['2']
| order by ['2'] asc;
(_summarizablemetrics
| as aggs);
(_data | count | as hitsTotal);
(_data | order by ['Timestamp'] desc | limit 500 | as hits)'
Timestamp=2022-01-19T08:13:05.7408043Z
ClientRequestId=K2Bridge.ExecuteQuery;94de483b-bfaa-4638-aa69-2a16e608d0dc
ActivityId=7776d65d-9297-48b4-be3d-6e1095f30c1b
ActivityType=DN.RestClient.ExecuteQuery
MachineName=k2bridge-555bb5ffb4-8swdq
ProcessName=dotnet
ProcessId=1
ThreadId=311
AppDomainName=K2Bridge
ActivityStack=(Activity stack: CRID=K2Bridge.ExecuteQuery;94de483b-bfaa-4638-aa69-2a16e608d0dc ARID=7776d65d-9297-48b4-be3d-6e1095f30c1b > DN.RestClient.ExecuteQuery/7776d65d-9297-48b4-be3d-6e1095f30c1b)
MonitoredActivityContext=(ActivityType=DN.RestClient.ExecuteQuery, Timestamp=2022-01-19T08:13:05.2680582Z, ParentActivityId=7776d65d-9297-48b4-be3d-6e1095f30c1b, TimeSinceStarted=472.759 [ms])ErrorCode=SYN0002
ErrorReason=BadRequest
ErrorMessage=A recognition error occurred.
DataSource=https://<REDACTED>.westeurope.kusto.windows.net/v1/rest/query
DatabaseName=log
ClientRequestId=K2Bridge.ExecuteQuery;94de483b-bfaa-4638-aa69-2a16e608d0dc
ActivityId=e3241ef1-b0ff-4392-bb4f-00d6a1c6725e
Text=let _data = database("log").InfraKustokibana | where (['Env'] == "dev-sre") and (['Timestamp'] >= todatetime("2022-01-17T08:13:05.1150000Z") and ['Timestamp'] <= todatetime("2022-01-19T08:13:05.1150000Z")) and (['Payload'].['@l'] == "Error") and (isnotnull(Payload.@l));
let _extdata = _data
| extend ['2'] = bin(['Timestamp'], 1h);
let _summarizablemetrics = _extdata
| summarize count() by ['2']
| order by ['2'] asc;
(_summarizablemetrics
| as aggs);
(_data | count | as hitsTotal);
(_data | order by ['Timestamp'] desc | limit 500 | as hits)
ParseErrors=A recognition error occurred.
Line=1
CharacterPositionInLine=266
Token=l
at Kusto.Cloud.Platform.Http.KustoHttpClient.ThrowKustoExceptionFromResponseMessageAsync(KustoExceptionContext exceptionContext, HttpResponseMessage responseMessage, ClientRequestProperties properties, Boolean shouldBuffer, Action`2 notify)
at Kusto.Data.Net.Client.RestClient2.MakeHttpRequestAsyncImpl(String address, String csl, String ns, String databaseName, Boolean streaming, ClientRequestProperties properties, ServiceModelTimeoutKind timeoutKind, String clientRequestId, Stream body, StreamProperties streamProperties)
at Kusto.Cloud.Platform.Utils.MonitoredActivity.InvokeAsync[TActivityType,TResult](TActivityType activityType, Func`1 func, String clientRequestId)
at Kusto.Cloud.Platform.Utils.MonitoredActivity.InvokeAsync[TActivityType,TResult](TActivityType activityType, Func`1 func, String clientRequestId)
at Kusto.Data.Net.Client.RestClient2.MakeHttpRequestAsync(ActivityType activityType, String baseAddress, String relativeAddress, String clientRequestIdPrefix, String ns, String databaseName, String csl, String addr, Boolean streaming, ClientRequestProperties properties, ServiceModelTimeoutKind timeoutKind, StreamProperties streamProperties)
at Kusto.Data.Net.Client.RestClient2.ExecuteQueryAsync(String databaseName, String query, ClientRequestProperties properties)
at K2Bridge.KustoDAL.CslQueryProviderExtensions.ExecuteMonitoredQueryAsync(ICslQueryProvider client, String query, ClientRequestProperties clientRequestProperties, Metrics metrics) in /app/K2Bridge/KustoDAL/CslQueryProviderExtensions.cs:line 40
at K2Bridge.KustoDAL.KustoQueryExecutor.ExecuteQueryAsync(QueryData queryData, RequestContext requestContext) in /app/K2Bridge/KustoDAL/KustoQueryExecutor.cs:line 117
Looks like, it should be Payload["@l"] instead of Payload.@l in query.
Community Note
Currently K2Bridge only supports the 'Discover' tab but we're interesting in learning if this is a valuable feature to include in the future.
Kibana uses _id
field for several features:
Currently we generate our _id
fields from Filebeat with add_id
processor. https://www.elastic.co/guide/en/beats/filebeat/current/add-id.html
In order to have Kibana experience consistent with elastic, we have to use _id
from Kusto if the table contains it.
Description of the problem:
Kibana trims / rounds real values, users who would like to filter according to these values are not getting any results (or wrong results)
How to reproduce:
Hi,
It's possible to use the newes version of Elasticsearch "eck-elasticsearch" or use the latest version 7 (7.17.3) ?
Anyone deploy that in the latest version?
Best Regards
gar0t0
Hi. It seems that double-quotes escape in filters doesnt'work:
Repro:
Get field and value with double-quotes, click "+", receive error.
The error itself:
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"SyntaxException","reason":"Syntax error: Query could not be parsed: I could not parse that, sorry.. Query: 'let fromUnixTimeMilli = (t:long) {datetime(1970 - 01 - 01) + t * 1millisec};\nlet _data = database(\"database\").Table | where (message == \"Health check \"LF\" completed after 35.422ms with status Unhealthy and '\"{\\\"Url\\\":\\\"http://url\\\"}\"'\") and (Timestamp >= fromUnixTimeMilli(1584385027085) and Timestamp <= fromUnixTimeMilli(1584385927085));\n(_data | summarize count() by Timestamp = bin(Timestamp, 30s)\n| order by Timestamp asc | as aggs);\n(_data | order by Timestamp desc | limit 500 | as hits)'","index_uuid":"index","index":"index"}],"type":"QueryException","reason":"Failed executing azure data explorer query","phase":"query"},"status":500}
Lucene highlighter failed when filter by text with { }
symbols.
I am trying to use filter and it filters as expected:
{
"query": {
"match_phrase": {
"Payload.@mt": "Lucene highlighter is enabled: {isHighlight}"
}
}
}
But Kibana did not highlight text and I can see next errors at K2Bridge log:
| Payload.@l | Error
| Payload.@mt | Failure creating highlighters for {value}
| Payload.@t | Jan 19, 2022 @ 15:04:25.122
| Payload.@x | Lucene.Net.QueryParsers.ParseException: Cannot parse 'Lucene highlighter is enabled: {isHighlight}': Encountered " "}" "} "" at line 1, column 43. Was expecting one of: "TO" ... <RANGEEX_QUOTED> ... <RANGEEX_GOOP> ... ---> Lucene.Net.QueryParsers.ParseException: Encountered " "}" "} "" at line 1, column 43. Was expecting one of: "TO" ... <RANGEEX_QUOTED> ... <RANGEEX_GOOP> ... at Lucene.Net.QueryParsers.QueryParser.Jj_consume_token(Int32 kind) at Lucene.Net.QueryParsers.QueryParser.Term(String field) at Lucene.Net.QueryParsers.QueryParser.Clause(String field) at Lucene.Net.QueryParsers.QueryParser.Query(String field) at Lucene.Net.QueryParsers.QueryParser.Parse(String query) --- End of inner exception stack trace --- at Lucene.Net.QueryParsers.QueryParser.Parse(String query) at K2Bridge.KustoDAL.LuceneHighlighter.MakeValueHighlighter(QueryParser parser, String value, String highlightPreTag, String highlightPostTag) in /app/K2Bridge/KustoDAL/LuceneHighlighter.cs:line 171
| Payload.ActionId | af9a55f3-a30f-4f19-a348-674aa037d0c3
| Payload.ActionName | K2Bridge.Controllers.QueryController.SingleSearchAsync (K2Bridge)
| Payload.ConnectionId | 0HMER3KCK6TR2
| Payload.CorrelationId | 3f885b0f-be58-4021-be3d-02360c800848
| Payload.RequestId | 0HMER3KCK6TR2:00000329
| Payload.RequestPath | /log:InfraKustokibana/_search
| Payload.SourceContext | K2Bridge.KustoDAL.KustoResponseParser
Add a dedicated range aggregation for IP typed fields
"aggs": {
"ip_ranges": {
"ip_range": {
"field": "ip",
"ranges": [
{ "to": "10.0.0.5" },
{ "from": "10.0.0.5" }
]
}
}
}
Response:
{
...
"aggregations": {
"ip_ranges": {
"buckets": [
{
"key": "*-10.0.0.5",
"to": "10.0.0.5",
"doc_count": 10
},
{
"key": "10.0.0.5-*",
"from": "10.0.0.5",
"doc_count": 260
}
]
}
}
}
Reference : IP range aggregation
Useful Kusto functions:
Dynamic columns, meaning json objects stored in ADX are partially supported.
One can view and search them by referencing properties directly in the search box, but cannot use UI features such as filtering
Context:
Sometimes the search fails, whether due to a bug or a bad input from the user, e.g. when searching for an invalid field
NotARealColumn:something
The error Kibana displays is "Internal server error".
This issue suggest:
a. provide a more meaningful error
b. if possible wrap with user friendly error msg -> "invalid column try again"
There is an issue while creating Kibana index pattern while using the K2Bridge image version 7.16_latest, and also with the image 0.5.15164 that are the latest images at the moment.
When trying to create an index pattern there is an available list of all the tables on the ADX database, and when trying to create an index pattern for one of the tables there is a message: The indices which match this index pattern don't contain any time fields
.
But, in the table timestamp as information exists in the field timestamp.
The index pattern fails to be created with an error: Error fetching fields for index pattern
Here are some of the screenshots:
From the k2bridge deployment logs, there are some that seem to be produced at the time of failed index pattern creation and that could be useful for troubleshooting the issue. If there are some other logs and information that could be useful, they can also be provided at request.
{"@t":"2023-06-28T10:49:52.8278841Z","@mt":"Translate params: header:{@header}, query:{@query}","@l":"Debug","header":"{\"index\":\"loggingAppDB:GELFLogs*\"}","query":{"Data":"{\"size\":0,\"aggs\":{\"indices\":{\"terms\":{\"field\":\"_index\",\"size\":200}}}}"},"SourceContext":"K2Bridge.ElasticQueryTranslator","ActionId":"b23454a6-cdea-452d-a72d-df33c221fd89","ActionName":"K2Bridge.Controllers.QueryController.SingleSearchAsync (K2Bridge)","RequestId":"0HMRNJV1G883V:00000016","RequestPath":"/loggingAppDB:GELFLogs*/_search","ConnectionId":"0HMRNJV1G883V","CorrelationId":"f2e4d341-2453-4235-ab27-834cc9ce6ad4"}
{"@t":"2023-06-28T10:49:52.8292286Z","@mt":"Failed to execute translate operation.","@l":"Error","@x":"System.ArgumentNullException: Query cannot be null (Parameter 'Query')\n at K2Bridge.Ensure.ConstructMessageAndThrowArgumentOrNullArgument[T](T arg, String argName, String predefinedMessage, ILogger logger) in /app/K2Bridge/Ensure.cs:line 92\n at K2Bridge.ElasticQueryTranslator.TranslateQuery(String header, String query) in /app/K2Bridge/ElasticQueryTranslator.cs:line 49","SourceContext":"K2Bridge.ElasticQueryTranslator","ActionId":"b23454a6-cdea-452d-a72d-df33c221fd89","ActionName":"K2Bridge.Controllers.QueryController.SingleSearchAsync (K2Bridge)","RequestId":"0HMRNJV1G883V:00000016","RequestPath":"/loggingAppDB:GELFLogs*/_search","ConnectionId":"0HMRNJV1G883V","CorrelationId":"f2e4d341-2453-4235-ab27-834cc9ce6ad4"}
{"@t":"2023-06-28T10:49:52.8295303Z","@mt":"Failed translating elasticsearch query","@l":"Error","SourceContext":"K2Bridge.Controllers.QueryController","ActionId":"b23454a6-cdea-452d-a72d-df33c221fd89","ActionName":"K2Bridge.Controllers.QueryController.SingleSearchAsync (K2Bridge)","RequestId":"0HMRNJV1G883V:00000016","RequestPath":"/loggingAppDB:GELFLogs*/_search","ConnectionId":"0HMRNJV1G883V","CorrelationId":"f2e4d341-2453-4235-ab27-834cc9ce6ad4"}
{"@t":"2023-06-28T10:52:37.5007653Z","@mt":"Listing tables matching '{@indexName}'","@l":"Debug","indexName":"*:*","SourceContext":"K2Bridge.KustoDAL.KustoDataAccess","ActionId":"1943a0e3-4938-4e84-9026-a2548e0b8332","ActionName":"K2Bridge.Controllers.IndexListController.Resolve (K2Bridge)","RequestId":"0HMRNJV1G885U:00000002","RequestPath":"/_resolve/index/*:*","ConnectionId":"0HMRNJV1G885U","CorrelationId":"fb1293f7-933a-4d7a-94d5-d800bf977afc"}
{"@t":"2023-06-28T10:52:37.5010166Z","@mt":"Calling adminClient.ExecuteControlCommand with the command: {@command}","@l":"Debug","command":{"Data":".show databases schema | where TableName != '' | distinct TableName, DatabaseName | search TableName: '*' | search DatabaseName: '*' | project strcat(DatabaseName, \":\", TableName)"},"SourceContext":"K2Bridge.KustoDAL.KustoQueryExecutor","ActionId":"1943a0e3-4938-4e84-9026-a2548e0b8332","ActionName":"K2Bridge.Controllers.IndexListController.Resolve (K2Bridge)","RequestId":"0HMRNJV1G885U:00000002","RequestPath":"/_resolve/index/*:*","ConnectionId":"0HMRNJV1G885U","CorrelationId":"fb1293f7-933a-4d7a-94d5-d800bf977afc"}
{"@t":"2023-06-28T10:52:37.5504433Z","@mt":"Translate params: header:{@header}, query:{@query}","@l":"Debug","header":"{\"index\":\"*:*\"}","query":{"Data":"{\"size\":0,\"aggs\":{\"indices\":{\"terms\":{\"field\":\"_index\",\"size\":200}}}}"},"SourceContext":"K2Bridge.ElasticQueryTranslator","ActionId":"b23454a6-cdea-452d-a72d-df33c221fd89","ActionName":"K2Bridge.Controllers.QueryController.SingleSearchAsync (K2Bridge)","RequestId":"0HMRNJV1G883V:00000022","RequestPath":"/*:*/_search","ConnectionId":"0HMRNJV1G883V","CorrelationId":"7b3eb7ca-b770-4157-a3c6-2fec1c09c2eb"}
{"@t":"2023-06-28T10:52:37.5507401Z","@mt":"Failed to execute translate operation.","@l":"Error","@x":"System.ArgumentNullException: Query cannot be null (Parameter 'Query')\n at K2Bridge.Ensure.ConstructMessageAndThrowArgumentOrNullArgument[T](T arg, String argName, String predefinedMessage, ILogger logger) in /app/K2Bridge/Ensure.cs:line 92\n at K2Bridge.ElasticQueryTranslator.TranslateQuery(String header, String query) in /app/K2Bridge/ElasticQueryTranslator.cs:line 49","SourceContext":"K2Bridge.ElasticQueryTranslator","ActionId":"b23454a6-cdea-452d-a72d-df33c221fd89","ActionName":"K2Bridge.Controllers.QueryController.SingleSearchAsync (K2Bridge)","RequestId":"0HMRNJV1G883V:00000022","RequestPath":"/*:*/_search","ConnectionId":"0HMRNJV1G883V","CorrelationId":"7b3eb7ca-b770-4157-a3c6-2fec1c09c2eb"}
{"@t":"2023-06-28T10:52:37.5510253Z","@mt":"Failed translating elasticsearch query","@l":"Error","SourceContext":"K2Bridge.Controllers.QueryController","ActionId":"b23454a6-cdea-452d-a72d-df33c221fd89","ActionName":"K2Bridge.Controllers.QueryController.SingleSearchAsync (K2Bridge)","RequestId":"0HMRNJV1G883V:00000022","RequestPath":"/*:*/_search","ConnectionId":"0HMRNJV1G883V","CorrelationId":"7b3eb7ca-b770-4157-a3c6-2fec1c09c2eb"}
Hi. In our work we use a lot of index patterns with dashes, i.e. 'dev-sre'.
Because of the way Kusto works, you can't directly create view 'dev-sre', you have to create it as ['dev-sre']
.
Even when you create view like that, Kusto Kibana can't use it then, returning an exception:
//creating a functions
.create function ['abc-dev'] { print 1}
//making sure it works
['abc-dev']
| take 10
Then you add it to index patterns as usual, and when you try to view it:
K2 Exception:
K2Bridge.KustoDAL.QueryException: Failed executing azure data explorer query
---> [0]Kusto.Data.Exceptions.SemanticException: Semantic error: 'abc-dev | getschema | project ColumnName, ColumnType=DataType' has the following semantic error: Scalar is not expected in the current context.
Timestamp=2020-03-17T07:32:39.5549754Z
ClientRequestId=K2Bridge.ExecuteQuery;6950ffc7-1a5c-4056-9288-31abe0f02abf
ActivityId=7a7e529a-077b-4dc3-abb6-6c806149a0b8
ActivityType=DN.RestClient.ExecuteQuery
MachineName=k2bridge-5c4597fb58-nqbk9
ProcessName=dotnet
ProcessId=1
ThreadId=14559
AppDomainName=K2Bridge
ActivityStack=(Activity stack: CRID=K2Bridge.ExecuteQuery;6950ffc7-1a5c-4056-9288-31abe0f02abf ARID=7a7e529a-077b-4dc3-abb6-6c806149a0b8 > DN.RestClient.ExecuteQuery/7a7e529a-077b-4dc3-abb6-6c806149a0b8)
MonitoredActivityContext=ActivityType=DN.RestClient.ExecuteQuery, Timestamp=2020-03-17T07:32:39.5403952Z, ParentActivityId=7a7e529a-077b-4dc3-abb6-6c806149a0b8, TimeSinceStarted=14.6528 [ms]
ErrorCode=SEM0062
ErrorReason=BadRequest
ErrorMessage=Scalar is not expected in the current context
DataSource=
DatabaseName=log
ClientRequestId=K2Bridge.ExecuteQuery;6950ffc7-1a5c-4056-9288-31abe0f02abf
ActivityId=7005969f-0bb0-4859-a4a9-d262371efe54
Text=abc-dev | getschema | project ColumnName, ColumnType=DataType
SemanticErrors=Scalar is not expected in the current context
at Kusto.Cloud.Platform.Http.KustoHttpClient.ThrowKustoExceptionFromResponseMessageAsync(KustoExceptionContext exceptionContext, HttpResponseMessage responseMessage, ClientRequestProperties properties, Boolean shouldBuffer, Action`2 notify)
at Kusto.Data.Net.Client.RestClient2.MakeHttpRequestAsyncImpl(String address, String csl, String ns, String databaseName, Boolean streaming, ClientRequestProperties properties, ServiceModelTimeoutKind timeoutKind, String clientRequestId, Stream body, StreamProperties streamProperties)
at Kusto.Cloud.Platform.Utils.MonitoredActivity.InvokeAsync[TActivityType,TResult](TActivityType activityType, Func`1 func, String clientRequestId)
at Kusto.Cloud.Platform.Utils.MonitoredActivity.InvokeAsync[TActivityType,TResult](TActivityType activityType, Func`1 func, String clientRequestId)
at Kusto.Data.Net.Client.RestClient2.MakeHttpRequestAsync(ActivityType activityType, String baseAddress, String relativeAddress, String clientRequestIdPrefix, String ns, String databaseName, String csl, String addr, Boolean streaming, ClientRequestProperties properties, ServiceModelTimeoutKind timeoutKind, StreamProperties streamProperties)
at Kusto.Data.Net.Client.RestClient2.ExecuteQueryAsync(String databaseName, String query, ClientRequestProperties properties)
at K2Bridge.KustoDAL.CslQueryProviderExtensions.ExecuteMonitoredQueryAsync(ICslQueryProvider client, String query, ClientRequestProperties clientRequestProperties, Metrics metrics) in /app/K2Bridge/KustoDAL/CslQueryProviderExtensions.cs:line 50
at K2Bridge.KustoDAL.KustoQueryExecutor.ExecuteQueryAsync(QueryData queryData, RequestContext requestContext) in /app/K2Bridge/KustoDAL/KustoQueryExecutor.cs:line 117
--- End of inner exception stack trace ---
at K2Bridge.KustoDAL.KustoQueryExecutor.ExecuteQueryAsync(QueryData queryData, RequestContext requestContext) in /app/K2Bridge/KustoDAL/KustoQueryExecutor.cs:line 117
at K2Bridge.KustoDAL.KustoDataAccess.GetFieldCapsAsync(String indexName) in /app/K2Bridge/KustoDAL/KustoDataAccess.cs:line 76
at K2Bridge.Controllers.FieldCapabilityController.Process(String indexName) in /app/K2Bridge/Controllers/FieldCapabilityController.cs:line 57
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Prometheus.HttpMetrics.HttpRequestDurationMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpRequestCountMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpInProgressMiddleware.Invoke(HttpContext context)
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
Helm chart has an ingress flag, but it's not obvious how to use it; extra parameters must be set as well.
Syntax seems to be
--set 'ingress.hosts[0].host=<HOST>' --set 'ingress.hosts[0].paths[0]=/'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.