azure / azure-kusto-java Goto Github PK

Describe the bug
Using the Java SDK I'm trying to execute a query to get data from Application Insights from the Azure Data Explorer Cluster. The data I'm trying to get are customEvents.

The adjusted error message I'm getting is the following:

com.microsoft.azure.kusto.data.exceptions.DataServiceException: not authorized to access resource name: -, resource id: , in cluster, scoped to subscription -, resource group telemetry, ActivityId='-'
	at com.microsoft.azure.kusto.data.Utils.createExceptionFromResponse(Utils.java:162)
	at com.microsoft.azure.kusto.data.Utils.post(Utils.java:65)
	at com.microsoft.azure.kusto.data.ClientImpl.executeToJsonResult(ClientImpl.java:118)
	at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:77)
	at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:72)
	at Main.main(Main.java:24)

To Reproduce
Steps to reproduce the behavior:

Followed the Prerequisites based on the README

1. Created Azure Data Explorer Cluster
2. Created Azure Active Directory App Registration
3. Added Azure Application Insights Cluster in Azure Data Explorer Portal

Expected behavior
The query should work and return customEvents.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: macOS Big Sur
• Version 11.4
• Kusto Query SDK: com.microsoft.azure.kusto:kusto-ingest:2.8.2

Additional context
A service request was created under #2108040050000668 whereas the recommendation was to create a issue on GitHub. Is there an official support of the Java SDK to query Azure Application Insights data trough Azure Data Explorer.

The primitive getXXX methods in KustoResultSetTable throw NPEs when the column value is null. This is caused by casting the nullable Object to the different primitive types without checking for nullability. For example:

@Override
public int getInt(int i)  {
    return (int) get(i);
}

@Override
public long getLong(int i)  {
    Object obj = get(i);
    if(obj instanceof Integer){
        return ((Integer)obj).longValue();
    }
    return (long) obj;
}

@Override
public float getFloat(int i)  {
    return (float) get(i); }

@Override
public double getDouble(int i)  {
    return (double) get(i);
}

This can be with a browser popup or other means

This issue is with high importance.
We need to upgrade to storage client to V10 in order to get async operations.
The current implementation is not async.

In version 10 the Azure Storage Java SDK was separated into 2 sub-modules:

azure-storage-blob
azure-storage-queue

In order to use these modules, they must be added as dependencies to the ingest/pom.xml file, instead of the currently used azure-storage dependency: (Correct to the date of writing this comment)

<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
    <version>10.1.0</version>
</dependency>
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-storage-queue</artifactId>
    <version>10.0.0-Preview</version>
</dependency>

Note that currently the azure-storage-queue is a preview version

Migrating to version 10 will require a substantial rewrite of any component that interfaces with Azure Storage.
Here is a list of the classes in the “ingest” module that are involved with the Azure Storage SDK: (correct to the date of writing this comment)

• AzureStorageClient.java
• IngestClientImpl.java
• IngestionStatus.java
• TableReportIngestionResult.java
• IngestClientImplTest.java
• AzureStorageClientTest.java

We need to refactor the client to implement an interface.
likely to be included:

• Client (interface)
• ClientImpl (class implementing the interface)
• ClientFactory

• An example can be seen in the ingest package.

In this task you should also take care of removing the Kusto name from class and method names.

On parity with .Net, app AuthN with locally deployed cert by thumbprint and how this is possible with ADAL-java

In creation of Results from response, calling new JSONArray creates integers from long

Is your feature request related to a problem? Please describe.
I need to import CSV files

Describe the solution you'd like
ignoreFirstRecord option on IngestionProperties

Describe alternatives you've considered
modifying the file...

Additional context
blocker to use ingestFromFile with CSV

Describe the bug
The HttpClientProperties reference in the samples/../Query.java example does not compile with the version of the kusto client in the library (3.x).

With the 3.x changes, HttpClientProperties reference is invalid:

[ERROR] /..../azure-kusto-java/samples/src/main/java/Query.java:[6,38] cannot find symbol
[ERROR]   symbol:   class HttpClientProperties
[ERROR]   location: package com.microsoft.azure.kusto.data
[ERROR] /..../azure-kusto-java/samples/src/main/java/Query.java:[24,13] cannot find symbol
[ERROR]   symbol:   class HttpClientProperties
[ERROR]   location: class Query
[ERROR] /..../azure-kusto-java/samples/src/main/java/Query.java:[24,47] cannot find symbol
[ERROR]   symbol:   variable HttpClientProperties
[ERROR]   location: class Query

To Reproduce

• Clone repo
• Follow samples/README.md instruction to run the Query example

Expected behavior
Should run as documented

Screenshots

Desktop (please complete the following information):

Additional context

As in Node library - take from env if provided

Is your feature request related to a problem? Please describe.
Including these 2 fields would Improve telemetry.

Describe the solution you'd like
Add the 2 headers, x-ms-activitycontext and x-ms-user-id, to all POST requests, using the same approach used by the .NET client.

Describe alternatives you've considered
N/A

Additional context
x-ms-activitycontext is an object containing ActivityId, ClientActivityId, RootActivityId

Add/modify headers in post() method in Utils class, to include the headers as they appear here.

When importing kusto-ingest via pom.xml :

<dependency>
    <groupId>com.microsoft.azure.kusto</groupId>
    <artifactId>kusto-ingest</artifactId>
    <version>2.8.2</version>
</dependency>

Kusto-Ingest pulls the following deprecated libraries:

• com.microsoft.azure/azure-keyvault-core
• com.microsoft.azure/azure-storage

The Microsoft Azure IntelliJ plugin complains about it. Can you update them to the Microsoft recommendation of

• com.azure/azure-storage-blob
• com.azure/azure-storage-file-share
• com.azure/azure-storage-queue

Describe the bug
When I use the ADX Kusto ClientImpl and call .execute, there's a request every time to the AAD Token endpoint to get a new access token before the actual AAD call. This should be fetched the first time then cached.

To Reproduce
I have a modified sample here: https://github.com/andrejpk/azure-kusto-java/blob/apk/not-caching-msal-token/samples/src/main/java/Query.java

This:

• removes outdated reference that no longer works for 3.x (so it works) see #219
• adds Log4J to get logging output from MSAL
• loops to call the query 3x times

Run using the instructions in the samples/README

Expected behavior
I expect the call to the token endpoint on the first loop but subsequent calls should use a cached token.

This wasn't the case in a prior POC project where we used the 2.x client.

Screenshots
Here are the logs emitted. You can see the Token not found in the cache messages and subsequent calls to fetch a new one each time.

[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: 82859f65-e184-4589-8373-235d41b0b9d1] Execution of class com.microsoft.aad.msal4j.AcquireTokenSilentSupplier failed.
com.microsoft.aad.msal4j.MsalClientException: Token not found in the cache
	at com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:98)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - SkipCache set to false. Attempting cache lookup
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - Cache lookup failed: Token not found in the cache
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: 06d7e772-fcaf-4c7a-ae3f-a3644346e614] Access Token was returned
Kusto sent back 1 rows.
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: ca61874f-f414-497e-8b16-af64b8487174] Execution of class com.microsoft.aad.msal4j.AcquireTokenSilentSupplier failed.
com.microsoft.aad.msal4j.MsalClientException: Token not found in the cache
	at com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:98)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - SkipCache set to false. Attempting cache lookup
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - Cache lookup failed: Token not found in the cache
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: a9eb0ca4-af48-47c7-bb11-780e21bf47b8] Access Token was returned
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: 114c1aee-a9e8-444f-9187-b6e2ce849899] Execution of class com.microsoft.aad.msal4j.AcquireTokenSilentSupplier failed.
com.microsoft.aad.msal4j.MsalClientException: Token not found in the cache
	at com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:98)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - SkipCache set to false. Attempting cache lookup
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - Cache lookup failed: Token not found in the cache
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: cd4ff010-1a6f-4bd6-b380-1d9b043127f3] Access Token was returned
Kusto sent back 1 rows.
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: 792cd658-232b-4055-ab35-0180d21af300] Execution of class com.microsoft.aad.msal4j.AcquireTokenSilentSupplier failed.
com.microsoft.aad.msal4j.MsalClientException: Token not found in the cache
	at com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:98)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - SkipCache set to false. Attempting cache lookup
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - Cache lookup failed: Token not found in the cache
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: 68aa429b-b1e8-4ffc-a2d5-b6e1c62467df] Access Token was returned
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: bf712def-55fe-42ef-9a07-944f60e7072d] Execution of class com.microsoft.aad.msal4j.AcquireTokenSilentSupplier failed.
com.microsoft.aad.msal4j.MsalClientException: Token not found in the cache
	at com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:98)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - SkipCache set to false. Attempting cache lookup
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - Cache lookup failed: Token not found in the cache
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: a8ac8936-6de3-4ba9-a1e1-9960028cf533] Access Token was returned
Kusto sent back 1 rows.
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: 7bf4d2be-2387-4b71-8ddf-3764620843f8] Execution of class com.microsoft.aad.msal4j.AcquireTokenSilentSupplier failed.
com.microsoft.aad.msal4j.MsalClientException: Token not found in the cache
	at com.microsoft.aad.msal4j.AcquireTokenSilentSupplier.execute(AcquireTokenSilentSupplier.java:98)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
	at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - SkipCache set to false. Attempting cache lookup
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier - Cache lookup failed: Token not found in the cache
[ForkJoinPool.commonPool-worker-19] DEBUG com.microsoft.aad.msal4j.ConfidentialClientApplication - [Correlation ID: ff8d5a18-4210-4c80-9709-7aa83a4ea99d] Access Token was returned

The query runs but spends more time calling AAD than ADX. This mirrors the behavior in our application.

Desktop (please complete the following information):

• OS: OSX
• Version [e.g. 22]

Additional context
Feel free to reach out to me internally; I can share traces.

Version 1.9.13 of jackson-mapper-asl is vulnerable to CVE-2019-10172.

Additionally jackson-mapper-asl is EOL and has moved to https://github.com/FasterXML/jackson-databind. See: https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl

This can be fixed by migrating to jackson-databind.

Implement in such a way that resources cached in the client will be handled by a single thread at a time.
This way, the refresh procedures will not happen more than one time in parallel.

• How to install
• Sample usage

The current implementation (0.2.0) does not work behind a proxy.

Steps to recreate

1. Use a computer behind a proxy that does not have internet access
2. checkout azure-kusto-java
3. run "mvn install"
4. cd samples
5. (edit the https.proxyHost and https.proxyPort to match your environment) mvn exec:java -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8888 -Dexec.mainClass="Query"

result:
[pool-1-thread-1] INFO com.microsoft.aad.adal4j.UserDiscoveryRequest - [Correlation ID: null] Sent (null) Correlation Id is not same as received (null).
[pool-1-thread-1] INFO com.microsoft.aad.adal4j.AuthenticationAuthority - [Correlation ID: 0a60824f-7bf6-40c3-bff5-fa0862e01842] Instance discovery was successful
org.apache.http.conn.HttpHostConnectException: Connect to bmw.kusto.windows.net:443 [bmw.kusto.windows.net/40.112.88.57] failed: Connection timed out: connect
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.microsoft.azure.kusto.data.Utils.post(Utils.java:40)
at com.microsoft.azure.kusto.data.KustoClient.execute(KustoClient.java:40)
at com.microsoft.azure.kusto.data.KustoClient.execute(KustoClient.java:30)
at Query.main(Query.java:21)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection timed out: connect

The fix is simple, change the com.microsoft.azure.kusto.data.Utils

"HttpClient httpClient = HttpClients.createDefault();"
to
"HttpClient httpClient = HttpClients.createSystem();"

The fix will work when no proxy is used as well.

More info here:
https://stackoverflow.com/questions/5165126/without-changing-code-how-to-force-httpclient-to-use-proxy-by-environment-varia

Describe the bug
When using the QueuedIngestClient to conduct multi-thread ingestions to ADX, the code always throws "java.lang.IllegalStateException: Timer already cancelled." even though the ingestion is successful.

Stacktrace:
Exception in thread "Timer-58" java.lang.IllegalStateException: Timer already cancelled.
at java.util.Timer.sched(Timer.java:397)
at java.util.Timer.schedule(Timer.java:193)
at com.microsoft.azure.kusto.ingest.ResourceManager$1RefreshIngestionResourcesTask.run(ResourceManager.java:96)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

To Reproduce
Steps to reproduce the behavior:
Use the QueuedIngestClient for large numbers of parallel ingestions with Service Bus Trigger (Azure Function App)

1. ingestClient = IngestClientFactory.createClient(ConnectionStringBuilder csb);
2. ingestClient.ingestFromStream(StreamSourceInfo streamSourceInfo, IngestionProperties ingestionProperties).

Expected behavior
Ingestion is successful without "Timer already cancelled" Exception

Desktop (please complete the following information):

• OS: Unbuntu
• Version 18.04

Additional Information
The Version of azure-kusto-java in use is 2.3.1

Describe the bug
I managed to create externalTable via the api, but it failed to create continue export.
Behaviour is not consistent
the error:
Caused by: com.microsoft.azure.kusto.data.exceptions.DataWebException: Bad request: Control commands (starting with a dot '.') cannot be served from the query endpoint unless they are .show control commands.
Please provide the following information when contacting the Kusto team:
ClientRequestId='KJC.execute;be2d9787-6e7d-44db-af0d-c370c88de3b9', ActivityId='78661026-0b7b-4ba8-bf83-b694d5381c5d', Timestamp='2020-08-05T12:51:31.0018478Z'.
... 55 more

Did not find any other api for the control plan.

To Reproduce
scala example:
def createContQuery(contExportName: String, origTableName: String, externalTableName: String, queryWithProjection: String): String = {
s"""
create-or-alter continuous-export $contExportName
over ($origTableName)
to table $externalTableName
with
(intervalBetweenRuns=1m,
sizeLimit=104857600
)
<| $queryWithProjection
"""
}

val contExportResult: KustoOperationResult =  kustoConn.execute(database, contExportQuery)

Expected behavior
create continue export
Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]
  mac os catalina
  java 8

Additional context
Add any other context about the problem here.

Describe the bug
the client create api nowhere to specify the cloud china, it always using the global azure login endpoint.

To Reproduce
Steps to reproduce the behavior: using the china adx cluster address do connection

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Constantly getting error in databricks trying to load dependencies. Tried locally installing the javax.mail 1.4.7 jar and javax.mail-api 1.6.1 jar with no luck. It appears an upstream dependency is taken on javax.mail instead of javax.mail-api.

databricks runtime version 5.4 (includes Apache Spark 2.4.3, Scala 2.11)

Is your feature request related to a problem? Please describe.
When I originally replaced ADL with MSAL in the Java SDK, the latest Java MSAL API (1.7.0) did not support SNI auth.

However, the very next version (1.7.1) added support for it (see AzureAD/microsoft-authentication-library-for-java#219).

Describe the solution you'd like
Implement SNI.

Describe alternatives you've considered
Can be considered low priority.

Additional context
N/A

current version: 20090211.

consumers of the package which are using a newer version of it can end up having conflicts, at it seems that newer versions have implicit casting when reading Kusto's response, which breaks the current implementation of the library:

                JSONArray resultsRows = table0.getJSONArray("Rows");
                ArrayList<ArrayList<String>> values = new ArrayList<>();
                for (int i = 0; i < resultsRows.length(); i++) {
                    JSONArray row = resultsRows.getJSONArray(i);
                    ArrayList<String> rowVector = new ArrayList<>();
                    for (int j = 0; j < row.length(); j++) {
                        rowVector.add(row.getString(j)); <--- this breaks with newer versions, as the JSONArray implicitly casts numerics
                    }
                    values.add(rowVector);

This was opened per @toshetah request as part of the review of #58.

We need to think about the necessity of such a factory since currently the "basic" AzureStorageHelper is used internally and created directly when required. The IngestClientImp constructor that allows setting the AzureStorageHelper isn't publicly visible to users and only used internally in the package.

Investigate how to release:

• jar signing
• maven definitions
• automating test-build-release

• defer partial results
• option server

For every call of the KustoClient execute(String command) and execute(String database, String command) a new AccessToken is requested and this token is valid for one day.

This adds about 500ms to every request

Please cache this JWT and only renew once expired.

This is an example:
https://github.com/Azure-Samples/compute-java-manage-vm

1. For historical reasons, the Java ResultSet uses 1-based column numbering (see javadoc) while KustoResultSetTable uses 0-based numbering

2. According to the Javadoc, ResultSet's getObject method should return null for cells with a null value, but that's not what KustoResultSetTable does. For an empty string cell, we get an empty String instead of null. This is especially problematic for columns in which empty and null values have different meanings.

Kusto-ingest 2.8.2 pulls com.google.guava:guava v20 with 2 CVEs from 2018/2020:

• CVE-2018-10237
• CVE-2020-8908

This triggers various warnings in build pipelines that perform CVE validation (e.g. WhiteSource, etc.)
Please update the reference to com.google.guava:guava v30.

Describe the bug
Any attempt to start AAD device code authentication fails with,

1. Caused by java.lang.UnsupportedOperationException
   The BROWSE action is not supported on the current platform!

To Reproduce
Steps to reproduce the behavior:

 ConnectionStringBuilder csb = ConnectionStringBuilder.createWithDeviceCodeCredentials(
    "<cluster_uri>"
);

And try to run this sample: https://github.com/Azure/azure-kusto-java/blob/master/samples/src/main/java/Query.java

Expected behavior
Should be able to recover from a browser not found error in a non-GUI env.

Desktop (please complete the following information):

• OS: Windows 10 / WSL 2
• Version WSL 2

Additional context
More info here: https://bugs.launchpad.net/ubuntu/+source/openjdk-8/+bug/1574879

Basically it boils down to GNOME dependencies not present in a WSL env since WSL is non-GUI.

The Kusto client takes raw Strings as queries, which is a very dangerous practice.

If part of the query contains a user-provided String (e.g. in a where clause), a malicious user could easily modify the query to run arbitrary code against the cluster.

For example consider something like String query = "Users | where UserId = " + userIdParam"; If userIdParam turns out to be the string 123 ; Users | take 100, we'd end up running the query `"Users | where UserId = 123; Users | take 100" which would return both the records for user 123 as well as 100 other records that might belong to other users. This can be used as a way of gaining access to forbidden information as well as performing denial of service attacks by running computationally expensive queries.

Every database driver I've ever seen (both relational and NoSQL) has protections against SQL injection in the form of either query builders or prepared statements, but I don't see anything of that sort in Kusto.

Hi, folks, we're using a few Azure Data Explorer clusters in a multi-region deployment of a service. The service exposes a REST API, allowing its users to query data which is found in the ADX cluster -- so we're talking about a long-lived application here.

The service uses the kusto-data library to execute the queries against the ADX cluster, specifically one of the flavours of com.microsoft.azure.kusto.data.Client.execute() see https://github.com/Azure/azure-kusto-java/blob/master/data/src/main/java/com/microsoft/azure/kusto/data/Client.java#L12

What we noticed is that the implementation of the Client:

• creates a new HTTP client object for each request, along with everything it entails such as an associated connection pool, a pool manager, etc.
• does not reuse HTTP connections or clients;
• does not close the HTTP client or the connection it uses in a request.

This can be observed here:

This is problematic from multiple aspects:

• from a performance perspective, each query the service makes to ADX incurs the price of creating the new client and its associated resources, creating a new connection, SSL handshake, etc.
• from a resource perspective, this is a waste of server resources.

What the library should probably do is create and reuse an HTTP client across multiple queries / requests. The client could be created during some sort of initialisation phase and should be closed during a "close" phase.

Any thoughts or feedback about this would be greatly appreciated.

Things to think about:

1. Make methods non static
2. Keep container, queue references and not create new ones on every call (will require some kind of resource manager sync)

• New method in the client factory
• KustoConnectionStringBuilder should support a string parameter
• In the client implementation class, write to the log which option was used

Result should include all tables returned by the client
Result can also be generic - returning the generic type passed to the ingest

As per @vladikbr comment - "common" should be used as the default AAD tenant when no specific one was defined in the connection builder

Define the methods on the interface and implement the client accordingly

This SDK (Kusto JAVA SDK) isn't compatible with Android as it uses the Apache's Http Client and that is deprecated on Android.

See these for more context:

• https://developer.android.com/about/versions/pie/android-9.0-changes-28#apache-p
• https://developer.android.com/about/versions/marshmallow/android-6.0-changes
• https://stackoverflow.com/questions/54652322/how-to-use-apache-http-client-4-5-x-used-by-external-library-on-android

The question is, do we actually have a Kusto SDK specifically for Android? or one at least compatible with Android that doesn't use the Apache's Http Client. If not, is there a plan to support Android?

This is a good example:
https://github.com/Azure/azure-libraries-for-java

The KustoResultSetTable class implements java.sql.ResultSet, which as its name suggests is part of Java's sql api, however it looks like Kusto chooses to implement only this one little snippet of the API without providing a JDBC driver. Without a JDBC driver, the various Java libraries for working with databases can't work with Kusto.

So why the ResultSet? Is a JDBC driver being worked on? Can someone please share the reasoning behind this?