while retrieving chunks of data through URL it was throwing an exception about snowflake-connector-net HOT 12 CLOSED

snowflakedb commented on August 29, 2024

while retrieving chunks of data through URL it was throwing an exception

from snowflake-connector-net.

Comments (12)

howryu commented on August 29, 2024

This exception is thrown while retrieving a large chunk of data. Normally, server will write data to a S3 bucket and client will download result from the s3 bucket. This looks like request timed out when trying to download data from s3. Just wondering if your host can connect to s3 or not? Do you have any proxy config?

from snowflake-connector-net.

CPavanKishore commented on August 29, 2024

Yes we have large chunk of data around min of 30k , even we have a proxy firewall it is getting 1st chunk of data and showing it in the console after continuously hitting the rest chunks it is throwing error

from snowflake-connector-net.

CPavanKishore commented on August 29, 2024

Is there any way to resolve this issue....?

from snowflake-connector-net.

howryu commented on August 29, 2024

@CPavanKishore First chunk of data is returned from snowflake server directly. So basically, the first response for a large result set contains result set metadata, a few rows of data, and the urls to the rest of data in s3 bucket. So my question to you is: are you able to connect s3 from your dev box? A simple curl (or whatever cmd tool on windows) command can verify if you are able to connect to s3.

from snowflake-connector-net.

CPavanKishore commented on August 29, 2024

i'm able to connect s3 bucket and get the data

i tried all the chunk URL's through postman @howryu

Check in the above picture

from snowflake-connector-net.

howryu commented on August 29, 2024

Hmm. Ok I need to look into it since this cannot reproduced in my environment.

Can you try to enable logging and pass me the log file? I have some docs in https://github.com/snowflakedb/snowflake-connector-net/blob/master/README.md

from snowflake-connector-net.

CPavanKishore commented on August 29, 2024

Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.RemoteDependency","time":"2018-01-31T07:52:46.0413218Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"rdddsc:2.4.1-1362","ai.operation.parentId":"|e17a9f54-413f5b2c42453c9b.","ai.operation.id":"e17a9f54-413f5b2c42453c9b","ai.operation.name":"GET Values/GetAsync","ai.application.ver":"1.0.0.0"},"data":{"baseType":"RemoteDependencyData","baseData":{"ver":2,"name":"POST /session/v1/login-request","id":"|e17a9f54-413f5b2c42453c9b.1.","data":"https://juniperdev.snowflakecomputing.com:443/session/v1/login-request?warehouse=EDP_WH&databaseName=EDP_DB&schemaName=&requestId=7aa6857c-deaa-46e2-9479-c0df8bbf7fad","duration":"00:00:02.0985810","resultCode":"200","success":true,"type":"Http","target":"juniperdev.snowflakecomputing.com","properties":{"AspNetCoreEnvironment":"Development","DeveloperMode":"true"}}}}
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.RemoteDependency","time":"2018-01-31T07:52:46.7594122Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"rdddsc:2.4.1-1362","ai.operation.parentId":"|e17a9f54-413f5b2c42453c9b.","ai.operation.id":"e17a9f54-413f5b2c42453c9b","ai.operation.name":"GET Values/GetAsync","ai.application.ver":"1.0.0.0"},"data":{"baseType":"RemoteDependencyData","baseData":{"ver":2,"name":"POST /queries/v1/query-request","id":"|e17a9f54-413f5b2c42453c9b.2.","data":"https://juniperdev.snowflakecomputing.com:443/queries/v1/query-request?requestId=7297efe3-d13b-45c0-b8e6-ff80c4710139","duration":"00:00:00.5269655","resultCode":"200","success":true,"type":"Http","target":"juniperdev.snowflakecomputing.com","properties":{"AspNetCoreEnvironment":"Development","DeveloperMode":"true"}}}}
Time Elapsed in Custom : 4314.7858'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.3\System.Core.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'dotnet.exe' (CoreCLR: clrhost): Loaded 'cxni12yu.hni'. Cannot find or open the PDB file.
'dotnet.exe' (CoreCLR: clrhost): Loaded 'hygaco1y.bm5'. Cannot find or open the PDB file.
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.Message","time":"2018-01-31T07:52:48.2592901Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"aspnet5c:2.1.1","ai.operation.parentId":"|e17a9f53-413f5b2c42453c9b.","ai.operation.id":"e17a9f53-413f5b2c42453c9b","ai.operation.name":"GET Home/Index","ai.application.ver":"1.0.0.0"},"data":{"baseType":"MessageData","baseData":{"ver":2,"message":"Executing ViewResult, running view at path /Views/Home/Index.cshtml.","severityLevel":"Information","properties":{"Path":"/Views/Home/Index.cshtml","CategoryName":"Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ViewResultExecutor","AspNetCoreEnvironment":"Development","DeveloperMode":"true","{OriginalFormat}":"Executing ViewResult, running view at path {Path}."}}}}
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\store\x64\netcoreapp2.0\microsoft.aspnetcore.webutilities\2.0.1\lib\netstandard2.0\Microsoft.AspNetCore.WebUtilities.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.RemoteDependency","time":"2018-01-31T07:52:48.7391938Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"rdddsc:2.4.1-1362","ai.operation.parentId":"|e17a9f54-413f5b2c42453c9b.","ai.operation.id":"e17a9f54-413f5b2c42453c9b","ai.operation.name":"GET Values/GetAsync","ai.application.ver":"1.0.0.0"},"data":{"baseType":"RemoteDependencyData","baseData":{"ver":2,"name":"GET /mgz0-s-ssca0741/results/4bca04e6-5be3-47cf-acaf-2aeda65c2b01/main/data_0_0_1","id":"|e17a9f54-413f5b2c42453c9b.3.","data":"https://sfc-ds1-customer-stage.s3.amazonaws.com:443/mgz0-s-ssca0741/results/4bca04e6-5be3-47cf-acaf-2aeda65c2b01/main/data_0_0_1?x-amz-server-side-encryption-customer-algorithm=AES256&response-content-encoding=gzip&AWSAccessKeyId=AKIAITDIMWYCMJHC6WAQ&Expires=1517406766&Signature=aHGT%2BfduoE9z9%2BR%2B9fyr4%2BLibYo%3D","duration":"00:00:01.7588694","resultCode":"200","success":true,"type":"Http","target":"sfc-ds1-customer-stage.s3.amazonaws.com","properties":{"AspNetCoreEnvironment":"Development","DeveloperMode":"true"}}}}
'dotnet.exe' (CoreCLR: clrhost): Loaded 'gb0zvkbd.mmk'. Cannot find or open the PDB file.
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.RemoteDependency","time":"2018-01-31T07:52:48.8229514Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"rdddsc:2.4.1-1362","ai.operation.parentId":"|e17a9f54-413f5b2c42453c9b.","ai.operation.id":"e17a9f54-413f5b2c42453c9b","ai.operation.name":"GET Values/GetAsync","ai.application.ver":"1.0.0.0"},"data":{"baseType":"RemoteDependencyData","baseData":{"ver":2,"name":"GET /mgz0-s-ssca0741/results/4bca04e6-5be3-47cf-acaf-2aeda65c2b01/main/data_0_0_0","id":"|e17a9f54-413f5b2c42453c9b.4.","data":"https://sfc-ds1-customer-stage.s3.amazonaws.com:443/mgz0-s-ssca0741/results/4bca04e6-5be3-47cf-acaf-2aeda65c2b01/main/data_0_0_0?x-amz-server-side-encryption-customer-algorithm=AES256&response-content-encoding=gzip&AWSAccessKeyId=AKIAITDIMWYCMJHC6WAQ&Expires=1517406766&Signature=DjCN5ct46udjvcKNEFdgU8CiVhw%3D","duration":"00:00:01.8426168","resultCode":"200","success":true,"type":"Http","target":"sfc-ds1-customer-stage.s3.amazonaws.com","properties":{"AspNetCoreEnvironment":"Development","DeveloperMode":"true"}}}}
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.3\Microsoft.CSharp.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'dotnet.exe' (CoreCLR: clrhost): Loaded 'u1zvlzv1.ln0'. Cannot find or open the PDB file.
'dotnet.exe' (CoreCLR: clrhost): Loaded '2bcurmqa.2in'. Cannot find or open the PDB file.
'dotnet.exe' (CoreCLR: clrhost): Loaded '3vpp3n2o.jbq'. Cannot find or open the PDB file.
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.Message","time":"2018-01-31T07:52:49.9292117Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"aspnet5c:2.1.1","ai.operation.parentId":"|e17a9f53-413f5b2c42453c9b.","ai.operation.id":"e17a9f53-413f5b2c42453c9b","ai.operation.name":"GET Home/Index","ai.application.ver":"1.0.0.0"},"data":{"baseType":"MessageData","baseData":{"ver":2,"message":"Executed action LegalAnalyticsPlatform.Controllers.HomeController.Index (LegalAnalyticsPlatform) in 8922.2328ms","severityLevel":"Information","properties":{"CategoryName":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","AspNetCoreEnvironment":"Development","DeveloperMode":"true","ElapsedMilliseconds":"8922.2328","{OriginalFormat}":"Executed action {ActionName} in {ElapsedMilliseconds}ms","ActionName":"LegalAnalyticsPlatform.Controllers.HomeController.Index (LegalAnalyticsPlatform)"}}}}
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.Message","time":"2018-01-31T07:52:49.9392628Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"aspnet5c:2.1.1","ai.operation.parentId":"|e17a9f53-413f5b2c42453c9b.","ai.operation.id":"e17a9f53-413f5b2c42453c9b","ai.operation.name":"GET Home/Index","ai.application.ver":"1.0.0.0"},"data":{"baseType":"MessageData","baseData":{"ver":2,"message":"Request finished in 9584.2819ms 200 text/html; charset=utf-8","severityLevel":"Information","properties":{"CategoryName":"Microsoft.AspNetCore.Hosting.Internal.WebHost","StatusCode":"200","AspNetCoreEnvironment":"Development","DeveloperMode":"true","ContentType":"text/html; charset=utf-8","ElapsedMilliseconds":"9584.2819"}}}}
Application Insights Telemetry (unconfigured): {"name":"Microsoft.ApplicationInsights.Dev.Request","time":"2018-01-31T07:52:40.3087757Z","tags":{"ai.cloud.roleInstance":"pkreddy-T440","ai.location.ip":"::1","ai.internal.nodeName":"pkreddy-T440","ai.internal.sdkVersion":"aspnet5c:2.1.1","ai.operation.id":"e17a9f53-413f5b2c42453c9b","ai.operation.name":"GET Home/Index","ai.application.ver":"1.0.0.0"},"data":{"baseType":"RequestData","baseData":{"ver":2,"id":"|e17a9f53-413f5b2c42453c9b.","name":"GET Home/Index","duration":"00:00:09.6721309","success":true,"responseCode":"200","url":"http://localhost:50779/","properties":{"AspNetCoreEnvironment":"Development","DeveloperMode":"true","httpMethod":"GET"}}}}
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.3\Microsoft.Win32.Primitives.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
An unhandled exception of type 'Snowflake.Data.Client.SnowflakeDbException' occurred in System.Private.CoreLib.dll
External component has thrown an exception.

'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.3\System.Diagnostics.StackTrace.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
The program '[14096] dotnet.exe' has exited with code 0 (0x0).

is this the one which you asked ..?

from snowflake-connector-net.

howryu commented on August 29, 2024

@CPavanKishore No. I mean the log file generated by .NET Driver.
Here is the example https://github.com/snowflakedb/snowflake-connector-net/blob/master/Snowflake.Data.Tests/App.config on how to config to enable logging.

More importantly, I probably need the network trace to understand why you cannot connect to S3.
https://docs.microsoft.com/en-us/dotnet/framework/network-programming/how-to-configure-network-tracing

The example I provided (App.Config) also includes the network trace. Can you generate the trace to see what's going on?

from snowflake-connector-net.

CPavanKishore commented on August 29, 2024

i tried but common.logging supports up to .netcore1.0 it doesn't support 2.0,Can you try to retrieve a large amount of data from your side to check

from snowflake-connector-net.

howryu commented on August 29, 2024

I compiled the .net driver against .net framework 4.6, but not .net core (there is a plan porting it)
#7
So I cannot make sure if it worked with .net core or not.

I do have a large result tests in
https://github.com/snowflakedb/snowflake-connector-net/blob/master/Snowflake.Data.Tests/SFDbCommandIT.cs
which runs successfully on appveyor.

from snowflake-connector-net.

CPavanKishore commented on August 29, 2024

Can i get any paid session to configure this snowflake connector in my project @howryu ....?

or any alternatives for this ..?

from snowflake-connector-net.

howryu commented on August 29, 2024

This is already fixed in #10. More specifically, in

snowflake-connector-net/Snowflake.Data/Core/SFChunkDownloader.cs

Line 125 in 45d1a0f

timeout = TimeSpan.FromHours(1),

Reason is because the http request timeout is in milliseconds. It should be 1 hour (3600 * 1000 millis) in stead of (3600 millis).

from snowflake-connector-net.

while retrieving chunks of data through URL it was throwing an exception about snowflake-connector-net HOT 12 CLOSED

Comments (12)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent