Comments (29)
adam-fowler
commented on May 26, 2024
1
Ok one possibility is a thread pool is being created every time you call that function and this is causing issues. Instead you could create a global NIOThreadPool and pass it to s3. multipartUpload as threadPoolProvider: .shared(myThreadPool).
Recently swift-nio supplied a singleton version of the NIOThreadPool. I can change the default to use this singleton instead of creating a new one every time the function is called
from soto.
adam-fowler
commented on May 26, 2024
1
PR for using singleton instead #695
from soto.
adam-fowler
commented on May 26, 2024
1
While waiting for a new release you should be able to use threadPoolProvider: .shared(NIOThreadPool.singleton) in your multipartUpload call to resolve your issue.
from soto.
adam-fowler
commented on May 26, 2024
1
As a test can you stick the following lines somewhere in the initialisation for your application
DispatchQueue.global(qos: .background).async {
_ = NIOThreadPool.singleton
}from soto.
adam-fowler
commented on May 26, 2024
1
I think the issue you were seeing was related to the fact each HTTPClient you create, creates a new EventLoopGroup and a new connection pool. Your application was creating an HTTPClient for every request which could cause a thread explosion.
from soto.
adam-fowler
commented on May 26, 2024
1
6.8 has been released with the singleton stuff
from soto.
adam-fowler
commented on May 26, 2024
What QoS is calling code running at?
from soto.
nastasiupta
commented on May 26, 2024
On the main...
from soto.
nastasiupta
commented on May 26, 2024
@adam-fowler new version release ?
from soto.
adam-fowler
commented on May 26, 2024
The PR needs approved first
from soto.
nastasiupta
commented on May 26, 2024
@adam-fowler can you solve today?
from soto.
nastasiupta
commented on May 26, 2024
Hmm, I tried but is not solving the issue:
Thread Performance Checker: Thread running at User-interactive quality-of-service class waiting on a lower QoS thread running at Default quality-of-service class. Investigate ways to avoid priority inversions
PID: 39945, TID: 5500604
Backtrace
=================================================================
3 AppNameHere 0x0000000104d6f540 $s8NIOPosix13NIOThreadPoolC6_start16threadNamePrefixySS_tF + 1524
4 AppNameHere 0x0000000104d6eed4 $s8NIOPosix13NIOThreadPoolC021_makePerpetualStartedC015numberOfThreads16threadNamePrefixACSi_SStFZ + 104
5 AppNameHere 0x0000000104da30cc $s8NIOPosix23globalPosixBlockingPool33_54795181D6A4669A6950FD635B385752LLAA09NIOThreadE0CvpfiAEyXEfU_ + 132
6 AppNameHere 0x0000000104da3038 $s8NIOPosix23globalPosixBlockingPool33_54795181D6A4669A6950FD635B385752LL_WZ + 12
7 libdispatch.dylib 0x0000000107c02b4c _dispatch_client_callout + 20
8 libdispatch.dylib 0x0000000107c04a64 _dispatch_once_callout + 160
9 AppNameHere 0x0000000104da2e68 $s8NIOPosix23globalPosixBlockingPool33_54795181D6A4669A6950FD635B385752LLAA09NIOThreadE0Cvau + 80
10 AppNameHere 0x0000000104da2dfc $s7NIOCore13NIOSingletonsO8NIOPosixE23posixBlockingThreadPoolAD09NIOThreadG0CvgZ + 16
11 AppNameHere 0x0000000104da2ed0 $s8NIOPosix13NIOThreadPoolC9singletonACvgZ + 24
12 AppNameHere 0x000000010426f354 $s9AppNameHere19AWSFileUploadWorkerC7execute4data10completionyAC7RequestV_ySSSg_AC8ResponseVSgtcSgtF + 4140
13 AppNameHere 0x00000001040d678c $s9AppNameHere16FileUploadWorkerC7execute4data10completionyAC7RequestC_ySSSg_AC8ResponseVSgtcSgtF05startcD0L_yyF + 536
14 AppNameHere 0x00000001040d651c $s9AppNameHere16FileUploadWorkerC7execute4data10completionyAC7RequestC_ySSSg_AC8ResponseVSgtcSgtF + 984
15 AppNameHere 0x0000000104388a98 $s9AppNameHere13LFFileManagerC18uploadPendingFiles33_A43F4D69DB085DD12FA6694232CB9D21LLyyF6handleL_0E7RequestyAA16FileUploadWorkerC0Q0C_tF + 348
16 AppNameHere 0x0000000104388910 $s9AppNameHere13LFFileManagerC18uploadPendingFiles33_A43F4D69DB085DD12FA6694232CB9D21LLyyF + 344
17 AppNameHere 0x00000001043805f4 $s9AppNameHere13LFFileManagerC11uploadFiles4urlsySay10Foundation3URLVG_tF + 1152
18 AppNameHere 0x0000000104312d00 $s9AppNameHere21FileManagerHeaderViewV4bodyQrvg7SwiftUI05TupleF0VyAE0F0PAEE7paddingyQrAE4EdgeO3SetV_12CoreGraphics7CGFloatVSgtFQOyAiEE5frame5width6height9alignmentQrAR_ArE9AlignmentVtFQOyAE6HStackVyAGyAE06ScrollF0VyAZyAGyAC09componentF033_FF7633ACFDEA3379B9E4F13A23535CF6LL4text10isSelected12onTapGestureQrAA10BaseStringO_SbyyctFQOy_Qo__AE7ForEachVySayAA6FolderVGSSA8_GtGGGSg_AE6SpacerVAGyAA11ImageButtonVSg_A23_A22_tGSgAiEEAstuVQrAR_ArXtFQOyA20__Qo_A22_A22_A26_AE6VStackVyAiEE12cornerRadius_11antialiasedQrAQ_SbtFQOyAZyAGyAiEE11buttonStyleyQrqd__AE20PrimitiveButtonStyleRd__lFQOyAE6ButtonVyAiEE10background_AVQrqd___AXtAeHRd__lFQOyAiEEAstuVQrAR_ArXtFQOyAiEE15foregroundColoryQrAE5ColorVSgFQOyAE5ImageV_Qo__Qo__A38_Qo_G_AE16PlainButtonStyleVQo__A48_tGG_Qo_GtGG_Qo__Qo__AiEEAJyQrAN_ARtFQOyAiEEAstuVQrAR_ArXtFQOyAZyAGyAiEEA31_yQrqd__AEA32_Rd__lFQOyA34_yA41_G_A47_Qo_Sg_AE19_ConditionalContentVyAE4TextVA63_GA20_AZyAGyAA15SearchTextFieldV_AiEE9fixedSizeQryFQOyAiEE9menuStyleyQrqd__AE9MenuStyleRd__lFQOyAE4MenuVyAiEEA29__A30_QrAQ_SbtFQOyAiEEA35__AVQrqd___AXtAeHRd__lFQOyAiEEAstuVQrAR_ArXtFQOyA28_yA61_yA41_A41_GG_Qo__A38_Qo__Qo_AGyA28_yAGyA63__A34_yAGyA72__A63_tGGA78_A78_A78_A78_AA09SeparatorF0VtGG_A82_A28_yAGyA63__A78_A78_tGGtGG_AE25BorderlessButtonMenuStyleVQo__Qo_AA15SecondaryButtonVA92_tGGtGG_Qo__Qo_tGyXEfU_A95_yXEfU0_A93_yXEfU0_yycfU3_ySayAA012ErrorOverlayF0V6ActionVGSg_Say10Foundation3URLVGSgtcfU_ + 1688
19 AppNameHere 0x000000010420e928 $s9AppNameHere17FilesPickerWorkerC7execute4data10completionyAA0cD13ConfigurationV_ySayAA16ErrorOverlayViewV6ActionVGSg_Say10Foundation3URLVGSgtcSgtF + 2376
20 AppNameHere 0x000000010431263c $s9AppNameHere21FileManagerHeaderViewV4bodyQrvg7SwiftUI05TupleF0VyAE0F0PAEE7paddingyQrAE4EdgeO3SetV_12CoreGraphics7CGFloatVSgtFQOyAiEE5frame5width6height9alignmentQrAR_ArE9AlignmentVtFQOyAE6HStackVyAGyAE06ScrollF0VyAZyAGyAC09componentF033_FF7633ACFDEA3379B9E4F13A23535CF6LL4text10isSelected12onTapGestureQrAA10BaseStringO_SbyyctFQOy_Qo__AE7ForEachVySayAA6FolderVGSSA8_GtGGGSg_AE6SpacerVAGyAA11ImageButtonVSg_A23_A22_tGSgAiEEAstuVQrAR_ArXtFQOyA20__Qo_A22_A22_A26_AE6VStackVyAiEE12cornerRadius_11antialiasedQrAQ_SbtFQOyAZyAGyAiEE11buttonStyleyQrqd__AE20PrimitiveButtonStyleRd__lFQOyAE6ButtonVyAiEE10background_AVQrqd___AXtAeHRd__lFQOyAiEEAstuVQrAR_ArXtFQOyAiEE15foregroundColoryQrAE5ColorVSgFQOyAE5ImageV_Qo__Qo__A38_Qo_G_AE16PlainButtonStyleVQo__A48_tGG_Qo_GtGG_Qo__Qo__AiEEAJyQrAN_ARtFQOyAiEEAstuVQrAR_ArXtFQOyAZyAGyAiEEA31_yQrqd__AEA32_Rd__lFQOyA34_yA41_G_A47_Qo_Sg_AE19_ConditionalContentVyAE4TextVA63_GA20_AZyAGyAA15SearchTextFieldV_AiEE9fixedSizeQryFQOyAiEE9menuStyleyQrqd__AE9MenuStyleRd__lFQOyAE4MenuVyAiEEA29__A30_QrAQ_SbtFQOyAiEEA35__AVQrqd___AXtAeHRd__lFQOyAiEEAstuVQrAR_ArXtFQOyA28_yA61_yA41_A41_GG_Qo__A38_Qo__Qo_AGyA28_yAGyA63__A34_yAGyA72__A63_tGGA78_A78_A78_A78_AA09SeparatorF0VtGG_A82_A28_yAGyA63__A78_A78_tGGtGG_AE25BorderlessButtonMenuStyleVQo__Qo_AA15SecondaryButtonVA92_tGGtGG_Qo__Qo_tGyXEfU_A95_yXEfU0_A93_yXEfU0_yycfU3_ + 372
21 SwiftUI 0x00000001b6842188 OUTLINED_FUNCTION_2 + 900
22 SwiftUI 0x00000001b5ed8838 __swift_memcpy3_1 + 10188
23 SwiftUI 0x00000001b5edc760 __swift_memcpy3_1 + 26356
24 SwiftUI 0x00000001b5edc6ac __swift_memcpy3_1 + 26176
25 SwiftUI 0x00000001b64c52e8 OUTLINED_FUNCTION_0 + 5048
26 SwiftUI 0x00000001b530d558 OUTLINED_FUNCTION_37 + 1502636
27 SwiftUI 0x00000001b5fe5bd0 OUTLINED_FUNCTION_8 + 760
28 SwiftUI 0x00000001b606c0d0 OUTLINED_FUNCTION_2 + 40
29 SwiftUI 0x00000001b5fe5bd0 OUTLINED_FUNCTION_8 + 760
30 SwiftUI 0x00000001b682fd84 OUTLINED_FUNCTION_4 + 4424
31 SwiftUI 0x00000001b682f53c OUTLINED_FUNCTION_4 + 2304
32 SwiftUI 0x00000001b69dd1e4 OUTLINED_FUNCTION_3 + 1732
33 SwiftUI 0x00000001b5deadd0 OUTLINED_FUNCTION_16 + 32804
34 SwiftUI 0x00000001b5deb428 OUTLINED_FUNCTION_16 + 34428
35 SwiftUI 0x00000001b5df47c0 OUTLINED_FUNCTION_16 + 72212
36 SwiftUI 0x00000001b5df45dc OUTLINED_FUNCTION_16 + 71728
37 AppKit 0x000000018ec0de24 _routeMouseUpEvent + 132
38 AppKit 0x000000018e220b80 -[NSWindow(NSEventRouting) _reallySendEvent:isDelayedEvent:] + 384
39 AppKit 0x000000018e22082c -[NSWindow(NSEventRouting) sendEvent:] + 284
40 AppKit 0x000000018e8c7c08 -[NSApplication(NSEventRouting) sendEvent:] + 1556
41 AppKit 0x000000018e51b1bc -[NSApplication _handleEvent:] + 60
42 AppKit 0x000000018e0e8460 -[NSApplication run] + 512
43 AppKit 0x000000018e0bf708 NSApplicationMain + 880
44 AppNameHere 0x000000010447ea34 $sSo21NSApplicationDelegateP6AppKitE4mainyyFZ + 40
45 AppNameHere 0x000000010447e9fc $s9AppNameHere11AppDelegateC5$mainyyFZ + 44
46 AppNameHere 0x000000010447ebfc main + 28
47 dyld 0x000000018a4c5058 start + 2224
from soto.
adam-fowler
commented on May 26, 2024
Ok, It looks like a similar issue apple/swift-nio#2223 I've seen on the swift-nio repository. The suggested way to resolve this was initialise the singleton on a background thread. This is going to require a little more work to get rid of the warning.
Does the upload still fail after seeing the warning above?
from soto.
nastasiupta
commented on May 26, 2024
Yes... I mean is not failing (calling the completion function with an error), it's just stuck...
from soto.
nastasiupta
commented on May 26, 2024
@adam-fowler tested... in the debugger there is no warning displayed, but the upload process is stuck..
from soto.
adam-fowler
commented on May 26, 2024
@adam-fowler tested... in the debugger there is no warning displayed, but the upload process is stuck..
Ok looks like we have two issues here. The warning about thread priorities and then the hang.
For the hang can we clean the code up a little. And also add some logging to get a better idea what is going on
- The
AWSClientshould be global. You shouldn't create one for each upload operation. - With the current setup you must be calling
AWSClient.syncShutdownsomewhere otherwise you would get an error when the client is deleted. Where is this being called? If it is immediately after themultipartUploadcall then that'll be the probable cause of your issues.multipartUploadstarts an asynchronous process. That process is not complete when it returns. So callingsyncShutdownimmediately after will kill the process before it completes. - Lets pass a Logger to the multipartUpload operation to see if we get any output.
var logger = Logger(label: "s3hang")
logger.logLevel = .trace
// Your s3 multipart upload call
s3.multipartUpload(request, ..., logger: logger)- We can also get Soto to output details about every request sent to AWS
s3 = s3.with(middlewares: AWSLoggingMiddleware()from soto.
nastasiupta
commented on May 26, 2024
@adam-fowler I sent you an email with the logs file. Also, regarding the first point "The AWSClient should be global. You shouldn't create one for each upload operation." I might have an issue. The app supports multiple accounts and it can happen to have a different configuration for each account. Even if I optimise for use a single instance for an user, if there 10 accounts connected on the same app, I want to use different instances because each user has his own configuration.
from soto.
adam-fowler
commented on May 26, 2024
Are you uploading all of these at the same time? You might want to queue them. You could be opening too many connections.
from soto.
nastasiupta
commented on May 26, 2024
Maximum 5 uploads. So the user can select 100 files, but I take care to have 5 uploads at the same time, when one is ended, then I start another one. Do you think 5 uploads at a time maybe be to much?
It worked perfect for a long period of time... things are broken recenly
from soto.
adam-fowler
commented on May 26, 2024
Just looking through the logs you sent.
- From the log I only see 18 files being loaded. It appears the other 2 don't get loaded at all. Are you sure you are calling the multipart upload 20 times?
- Of those 18 uploads only one is larger than 5MB and might benefit from multipartUpload over putObject.
from soto.
nastasiupta
commented on May 26, 2024
Yes, all the files goes to multipart upload. In previous versions we had made some "hard core tests" like uploading more than 100+ files... selecting like 5 big videos and images, after the batch is ready, start again with other images and so on. Something happen in the last Soto SDK updates... We released the app in 2021 and we didn't had any use in the previous releases.
We use multipart because in 99% of the cases our users are having images bigger than 5mb :).
from soto.
adam-fowler
commented on May 26, 2024
I only see 18 Open file ... log entries. This is output as soon as the multipart upload function has opened a file ie the first operation it performs. Can you please verify the number of times you are calling the function by adding a log entry prior to calling multipartUpload.
from soto.
nastasiupta
commented on May 26, 2024
Check email, I sent you some screenshots as well and a new logs file, this time stopped sooner...
from soto.
adam-fowler
commented on May 26, 2024
Are you able to use a branch of Soto. If so could you use the threadpool-singleton branch. And send me the logs from that.
from soto.
adam-fowler
commented on May 26, 2024
Also if you aren't able to use one AWSClient could you at least use just the one HTTPClient instead of creating a new one with each AWSClient.
import AsyncHTTPClient
let globalHTTPClient = HTTPClient(eventLoopGroupProvider: .singleton)
let awsClient = AWSClient(..., httpClientProvider: .shared(globalHTTPClient))from soto.
nastasiupta
commented on May 26, 2024
Tried both, using singleton for HTTPClient cause a crash, I sent you an email with more details & logs
from soto.
adam-fowler
commented on May 26, 2024
Sorry I wasn't clear enough globalHTTPClient has to be a global variable. The idea is there is one HTTP client used by all of the AWSClients.
from soto.
nastasiupta
commented on May 26, 2024
I did as you said, tried 3 tests with 20 images each time, works like a charm.
from soto.
nastasiupta
commented on May 26, 2024
Oh wow, hmmm, should I wait for the SDK update before doing a new release? Maybe a mechanism, by default, to prevent things like this to happen?
from soto.
Related Issues (20)
- Add MachineLearning middleware for custom Predict endpoint
- "stream ended at an unexpected time" error when downloading objects. HOT 7
- Using Cloudflare R2 prompts STREAMING-AWS4-HMAC-SHA256-PAYLOAD not implemented HOT 2
- KMS throws fatal runtime error (Fatal error: Dictionary literal contains duplicate keys) HOT 1
- Add Cocoapods support HOT 5
- I use aws-cli get eks token like "aws eks get-token --cluster-name xxx", but i can't find it this, so i missed or not exist. HOT 7
- Multiplepart upload pause, resume and stop state manage HOT 4
- When I am trying to upload video using VPN HOT 5
- Textract.ValueType fails to parse HOT 9
- Can you consider to support cocoapods HOT 2
- Multiple fails uploads from Network lost issues causing CredentialProviderError.noProvider error to be thrown HOT 9
- EXC_BAD_ACCESS KERN_PROTECTION_FAILURE during s3FileTransferManager.copy HOT 4
- I am getting below error while uploading more then 8 images on same time. HOT 1
- How to Optimizing AWS S3 File Downloads for Multiple Objects in a Single Request HOT 1
- Issue with selectObjectContent SQL query HOT 2
- [7.x.x] watchOS : POSIXErrorCode(rawValue: 50): Network is down HOT 2
- Concurrently download multiple s3 "multipart" parts HOT 1
- Explicitly specify dependencies for each target/module HOT 3
- 7.x: Multipart upload progress is wrong when resuming a multipart upload
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from soto.