Comments (3)
Thanks for raising the issue, @amol-anand !
This looks like a default limit of the underlying storage, but ideally the SDK should handle the operation in the most appropriate manner.
cc @purplecabbage @sandeep-paliwal @moritzraho for feedback
from aio-lib-files.
JIRA issue created: https://jira.corp.adobe.com/browse/ACNA-946
from aio-lib-files.
After some investigation I can conclude that there are two issues here:
-
- Azure returns max 5000 files when listing,
list
is internally used bydelete
andcopy
which explains the bug withdelete
. We need to support storage with more than 5000 files, we should expose some sort of pagination and need to fix the returned number of files inlist
based on the memory constraints in runtime actions.files.delete('/')
should still delete all files but internally list with pagination once we support this. Some changes to handle > 5000 files will be required byfiles.copy
too.
- Azure returns max 5000 files when listing,
-
- Runtime action crashes when handling many files (5000) because execution hits resource limits both for memory (default is 256 mb) and number of opened file descriptors for open requests to Azure. To solve this, delete and copy should handle files in fix size batches (like 200 or so).
from aio-lib-files.
Related Issues (20)
- update node engine support to 12 or greater HOT 2
- test failures with new [email protected] dependency HOT 1
- codecov in Github Action needs to be updated HOT 1
- Typescript error
- Support private-only container for BYO credentials HOT 1
- document what `public` means for files HOT 1
- document expiryInSeconds
- Enhancement request for add support for listing files with filter and option to get number of files in a folder HOT 1
- allow some calls to use wildcard glob matching to search HOT 2
- append mode HOT 5
- List should support pagination with a documented default and max limit HOT 1
- azure v12 should have fixed getAccessPolicy, replace custom code with azure sdk call HOT 2
- Support presignUrls on public files or explicitely throw an error HOT 3
- presignUrl with write permissions and BYO credentials fail on PUT Blob HOT 1
- Introduce a `move` function HOT 10
- Generated presigned URLs do not work in the browser HOT 4
- presignedUrl file path must already exist HOT 1
- Azure blob store on-the-fly gzip compression HOT 3
- Allow all origins for presigned URL HOT 10
- CDN domain should be returned by the TVM HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aio-lib-files.