Comments (11)
actually, bring us to the question - how do we exclude files from cp/sync ?
from s5cmd.
aws/aws-sdk-go#2653 PR is required to address this issue.
from s5cmd.
Just checking if there is any news on this issues. I could make use of this feature in my workflow. Not sure that PR @igungor mentioned is merged/ what's stopping that merge.
from s5cmd.
Is there any update on the status of this issue?
from s5cmd.
Any updates?
from s5cmd.
Ran into this today - any updates on this or is there a work-around ?
from s5cmd.
I have also just encountered this issue, has something changed?
Note: this is between s3 buckets and not from EBS to S3
from s5cmd.
Thanks for the report. We're using the official AWS SDK. Copying objects (from S3 to S3) larger than 5GB
requires multipart uploading which we currently don't leverage.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
You create a copy of your object up to 5 GB in size in a single atomic operation using this API. However, for copying an object greater than 5 GB, you must use the multipart upload Upload Part - Copy API.
We're going to use the multipart Copy API to support large file transfers between S3 buckets. Please note that uploading a large file to S3 works as expected.
from s5cmd.
Similar problem is with sync
when file is larger than 5G.
from s5cmd.
case:
source - has objects
dest - is empty
s3cmd sync source/* dest/
"sync" fails silently on the first run after copying some objects.
"sync" will fail on the second run and report the offending object
from s5cmd.
some other go libs support this
ie
rclone/rclone@7b1274e
or
drakkan/sftpgo@6f8b71b#diff-53095c9aca755cd4fe0455d2549f3668df445ac69801f752b2371bcd8a4c5efaR810
from s5cmd.
Related Issues (20)
- s5cmd ls --region argument, and implied lists respect the cp argument
- Facilitating Python Usage: Proposal for s5cmd Wheels HOT 2
- README.md incorrect example "you can compress the data before uploading" HOT 1
- Path globbing syntax is not documented HOT 3
- README.md internal link is incorrect
- rm doesn't delete 0-byte folder placeholders
- Region setting ignored in credentials file (and awscli doesn't need the setting at all)
- Copying object onto itself with metadata fails
- sync command copies a new object everytime when the s3 version is on GLACIER
- run input syntax is not documented HOT 2
- "Beast mode" example is incorrect HOT 1
- remote-to-remote copy with --if-source-newer uses wrong region
- Support cat with glob
- Please create a new release with updated `github.com/lanrat/extsort` version HOT 2
- Parameters --no-clobber, -n with cp is working in a wrong way.
- Dualstack (IPv6) endpoint doesn't work
- Strange behaviour of sync HOT 1
- Global options and "Run multiple commands in parallel" feature HOT 1
- Error on empty bucket listing
- Support glob character in destination object?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from s5cmd.