dmolesuc / cos Goto Github PK

For very large key lists, it would be helpful to have some sense of throughput.

Writes fail with Object Not Found (see ncw/swift #134.

It looks like 404 for newly-created DLOs is expected behavior. Does that imply we should be able to recover from this error on our end?

Failing that, we can dig through the Swift docs and try to figure out whether we can somehow (separately) initialize an empty DLO before we try uploading to it.

The current set of key lists is fairly random and ad-hoc:

1. Default: default source (as naughty-strings, filtering out "", ".", and leading "..", all of which are known to fail in Amazon S3) (500 keys)
2. disallow-backslash: as default source, disallowing backlash (320 keys)
3. disallow-double-backslash: as default source, disallowing double backlash (498 keys)
4. misc: miscellenous potential problems, incl. path elements & unicode blocks (612 keys)
5. naughty-strings: Big List of Naughty Strings (504 keys)

The naughty-strings list is fairly reasonable, although not all that directly applicable to filenames; it tends to produce a lot of redundant failures since many of its Javascript escapes and whatnot are very similar. A better, shorter list could probably be made by boiling it down to more general cases.

The misc list is pretty ad-hoc and doesn't cover every Unicode code block, or even most of them.

We should come up with some systematic cases and some systematic lists based on those cases.

Steps to reproduce:

Execute the command below:

cos crvd -v s3://<BUCKET>/ --endpoint https://s3.us-west-2.amazonaws.com/ --size 1T

Expected:

Upload completes in about 4 hours (assuming a fast client network)

Actual

Upload fails at 50G with:

Error: MultipartUpload: upload multipart failed
	upload id: iuCCRxEvNJGjfH_ZoA30A1JyUqUxrZkyGJpymOIb8VWq1Yb.ysFPqMaGZQgvRDR4PjFdgsoKn8TvH.ZfoKTdyRMTkx442X9_gD8N5oMiHyLtoVSlPm_nfxY1o3Km.42le_ZTxp_1ZYwpOoqb4SbWcQ--
caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit

It's all very well to discover 189 invalid keys, but it doesn't tell you just what made each key invalid.

Passing - as a file argument should read keys from standard input, e.g.

cat mykeylist.txt | cos keys s3://uc3-s3mrt5001-stg/ \
  -e https://s3-us-west-2.amazonaws.com/ \
  --file -

If we want to do this right, we should probably put KeyList take a passed-in function and iterate over its own keys rather than having it expose a []string. This poses problems for KeyList.Count, but we don't use KeyList.Count much except for documentation and for final output.

We should record a version number & release date, and output both in the help text.

At least for cos suite, it's getting fairly difficult to sort out endpoints, URLs, etc. from flags.

This would also let us get out of using a fake swift:// protocol in the bucket or object URL.

There may be situations in which uname -a reports a non-EC2 system, but a user still wants EC2 role credentials. (See this comment for an example.)

• don't log progress by default
• separate --verbose / -v and --very-verbose / -vv log levels
  • --verbose would include progress, IAM auth errors, other stuff? (go through uses of log.Detail and Detailf)
  • --very-verbose would include extra debug output like parsed parameters etc.

• use case in Kubernetes
• plan to use https://github.com/spf13/viper

Currently the keys command outputs only either raw keys or quoted Go string literals. The first is problematic for keys with newlines or control characters; the second is Go-specific and somewhat confusing for anything involving a backslash; and also still exposes some sequences that can be problematic for viewing in a terminal, such as ANSI color escapes.

Offering strconv.QuoteToASCII as an option would mitigate some of the problems with the latter case, but probably shouldn't be the only additional option. What else makes sense—URL encoding? MIME quoted-printable? Base64?

Swift objects are read and hashed in 4K chunks, using a lot of CPU, and making the check process CPU-bound rather than IO-bound on fast connections.

This could be inherent to the Swift library we're using, or just Golang default behavior (see ncw/swift #132). But we can probably mitigate it by building up a larger slice before we pass it to the hash.

After updating to go 1.12, spinner doesn't spin and there's some cursor weirdness. Final output is OK, though.

Currently the key lists are all hard-coded; there should be a --keyfile option that reads from a file.

There are a couple of B2 libraries for Go out there:

• https://github.com/kurin/blazer
• https://github.com/kothar/go-backblaze

At some point it would be nice to add B2 support.