Add example of accessing S3 files about sergeant HOT 3 OPEN

hrbrmstr commented on June 3, 2024 2

Add example of accessing S3 files

from sergeant.

Comments (3)

davidski commented on June 3, 2024 1

If your example could include referencing a specific regional endpoint, that would be A++ good. My first attempt at getting IAM roles and a regionally scoped bucket call to work failed and I've yet to go back and make another attempt.

from sergeant.

hrbrmstr commented on June 3, 2024

I gave it a quick try the day 1.15.0 came out but didn't go back to it.

from sergeant.

davidski commented on June 3, 2024

Looks like I just needed to come back to this. Got this working on a us-west-2 S3 endpoint with the following (excessively verbose) storage configuration:

{
  "type": "file",
  "connection": "s3a://cloudy-mccloudface",
  "config": {
    "fs.s3a.aws.credentials.provider": "com.amazonaws.auth.InstanceProfileCredentialsProvider",
    "fs.s3a.endpoint": "s3.us-west-2.amazonaws.com"
  },
  "workspaces": {
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "root": {
      "location": "/",
      "writable": false,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "csvs": {
      "location": "/csvs",
      "writable": false,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    }
  },
  "formats": {
    "psv": {
      "type": "text",
      "extensions": [
        "tbl"
      ],
      "delimiter": "|"
    },
    "csv": {
      "type": "text",
      "extensions": [
        "csv"
      ],
      "delimiter": ","
    },
    "tsv": {
      "type": "text",
      "extensions": [
        "tsv"
      ],
      "delimiter": "\t"
    },
    "parquet": {
      "type": "parquet"
    },
    "json": {
      "type": "json",
      "extensions": [
        "json"
      ]
    },
    "avro": {
      "type": "avro"
    },
    "sequencefile": {
      "type": "sequencefile",
      "extensions": [
        "seq"
      ]
    },
    "csvh": {
      "type": "text",
      "extensions": [
        "csvh"
      ],
      "extractHeader": true,
      "delimiter": ","
    }
  },
  "enabled": true
}

A bit too shagged out to write this up properly right now, so dumping the config as a reminder for later.

from sergeant.

Recommend Projects