adgoji / onyx-amazon-s3 Goto Github PK

Onyx plugin for Amazon S3.

In your project file:

[org.onyxplatform/onyx-amazon-s3 "0.9.10.0"]

In your peer boot-up namespace:

(:require [onyx.plugin.s3-output])

Catalog entry:

{:onyx/name <<TASK_NAME>>
 :onyx/plugin :onyx.plugin.s3-output/output
 :s3/bucket <<BUCKET_NAME>>
 :s3/serializer-fn :clojure.core/pr-str
 :s3/key-naming-fn :onyx.plugin.s3-output/default-naming-fn
 :onyx/type :output
 :onyx/medium :s3
 :onyx/batch-size 20
 :onyx/doc "Writes segments to s3 files, one file per batch"}

Segments received by this task will be serialized by the :s3/serializer-fn, into a file per batch, placed at a key in the bucket which is named via the function defined at :s3/key-naming-fn. This function takes an event map and returns a string. Using the default naming function, :onyx.plugin.s3-output/default-naming-fn, will name keys in the following format in UTC time format: "yyyy-MM-dd-hh.mm.ss.SSS_batch_BATCH_UUID".

Lifecycle entry:

{:lifecycle/task <<TASK_NAME>>
 :lifecycle/calls :onyx.plugin.s3-output/s3-output-calls}

Many thanks to AdGoji for allowing this work to be open sourced and contributed back to the Onyx Platform community.

Pull requests into the master branch are welcomed.

Copyright © 2016 Distributed Masonry LLC

Distributed under the Eclipse Public License, the same as Clojure.