target-jsonl-blob's Introduction

target-jsonl-blob

JSONL Singer target for local storage, S3, GCS and Azure Blob Storage.

Installation

To install this Singer tap, you can download a prebuilt binary, or you can build it from source.

Configuration

Setting	Required	Default	Description
`bucket`	Yes	-	Blob storage bucket URL
`key_template`	No	`{{.StreamName}}.jsonl`	Template string for file keys

Bucket URLs

Storage	Example URL	Supported URL parameters
local	`file:///path/to/directory`	See supported parameters
S3	`s3://my-bucket`	See supported parameters
Azure	`azblob://my-container`	See supported parameters
GCS	`gs://my-bucket`	See supported parameters

Available fields for `key_template`

StreamName
Date (YYYY-MM-DD)
TimestampSeconds
Minute
Hour
Day
Month
Year

Example: {{.StreamName}}/{{.Year}}/{{.Month}}/{{.Day}}/{{.Hour}}/{{.Minute}}/{{.StreamName}}.jsonl

Build from source

go build -o target-jsonl-blob

Usage with Meltano

Download the appropriate asset
Add a custom Meltano plugin to your project
Run a pipeline

1. Download the appropriate asset

You can see the full list of assets in the release page: https://github.com/MeltanoLabs/target-jsonl-blob/releases/latest.

The gh tool makes downloading an asset easy:

gh release download v0.0.6 \
  -R MeltanoLabs/target-jsonl-blob \
  -p '*darwin_amd64*' \
  --clobber

tar -xvf target-jsonl-blob_0.0.6_darwin_amd64.tar.gz target-jsonl-blob

You can also install with Homebrew:

brew tap MeltanoLabs/target-jsonl-blob https://github.com/MeltanoLabs/target-jsonl-blob
brew install target-jsonl-blob

2. Add a custom Meltano plugin to your project

# meltano.yml
plugins:
  loaders:
  - name: target-jsonl-blob
    namespace: target_jsonl_blob
    executable: ./target-jsonl-blob
    settings:
    - name: bucket
      label: Bucket
      description: Target directory (local, S3, Azure Blob)
    - name: key_template
      label: Key Template
      description: Template string for file keys
    config:
      bucket: file://./output/my-bucket
      key_template: $MELTANO_EXTRACTOR_NAMESPACE/{{.StreamName}}.jsonl

You also need to ensure the local "bucket" exists:

mkdir output/my-bucket

3. Run a pipeline

meltano run tap-github target-jsonl-blob

target-jsonl-blob's People

Contributors

Stargazers

Watchers

target-jsonl-blob's Issues

Use goreleaser

https://goreleaser.com

attempting to install first time

I had a attempt to install this in my meltano project but.. wasnt quite hitting the mark unfortunately..

https://meltano.slack.com/archives/C01PKLU5D1R/p1647388209417079

I guess usually loaders are using pip because they are written in python.. in this case.. we're using golan so.. all I tried so far was

meltano add --custom loader target-jsonl-blob

but was met with errors

Installing loader 'target-jsonl-blob'...
Loader 'target-jsonl-blob' could not be installed: failed to install plugin 'target-jsonl-blob'.
ERROR: Could not find a version that satisfies the requirement target-jsonl-blob (from versions: none)
ERROR: No matching distribution found for target-jsonl-blob

Failed to install plugin(s)

Perhaps I need to simply

git clone < this project >

go build -o target-jsonl-blob

EDIT:

just for kicks I also tried

meltano add loader target-jsonl-blob

got

Loader 'target-jsonl-blog' is not known to Meltano

Add integration tests

Local
S3 (using MinIO)
GCS (using https://github.com/fsouza/fake-gcs-server)

`--help` option erroneously expects a config file to be passed

$ ./target-jsonl-blob --help
JSONLines Singer target for blob storages

Usage:
  target-jsonl-blob [flags]

Flags:
  -c, --config string   Config file
  -h, --help            help for target-jsonl-blob
  -i, --input string    Input file
2022/03/19 10:24:42 unable to decode into struct, Config File "config" Not Found in "[]"

As reported in #3 (comment)

Recommend Projects