Giter Club home page Giter Club logo

singer-tap-template's Introduction

singer-tap-template

A cookiecutter template for creating Singer taps.

Usage

The best way to demonstrate creating your tap structure is with an example. Below I will initialize the "tap-foobar" project:

Start by installing cookiecutter:

$ pip install cookiecutter

The next command will ask for some input. Enter the name of your tap:

$ cookiecutter https://github.com/singer-io/singer-tap-template.git
project_name [e.g. 'tap-facebook']: tap-foobar

For the package_name, I just hit enter since tap_foobar is what I wanted:

package_name [tap_foobar]:

Now that the project exists, make a virtual environment:

$ cd tap-foobar
$ python3 -m venv ~/.virtualenvs/tap-foobar
$ source ~/.virtualenvs/tap-foobar/bin/activate

Install the package:

$ pip install -e .

And invoke the tap in discovery mode to get the catalog:

$ tap-foobar -c sample_config.json --discover

The output should be a catalog with the single sample stream (from the schemas folder):

{
  "streams": [
    {
      "metadata": [],
      "schema": {
        "additionalProperties": false,
        "properties": {
          "string_field": {
            "type": [
              "null",
              "string"
            ]
          },
          "datetime_field": {
            "type": [
              "null",
              "string"
            ],
            "format": "date-time"
          },
          "double_field": {
            "type": [
              "null",
              "number"
            ]
          },
          "integer_field": {
            "type": [
              "null",
              "integer"
            ]
          }
        },
        "type": [
          "null",
          "object"
        ]
      },
      "stream": "sample_stream",
      "key_properties": [],
      "tap_stream_id": "sample_stream"
    }
  ]
}

If this catalog is saved to a catalog.json file, it can be passed back into the tap in sync mode:

tap-foobar -c sample_config.json --properties catalog.json

Now you build the tap!

Copyright © 2018 Stitch

singer-tap-template's People

Contributors

aaronsteers avatar b-ryan avatar dmosorast avatar nick-mccoy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

singer-tap-template's Issues

Add `Development` section to the README

This came up in #2.

Given that this template assumes that you are using virtualenvs to develop the tap, the generated README should include a section something like the following.

## Development

1. Create a virtualenv
1. Run `pip install -e '.[dev]'`
1. Run `tap-foo`

TypeError due to trying to find max between None with non None value

If a bookmark column is set the a TypeError is always thrown when running the tap.

As max_bookmark is initially set to None, when we try to see if max_bookmark needs updating by comparing it to the first value of the bookmark column a TypeError is thrown:

TypeError: '>' not supported between instances of 'int' and 'NoneType'

Remove Legacy Properties Option

The template differentiates between properties and catalog and will accept both. However, the --properties argument is deprecated in the Singer spec, and if it's passed in, then taps written with this template expecting the catalog will not function.

I would like to propose removing the properties argument handling here since all new taps should conform to the latest standards.

__init__.py Accesses Catalog dict Improperly

When the default init.py uses the catalog dict, it accesses it using "catalog.key" rather than "catalog.get('key')". This is not proper python code and will not run.
Examples:
line 54: "for stream in catalog.streams:"
line 55: "stream_metadata = metadata.to_map(stream.metadata)"

Use Singer-Python Convenience Functions

singer-python has gotten a few convenience functions that simplify more complicated pieces of writing a tap. This template should be edited to use these instead of the code currently in existence.

get_standard_metadata

This function is in the metadata module and writes things like available and table-key-properties to get fully-formed, standards compliant metadata out of the box that can be extended on a per-tap basis.

get_selected_streams
Catalog has been extended with the function get_selected_streams to simplify the handling of current_stream in the state and selection metadata. All taps use this, so it should replace the function in this template.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.