Giter Club home page Giter Club logo

tap-jotform's Introduction

tap-jotform

Singer Tap for Jotform. Built with the Meltano Singer SDK.

Capabilities

  • catalog
  • state
  • discover
  • about
  • stream-maps
  • schema-flattening

Settings

Setting Required Default Description
api_key True None Authentication key. See https://api.jotform.com/docs/#authentication
api_url False https://api.jotform.com API Base URL
user_agent False tap-jotform/0.0.1 User-Agent header
start_date False None Start date for data collection
requests_cache False None Cache configuration for HTTP requests
stream_maps False None Config object for stream maps capability. For more information check out Stream Maps.
stream_map_config False None User-defined config values to be used within map expressions.
flattening_enabled False None 'True' to enable schema flattening and automatically expand nested properties.
flattening_max_depth False None The max depth to flatten schemas.

A full list of supported settings and capabilities is available by running: tap-jotform --about

Streams

Stream name API endpoint API docs Notes
forms /user/forms https://api.jotform.com/docs/#user-forms Replication for this stream is opt-in. See instructions below.
questions /form/{form_id}/questions https://api.jotform.com/docs/#form-id-questions
submissions /user/submissions https://api.jotform.com/docs/#user-submissions Replication for this stream is opt-in. See instructions below.
reports /user/reports https://api.jotform.com/docs/#user-reports
user_history /user/history https://api.jotform.com/docs/#user-history

Configuring incremental replication

By default, the forms and submissions stream are synced with FULL_TABLE replication. Incremental replication can be enabled by setting the replication metadata in the stream's entry in the catalog file:

  • replication_method: set toINCREMENTAL
  • replication_key set to created_at or updated_at. The former will omit updated submissions, while the latter will omit new submissions.

For example, to enable incremental replication for the submissions stream:

{
  "streams": [
    {
      "tap_stream_id": "submissions",
      "stream": "submissions",
      "replication_method": "INCREMENTAL",
      "replication_key": "updated_at",
    }
  ]
}

Source Authentication and Authorization

To generate an API key, follow the instructions in https://api.jotform.com/docs/#gettingstarted.

Usage

You can easily run tap-jotform by itself or in a pipeline using Meltano.

Executing the Tap Directly

tap-jotform --version
tap-jotform --help
tap-jotform --config CONFIG --discover > ./catalog.json

Developer Resources

Initialize your Development Environment

pipx install poetry
poetry install

Create and Run Tests

Create tests within the tap_jotform/tests subfolder and then run:

poetry run pytest

You can also test the tap-jotform CLI interface directly using poetry run:

poetry run tap-jotform --help

Testing with Meltano

Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Your project comes with a custom meltano.yml project file already created. Open the meltano.yml and follow any "TODO" items listed in the file.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-jotform
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke tap-jotform --version
# OR run a test `elt` pipeline:
meltano elt tap-jotform target-jsonl

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.

tap-jotform's People

Contributors

aroder avatar dependabot[bot] avatar edgarrmondragon avatar pre-commit-ci[bot] avatar

Watchers

 avatar

Forkers

datateer

tap-jotform's Issues

incremental sync not robust enough to handle null values in updated_at

The jotform API consistently provides an updated_at field. However, for submissions, the initial value of this property is null.

That means queries used by the incremental syncing, such as {{urlBase}}/form/231005246607144/submissions?filter={"updated_at:gt":"2022-04-21 19:25:37"} exclude newly created submissions

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.