Giter Club home page Giter Club logo

tap-autopilot's Introduction

tap-autopilot

CircleCI

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

Quick start

  1. Install

    > pip install tap-autopilot
  2. Get your Autopilot API Key

    Login to your Autopilot account, navigate to your account settings and then to the Autopilot API section. Generate a New API Key, you'll need it for the next step.

  3. Create the config file

    Create a JSON file called config.json containing the api key you just generated and a start date, the tap will only return contacts who have been updated after the date chosen. Start dates should conform to the RFC3339 specification.

    {
        "api_key": "your-autopilot-api-token",
        "start_date": "2017-01-01T00:00:00Z"
    }
  4. Discover and Catalog

    Use the discover flag to explore the schema for each of this tap's resources

    > tap-autopulot --config config.json --discover

    Pipe the output of this file to a file that will serve as the catalog, where you will select which streams and properties to sync

    > touch catalog.json
    > tap-autopilot --config config.json --discover >> catalog.json

    The catalog is an object with a key streams that has an array of the streams for this tap. For each stream you want to sync, add a "selected": true property on the stream object. Below is an example of how you would select to sync the contacts stream. This property is recursive so it will select all children. If you don't want to sync a property, you can add "selected": false on that property.

            {
            "schema": {
                "properties": {...},
                "type": "object",
                "selected": true
            },
            "stream": "contacts",
            "tap_stream_id": "contacts"
        }
  5. [Optional] Add additional optional config parameters

    You can include a user_agent key in your config.json to further customize the behavior of this Tap.

    • user_agent should be set to something that includes a contact email address should the API provider need to contact you for any reason.

    If you were to use the user_agent, your complete config.json should look something like this.

    {
      "api_key": "your-api-token",
      "start_date": "2017-01-01T00:00:00Z",
      "user_agent": "Stitch ([email protected])"
    }
  6. [Optional] Create the initial state file

    You can provide JSON file that contains a date for the API endpoints to force the application to only fetch data newer than those dates. If you omit the file it will fetch all Autopilot data. State Files will be created for you after a successful run and will keep the status of the previous run for the next time it is invoked.

    {
      "contacts": "2017-01-17T20:32:05Z"
    }
    ``
    
  7. Run the application

    tap-autopilot can be run with:

    tap-autopilot --config config.json --catalog catalog.json [--state state.json]

Copyright © 2017 Stitch

tap-autopilot's People

Contributors

asaf-erlich avatar cosimon avatar dmosorast avatar glqstrauss avatar luandy64 avatar wcjohnson11 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tap-autopilot's Issues

Millions of rows for a small table

I am using this tap to pull autopilot data into stitchdata. Within a few hours, my list table had extracted over a million rows. There is approx. 100 list records that were extracted 34000 times each. Any idea why this is happening?

Setting dates in config.json or state.json does not prevent a full run.

Setting the start date in config.json does not affect the Tap - see data example below. Start date set to 2039-06-22 in config.json.

note that setting dates in State.json does not work either:
{
"contacts": "2030-06-20T20:32:05Z",
"lists": "2030-06-20T20:32:05Z",
"smart_segments": "2030-06-20T20:32:05Z",
"smart_segments_contacts": "2030-06-20T20:32:05Z"
}

INFO Starting sync. Will sync these streams: ['contacts', 'lists', 'smart_segments', 'smart_segments_contacts']
INFO Syncing contacts
INFO Only syncing contacts updated since 2039-06-22T20:00:00.000000Z
INFO GET https://api2.autopilothq.com/v1/contacts
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 1.3322679996490479, "tags": {"endpoint": "contacts", "http_status_code": 200, "status": "succeeded"}}
INFO GET https://api2.autopilothq.com/v1/contacts/person_0043779F-E327-421B-91D3-A2E64E5CB9FE
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.5752568244934082, "tags": {"endpoint": "person_0043779F", "http_status_code": 200, "status": "succeeded"}}
INFO GET https://api2.autopilothq.com/v1/contacts/person_00809040-CF63-48D2-BEEF-2C3AC53BEF05....
... tap continues to import.

Lists contacts not implemented

Though the README specifies support for retrieving lists_contacts (similar to smart_segments_contacts), this functionality is not actually implemented.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.