Giter Club home page Giter Club logo

tap-slack's Introduction

slack icon tap-slack

License: GPL v3 Python 3.7

Singer.io tap for extracting data from the Slack Web API

Installation

It is highly recommended installing tap-slack in it's own isolated virtual environment. For example:

python3 -m venv ~/.venvs/tap-slack
source ~/.venvs/tap-slack/bin/activate
pip3 install tap-slack
deactivate

Setup

The tap requires a Slack API token to interact with your Slack workspace. You can obtain a token for a single workspace by creating a new Slack App in your workspace and assigning it the relevant scopes. As of right now, the minimum required scopes for this App are:

  • channels:read
  • channels:history
  • users:read

Usage

It is recommended to follow Singer best practices when running taps either on their own or with a Singer target.

In practice, it will look something like the following:

~/.venvs/tap-slack/bin/tap-slack --config slack.config.json --catalog catalog.json | ~/.venvs/target-stitch/bin/target-stitch --config stitch.config.json

Replication

The Slack Conversations API does not natively store last updated timestamp information about a Conversation. In addition, Conversation records are mutable. Thus, tap-slack requires a FULL_TABLE replication strategy to ensure the most up-to-date data in replicated when replicating the following Streams:

  • Conversations
  • ConversationMembersStream
  • ConversationHistoryStream

The Users stream does store information about when a User record was last updated, so tap-slack uses that timestamp as a bookmark value and prefers using an INCREMENTAL replication strategy.

Table Schemas

Conversations

  • Table Name: conversations
  • Description:
  • Primary Key Column: id
  • Replication Strategy: FULL_TABLE
  • API Documentation: Link

Conversation Members

  • Table Name: conversation_members
  • Description:
  • Primary Key Column: N/A
  • Replication Strategy: FULL_TABLE
  • API Documentation: Link

Conversation History

  • Table Name: conversation_history
  • Description:
  • Primary Key Column: N/A
  • Replication Strategy: FULL_TABLE
  • API Documentation: Link

Users

  • Table Name: users
  • Description:
  • Primary Key Column: id
  • Replication Strategy: INCREMENTAL
  • API Documentation: Link

tap-slack's People

Contributors

dwallace0723 avatar

Stargazers

avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

singer-io monkidea

tap-slack's Issues

Rate limit error

I'm using the slack integration within the Stitch Data platform and am consistently getting extraction errors with no clear error message. Inspecting the logs, it looks like api calls are repeatedly hitting rate limits without waiting before retry (backing off for 0.0s).

Slack API failed.
2021-10-07 00:44:24,233Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:25,310Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:25,310Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:26,398Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:26,398Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:27,467Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:27,467Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:28,603Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:28,603Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:29,670Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:29,670Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:30,746Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:30,747Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:31,835Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:31,835Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:32,904Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:32,904Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:33,985Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:33,985Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:35,053Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:35,053Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:36,123Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:36,123Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:37,189Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:37,189Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:38,252Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:38,252Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:39,327Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:39,327Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:40,395Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:40,395Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:41,475Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:41,475Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:42,612Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:42,612Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:43,684Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:43,685Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:44,761Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:44,761Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:45,833Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:45,833Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:46,893Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:46,893Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:47,975Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:47,975Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:49,040Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:49,040Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:50,110Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:50,110Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:51,180Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:51,181Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:52,248Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:52,248Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:53,313Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:53,314Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:54,379Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:54,379Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:55,445Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:55,445Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:56,587Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:56,587Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:57,686Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:57,686Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:58,759Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:58,760Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:44:59,836Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:44:59,836Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:45:00,897Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:45:00,897Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:45:01,964Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:45:01,964Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:45:03,025Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:45:03,025Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:45:04,103Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:45:04,103Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:45:05,194Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:45:05,194Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:45:06,262Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:45:06,262Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:45:07,341Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.
2021-10-07 00:45:07,342Z    tap - The server responded with: {'ok': False, 'error': 'ratelimited'})
2021-10-07 00:45:08,438Z    tap - INFO Backing off get_messages(...) for 0.0s (slack.errors.SlackApiError: The request to the Slack API failed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.