singer-io / knots Goto Github PK

Docker Desktop for Windows is installed, when starting KNOTS it only says "Wake up Docker".
When opening a commandline and running "docker -v" it says "Docker version 18.09.3, build 774a1f4"
What am i doing wrong?

Any ideas? :)

How hard would it be to support local taps, or other taps directly from the github source?

right now I imagine I can create and host a docker image of my tap following some spec, and then adding a new record here:

I'd love to be able to add to these defaults at runtime and have them stored somewhere on my fs so I can use custom taps, and also show me instructions on how to build a KNOTS ready docker image from my tap 😄

See: https://www.electron.build/code-signing/, for requirements.

Set up view for data.world target configuration.

Acceptance criteria:

• Displays form including API token and dataset fields
• Allows API token to be set via data.world login (OAuth)
• Displays the user's datasets in a drop-down field
• Validates form submission

Design and implement in-memory model for a knot.

Acceptance criteria:

• Reusable Node.js code
• Represents tap and target selection, config(s), catalog, and state

Set up view for RedShift tap discovery execution.

Acceptance criteria:

• Displays loader image and message during execution
• Displays execution logs
• Displays execution errors

Set up view for catalog configuration.

Acceptance criteria:

• Displays all streams in a table
• Allows streams to be selected
• Allows a replication key to be selected for each selected stream
• Displays possible replication keys for each stream in a drop-down menu
• Allows start date to be selected
• Validates form submission

Implement data.world OAuth flow

Acceptance criteria:

• Obtains user token via data.world OAuth
• Updates view with obtained token

Set up view for target installation.

Acceptance criteria:

• Displays loader image and message during installation
• Displays installation errors

Users should be able to download a .zip file containing a Makefile and all the necessary configuration for running syncs outside of the Knot app.

Acceptance criteria:

• Makefile should include install, sync and full-sync targets

Set project up as an Electron app.

Acceptance criteria:

• Standard directory structure
• Boilerplate code
• Scripts to build and run
• Basic documentation in CONTRIBUTING.md

Allow user to select from existing datasets.

Acceptance criteria:

• Retrieves datasets the user has write access to
• Updates view with appropriate content in dataset dropdown

Please add support for the Google BigQuery Target (as mentioned on Singer.IO: https://github.com/RealSelf/target-bigquery), would love to use this with KNOTS :)

Today, we don't seem to be using Flow correctly. There is room to reuse types more often and lots of FlowFixMe and FlowIssue that require attention.

Today, introducing a new tap or target requires that we touch a number of different files.

I'd like us to improve that so that new taps or target can be introduced with the addition of a single module/file. Think about the lifecycle and what methods/functions that module needs to implement in order to properly describe itself (metadata), handle configuration (required, optional and default params) and render the various UI components that we require (selection tile, config form, etc).

Once implemented, this should be properly documented for the benefit of other project contributors (CONTRIBUTING.md).

IMPORTANT: This starts with a design proposal & review

The README states that you can download a .dmg file and use that to install this app, but the link provided just downloads a zip of the repo. How can I actually install this without having to package it myself?

Run tap in discovery mode and show feedback to user.

Acceptance criteria:

• Starts tap execution
• Updates view during execution (including live logs)
• Updates view in case of error

Define and implement a mechanism for executing containerized taps and targets.

Acceptance criteria:

• Reusable Node.js code
• Runs tap in discovery mode
• Runs tap | target
• Properly routes std IN, OUT and ERR
• Unit tested

I cannot make tap-facebook to work cause sdk is deprecated needs to be updated to 3.2. (C.F: screenshot)

When I try to setup a Tap for Adwords when I click the generate access token for the refresh token i get a redirect uri error/mismatch.

Do you have any idea on how to fix this ?

Set up view for RedShift tap configuration.

Acceptance criteria:

• Displays form with all configuration parameters
• Validates parameters

Define and implement a mechanism for detecting and installing required Docker images on demand.

Acceptance criteria:

• Reusable Node.js code
• Detects if Docker and docker compose are installed
• Detects if required Docker image is installed (taking version into account)
• Installs Docker image (taking version into account)
• Unit tested

Set up view for tap installation.

Acceptance criteria:

• Displays loader image & message during installation
• Displays installation errors

Learn more: https://www.electron.build/auto-update/

I have a knot that uses the postgres tap and the data.world target. It runs great from KNOTS but when I generate an export package and attempt to sync from the Makefile, it fails. After a little digging, it appears that the postgres tap requires the use of the --catalog argument, but Makefiles will always use --properties; reference. The KNOTS sync command is able to handle such a situation; reference.

Run tap|target and display feedback to user.

Acceptance criteria:

• Starts tap|target execution
• Updates view during execution (including live logs of tap and target separately)
• Updates view in case of success or error

Initially, error handling should be limited to displaying an error message and allowing users to retry or revisit configuration steps.

Set up knot execution screen.

Acceptance criteria:

• Displays basic info about tap and target (e.g. logos)
• Displays run button
• Displays execution logs for tap and target separately
• Displays execution errors
• Displays success message

Hi, error yelling back when trying to configure tap-adwords using Knots desktop:

suds.WebFault: Server raised fault: '[RequestError.UNSUPPORTED_VERSION @ ; trigger:'You are accessing an AdWords API version v201802 that has been discontinued.

seems like it's pulling from outdated file GET https://adwords.google.com/api/adwords/reportdownload/v201802/reportDefinition.xsd

The way we have implemented file operations makes the app prone to reoccurring bugs. This applies to any file in a knot, but let's take catalog.json as an example, at any point it can live in ./, ./configs, ./{knot}/tap or ./knots/{knot}/tap (also, what ./ means, changes between dev and prod builds). Instead, I'd like us to change this so:

1. If a knot is saved, it lives in ./knots/{knot}
2. Knots being edited live in ./tmp/{uuid}
3. Knots being run stay in ./knots/{knot}
4. ./tmp is garbage collected aggressively (e.g. when app is opened/closed, when wizard is entered/exited, etc)

Allow users to move forwards and backwards through the knot configuration flow.

Acceptance criteria:

• Saves state on forwards navigation (taking form validation into account)
• Restores view state on backwards navigation

Define how Dockerfiles will be managed and images will be built, and implement that for tap-redshift and target-datadotworld.

Acceptance criteria:

• Standard directory structure
• Dockerfiles for tap-redshift and target-datadotworld
• Basic documentation in CONTRIBUTING.md

Certain taps (mysql, postgres, oracle) support LOG_BASED as the replication_method, in addition to INCREMENTAL and FULL_TABLE. We should study it to understand how the best way to introduce support for it.

Populate catalog configuration form with output from tap discovery.

Acceptance criteria:

• Parses output of query discovery
• Updates view with one row per stream
• Updates view with appropriate content for replication key drop-downs