Giter Club home page Giter Club logo

dss-plugin-api-connect's Introduction

API-Connect Plugin

This Dataiku DSS plugin provides read connectors and recipes that can be configured to interact with a wide range of REST API.

Documentation: https://www.dataiku.com/product/plugins/api-connect/

Licence

Copyright 2021-2022 Dataiku SAS

This plugin is distributed under the Apache License version 2.0

dss-plugin-api-connect's People

Contributors

alexbourret avatar mayeulrousselet avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dss-plugin-api-connect's Issues

Redirect Causes a Lost Authorization Header

The API we're consuming has a base URL we initially hit but are then redirected to version-specific URL where the API request is ultimately made. If we use the base URL with the plugin, we're getting a 401 error, because the authorization header is not being passed to the redirected URL. This is similar to the StackOverflow article here: https://stackoverflow.com/questions/28564961/authorization-header-is-lost-on-redirect

Postman solves this by having a checkbox to explicitly allow this if the user requires. I'm assuming it is slightly less secure, so having it off by default may make sense here as well.

image

Next Page URL Not Returning Data

We're using "Next page URL provided" with a path that contains the dot syntax. While we're not encountering an error, we're only getting one page's worth of data. Furthermore, if I pass a completely invalid string into the "Key to next request URL" field ("asdfasdf") the first page's data is returned without an error.

In a copy of the version 1.0.2 plugin, I was able to bypass this issue by commenting out lines 57-68 of pagination.py, though I don't suppose this is the ultimate solution.

image

Unable to Access Nested Data via Dot Syntax

I'm getting a response back from an API that looks like this:

{ "Data": { "XRefCode": "Data_Warehouse_Integration", "Rows": [ { "Employee_XRefCode": "5", "Job_ShortName": "Nurse", "Employee_FirstName": "Melissa" }, { "Employee_XRefCode": "7", "Job_ShortName": "Nurse", "Employee_FirstName": "Nick" } ] }, "Paging": { "Next": "https://url-for-next-page-here" } }

If I set the "Data extraction key" as "Data" I get just two values back, XRefCode and Rows, which I'd expect. However, when I try to then drill to the next level to get the row data via "Data.Rows" I get an exception that no data is found for the extraction key. Am I using the wrong syntax for getting down to the nested Data in this format or is there a bug?

Need proxy support

This is a great plugin, but it's difficult to use in an enterprise environment since only internal urls can be reached without going through a proxy. Ideally, the underlying requests proxyDict can be set from the UI: http, https, ftp. Then the proxyDict can simply be passed if set for all underlying requests, allowing external urls to be reached.

Allow Bearer Token to be retrieved from Project Variable

We've got a requirement to enable an Oauth flow for authentication before data can be retrieved. In order to accomplish this, two API calls needs to be made, first, one using username/password auth to get a Bearer Token, then the second call using the Bearer Token to actually get the data from the API. To accomplish this, we could chain two calls to the plugin, or, make the initial API call to get the Bearer token using a python component. The problem is, there's no way to "pass in" the retrieved Bearer Token into the API Connect plugin at runtime. To enable this, we'd like to be able to set the Bearer Token value in the plugin to ${project_variable_name} and have the Bearer Token be pulled from the project (or global) variables.

Paging start from "1"

Pagination mechanism "per page" starts with 1.

But api may start with page 0.

Would it be possible to set starting index?

Infinite paging

I suppose that the paging stop incrementing when result become empty.

But with API that still send first page, event sending "pagingIndex=123...", the paging increment never stop (until server api drop for exceeding quota).

{ "content": "{\"totalCount\": 1, \"pageIndex\": 0, \"limit\": 50, \"isLastPage\": true, \"results\": .... }

Would it be possible to create parameter for "totalCount" or "isLastPage" key to check ?

Plugin changes true to True and false to False

I'm pulling data from a REST API that correctly returns the boolean values in Json as true and false (starting in lower case). However, when the API Connect plugin finishes with the data and puts them in a DSS Dataset the values have True and False (upper case). This makes parsing the Raw return more difficult. One has to pre-process the results before one can unnest the resulting JSON.
--Tom

API-Client calling GET instead of POST or PUT

The client is calling a GET request instead of a POST or PUT request, even when selecting the POST or PUT methods in the GUI.

Configuration:
Screen Shot 2021-05-24 at 7 58 34 PM

Server response ("read", should be "create" or "update"):
Screen Shot 2021-05-24 at 7 58 52 PM

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.