Giter Club home page Giter Club logo

kong-plugin-template-transformer's Introduction

Kong-plugin-template-transformer

Build Status

This is a Kong middleware to transform requests / responses, using pre-configured templates. This plugin has been tested to work along with kong >= 2.6.x, for a legacy version of this plugin, please check this link.

The Problem

When using Kong, you can create routes that proxy to an upstream. The problem lies when the upstream has an ugly response/request contract. Sometimes the kong bundled request transformer is not enough. This plugin was created for those situations, it can apply a template to JSON requests and responses, transforming them pretty much however you like.

Project Structure

The plugin folder should contain at least a schema.lua and a handler.lua, alongside with a spec folder and a .rockspec file specifying the current version of the package.

Writing Templates

We use lua-resty-template to write templates. It's also very important that you don't leave any \t in the files. We also only support JSON requests and responses for now.

Examples

Inside kong-plugin-template-transformer/template-transformer/validator/json/examples you'll find some examples for json templates and request files.

Rockspec Format

The .rockspec file should follow LuaRocks' conventions

Configuration

Enabling the plugin on a Route

Configure this plugin on a Route with:

curl -X POST http://kong:8001/routes/{route_id}/plugins \
    --data "name=kong-plugin-template-transformer"  \
    --data "config.request_template=http://new-url.com"
    --data "config.response_template=http://new-url.com"
    --data "config.hidden_fields=http://new-url.com"
  • route_id: the id of the Route that this plugin configuration will target.
  • config.request_template: Optional, the template to be applied before proxying to the upstream.
  • config.response_template: Optional, the template to be applied before returning to the client.
  • config.hidden_fields: Optional, a list of request or response fields that you do not want Kong to save them in a log file.

Developing

In docker

docker build . -t kong-plugin-template-transformer-dev
docker run -it -v ${PWD}/template-transformer:/template-transformer kong-plugin-template-transformer-dev bash

Credits

made with ❤️ by Stone Payments

kong-plugin-template-transformer's People

Contributors

akira28 avatar andremacdowell avatar andreolle avatar biancarosa avatar crismotinha avatar d0x2f avatar diegolucasb avatar gilbertovgl avatar italomarcel avatar jeancabral avatar jonathanlazaro1 avatar leoferlopes avatar mikefarah avatar pfadel avatar victorgtdacruz avatar wuerike avatar

Stargazers

avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

kong-plugin-template-transformer's Issues

Template validator

Issue Description

We need a good CLI tool that validates templates.

Expected Behavior

./validate-template.lua < input.json > output.json

No LuaRocks module found for kong.dao.errors

Issue Description

Install plugin in corresponding folder in kong container

luarocks make

export plugin

export KONG_PLUGINS=bundled, kong-plugin-template-transformer

restart kong

kong prepare && kong reload

Expected Behavior

Shoud init correctly, instead it throwing:

init_by_lua error: /usr/local/share/lua/5.1/kong/tools/utils.lua:590: ...kong/plugins/kong-plugin-template-transformer/schema.lua:2: module 'kong.dao.errors' not found:No LuaRocks module found for kong.dao.errors
	no field package.preload['kong.dao.errors']
	no file './kong/dao/errors.lua'
	no file './kong/dao/errors/init.lua'
	no file '/usr/local/openresty/site/lualib/kong/dao/errors.ljbc'
	no file '/usr/local/openresty/site/lualib/kong/dao/errors/init.ljbc'
	no file '/usr/local/openresty/lualib/kong/dao/errors.ljbc'
	no file '/usr/local/openresty/lualib/kong/dao/errors/init.ljbc'
	no file '/usr/local/openresty/site/lualib/kong/dao/errors.lua'
	no file '/usr/local/openresty/site/lualib/kong/dao/errors/init.lua'
	no file '/usr/local/openresty/lualib/kong/dao/errors.lua'
	no file '/usr/local/openresty/lualib/kong/dao/errors/init.lua'
	no file '/usr/local/openresty/luajit/share/luajit-2.1.0-beta3/kong/dao/errors.lua'
	no file '/usr/local/share/lua/5.1/kong/dao/errors.lua'
	no file '/usr/local/share/lua/5.1/kong/dao/errors/init.lua'
	no file '/usr/local/openresty/luajit/share/lua/5.1/kong/dao/errors.lua'
	no file '/usr/local/openresty/luajit/share/lua/5.1/kong/dao/errors/init.lua'
	no file '/home/kong/.luarocks/share/lua/5.1/kong/dao/errors.lua'
	no file '/home/kong/.luarocks/share/lua/5.1/kong/dao/errors/init.lua'
	no file '/usr/local/openresty/site/lualib/kong/dao/errors.so'
	no file '/usr/local/openresty/lualib/kong/dao/errors.so'
	no file './kong/dao/errors.so'
	no file '/usr/local/lib/lua/5.1/kong/dao/errors.so'
	no file '/usr/local/openresty/luajit/lib/lua/5.1/kong/dao/errors.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
	no file '/home/kong/.luarocks/lib/lua/5.1/kong/dao/errors.so'
	no file '/usr/local/openresty/site/lualib/kong.so'
	no file '/usr/local/openresty/lualib/kong.so'
	no file './kong.so'
	no file '/usr/local/lib/lua/5.1/kong.so'
	no file '/usr/local/openresty/luajit/lib/lua/5.1/kong.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
	no file '/home/kong/.luarocks/lib/lua/5.1/kong.so'
stack traceback:
	[C]: in function 'error'
	/usr/local/share/lua/5.1/kong/tools/utils.lua:590: in function 'load_module_if_exists'
	/usr/local/share/lua/5.1/kong/db/schema/plugin_loader.lua:182: in function 'load_subschema'
	/usr/local/share/lua/5.1/kong/db/dao/plugins.lua:194: in function 'load_plugin'
	/usr/local/share/lua/5.1/kong/db/dao/plugins.lua:239: in function 'load_plugin_schemas'
	/usr/local/share/lua/5.1/kong/init.lua:402: in function 'init'
	init_by_lua:3: in main chunk

gzip responses from upstream are not decoded

Issue Description

If my upstream service returns a gzipped response, if I use a transform template kong returns an empty response (and give an error on console)

Expected Behavior

if I send the Accept-Encoding gzip and the upstream service sends a gzipped response, I should receive the gzipped response

Migrate from VSTS to Travis CI

Issue Description

Our CI platform should be as close as possible to the OSS community. We need to add a .travis.yml file for build / tests.

Expected Behavior

Tests should run on Travis.

Advertising this plugin to the Kong community?

Hello,

Good job on the plugin!

Have you given some thought about advertising your plugin to the Kong community? You could submit it for publication to the Kong Hub (learn how to do so here - contact.
If you are not interested in publishing this plugin to the Hub, you may consider posting an announcement on our community forum, Kong Nation as well.

Best,

Version 1.1.0 is not available at LuaRocks

Issue Description

I'm trying to install the latest version of the plugin 1.1.0 but it's not available at https://luarocks.org/modules/stone-payments/kong-plugin-template-transformer: luarocks install kong-plugin-template-transformer 1.1.0-0. When trying it fails with this error:

Error: No results matching query were found for Lua 5.1.
To check if it is available for other Lua versions, use --check-lua-versions.

I even tried to use the git repo with no success:
luarocks install https://raw.githubusercontent.com/stone-payments/kong-plugin-template-transformer/v1.1.0/rockspec.template

Expected Behavior

Plugin is installed successfully running luarocks install kong-plugin-template-transformer 1.1.0-0

No LuaRocks module found for kong.dao.errors

Issue Description

Installed the plugin in docker by:

FROM kong:latest

RUN luarocks install kong-plugin-template-transformer

Then when running the container I got the following error in the logs:

stack traceback:
	[C]: in function 'error'
	/usr/local/share/lua/5.1/kong/tools/utils.lua:584: in function 'load_module_if_exists'
	/usr/local/share/lua/5.1/kong/db/dao/plugins.lua:229: in function 'load_plugin_schemas'
	/usr/local/share/lua/5.1/kong/init.lua:343: in function 'init'
	init_by_lua:3: in main chunk
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/tools/utils.lua:584: ...kong/plugins/kong-plugin-template-transformer/schema.lua:2: module 'kong.dao.errors' not found:No LuaRocks module found for kong.dao.errors
	no field package.preload['kong.dao.errors']
	no file './kong/dao/errors.lua'
	no file './kong/dao/errors/init.lua'
	no file '/usr/local/openresty/site/lualib/kong/dao/errors.ljbc'
	no file '/usr/local/openresty/site/lualib/kong/dao/errors/init.ljbc'
	no file '/usr/local/openresty/lualib/kong/dao/errors.ljbc'
	no file '/usr/local/openresty/lualib/kong/dao/errors/init.ljbc'
	no file '/usr/local/openresty/site/lualib/kong/dao/errors.lua'
	no file '/usr/local/openresty/site/lualib/kong/dao/errors/init.lua'
	no file '/usr/local/openresty/lualib/kong/dao/errors.lua'
	no file '/usr/local/openresty/lualib/kong/dao/errors/init.lua'
	no file '/usr/local/openresty/luajit/share/luajit-2.1.0-beta3/kong/dao/errors.lua'
	no file '/usr/local/share/lua/5.1/kong/dao/errors.lua'
	no file '/usr/local/share/lua/5.1/kong/dao/errors/init.lua'
	no file '/usr/local/openresty/luajit/share/lua/5.1/kong/dao/errors.lua'
	no file '/usr/local/openresty/luajit/share/lua/5.1/kong/dao/errors/init.lua'
	no file '/home/kong/.luarocks/share/lua/5.1/kong/dao/errors.lua'
	no file '/home/kong/.luarocks/share/lua/5.1/kong/dao/errors/init.lua'
	no file '/usr/local/openresty/site/lualib/kong/dao/errors.so'
	no file '/usr/local/openresty/lualib/kong/dao/errors.so'
	no file './kong/dao/errors.so'
	no file '/usr/local/lib/lua/5.1/kong/dao/errors.so'
	no file '/usr/local/openresty/luajit/lib/lua/5.1/kong/dao/errors.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
	no file '/home/kong/.luarocks/lib/lua/5.1/kong/dao/errors.so'
	no file '/usr/local/openresty/site/lualib/kong.so'
	no file '/usr/local/openresty/lualib/kong.so'
	no file './kong.so'
	no file '/usr/local/lib/lua/5.1/kong.so'
	no file '/usr/local/openresty/luajit/lib/lua/5.1/kong.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
	no file '/home/kong/.luarocks/lib/lua/5.1/kong.so'

Expected Behavior

Kong to start

Checklist

Getting Error in read_json_body()

Getting Error while transforming JSON Body

Response Headers :
Age →0
Connection →keep-alive
Content-Encoding →gzip
Content-Type →application/json
Date →Tue, 11 Oct 2022 12:46:08 GMT
ETag →"83d3f99d-48ae-11ed-963d-819ea9e4b46d-2022-10-10T15:16:29.157000000Z--gzip"
Last-Modified →Mon, 10 Oct 2022 15:16:29 GMT
Transfer-Encoding →chunked
Vary →Accept, Accept-Encoding
Via →kong/2.8.1.3-enterprise-edition
X-Backend-History →facade_service
X-Cache →MISS
X-Cache-Origin →EAST
X-Kong-Proxy-Latency →13
X-Kong-Upstream-Latency →309
X-Shard-History →varnish_east1
X-Varnish →121148358
requestId →1234

Error Log :
2022/10/11 12:42:59 [error] 27254#0: *4437 [lua] handler.lua:30: read_json_body(): {"devices":[{"identifier":"ANBV","role":"ce","hostname":"ce-123.on","network":"CRAN","asn":"33662","status":"IN_SERVICE","subNetwork":"COMMERCIAL"}]} while sending to client, client: 192.1.1.1, server: kong, request: "GET /ndsdeviceinfo?fqdn=cdummydata HTTP/1.1", upstream: "https://192.1.1.1:443/nds/v1/deviceInfo?fqdn=dummydata", host: "192.1.1.1:8000"

Can not add plugin

Issue Description

I tried to add a "kong-plugin-template-transformer" plug-in to kong running on the docker container, but it failed.

Configuring a Service
https://docs.konghq.com/0.13.x/getting-started/configuring-a-service/

Enabling the plugin on a Service

curl -X POST http://kong:8001/services/example-service/plugins \
    --data "name=kong-plugin-template-transformer"  \
    --data "config.request_template='{ "email": "{{body.user}}", "password": "{{body.password}}" }'" \
    --data "config.response_template='{ "status": "{{status}}", "message": "{{body.message}}" }'" \
    --data "config.hidden_fields='["password"]'"

{"config":"plugin 'kong-plugin-template-transformer' not enabled; add it to the 'custom_plugins' configuration property"}

Expected Behavior

Plugins will be added

Checklist

template from url

how to use a template from a url, I tried to use a template on a separate url but it doesn't work well, please help

Compatibility with Kong 1.x

Due to limited functionality of kong bundled request transformer , this plugin is life-saving. Any ETA for Kong 1.x compatibility?

Add common examples / use cases in a examples folder

Issue Description

At the moment, we have just simple examples but by using the plugin for half a year, we came across some complex use cases that would be worthy to share.

Expected Behavior

We should have more examples.

Checklist

  • Example with just key transforming
  • Example with variable injection from nginx context
  • Example with nullable fields
  • Example with optional fields

Unable to translate response containing json array

Issue Description

Hello,
I was wondering whether template transformer plugin can translate response containing json array.

I have a case, where api response returns an array of json objects.
e.g;

[{
    "_id": 1,
    "firstName": "abc",
    "lastName": "xyz",
    "email": "[email protected]",
    "role_id": [{
        "_id": 1,
        "name": "Administrator",
        "description": "All Access"
      }
    ]
  }
]

From the response above, I want to remove field '_id' via template transformer. But it still returns what I set treating the response as simple object (not like array).

I suspect that plugin is unable to understand that response is in array format.

Expected Behavior

RESPONSE VALUE:

[{
    "_id": 1,
    "firstName": "abc",
    "lastName": "xyz",
    "email": "[email protected]",
    "role_id": [{
        "_id": 1,
        "name": "Administrator",
        "description": "All Access"
      }
    ]
  }
]

RESPONSE-TEMPLATE:

{
	"firstName": "{{body.firstName}}",
	"lastName": "{{body.lastName}}",
	"email": "{{body.email}}"
}

EXPECTED RESULT:

{
	"firstName": "abc",
	"lastName": "xyz",
	"email": "[email protected]"
}

Checklist

How to install the plugin

Until now I just used official kong plugins and go or js plugins for my instance.
Sorry if it's a basic question, but how can I install this plugin?

Question: obtain path variables in templates

Issue Description

I would need to do some logic based on a path parameter.
Example: /users/<userId>

in the template something like:

if path_parameters['userId'] == 2202 ....

Is this possible? And with query parameters?

Object transformations returns 'tables'

Issue Description

using this template for the response:

{
    "zoo": {{body.headers}}
}

with this original response

{
    "headers": {
      "foo": "bar"
    }
}

I receive

{
    "zoo": table: 0x7f6068eacdd0
}

Expected Behavior

I receive

{
    "zoo": {
      "foo": "bar"
    }
}

Apply template to response only on 2xx status codes

Issue Description

If I receive a 4xx or 5xx from the internal service, the format would be totally different from a 2xx response. It would be nice to have an option to activate templates only on 2xx status.

Expected Behavior

If i receive a 5xx or 4xx from the internal service, I don't apply any transformation, if I passed a specific option when I create the plugin (something like --apply-only-on-success=true, or --apply-for-status=200)

how to use a template from a file?

how to use a template from a file, as /home/foo/api_xxx/req_transformer.lua

Screenshot 2022-03-09 100632

req_transformer.lua file:
{
"Id": {{body.MarketInstanceIds[1]}},
"SampleString": "{{body.TestString}}"
}

please help

Compatibility with kong 2.x

Issue Description

Getting error when enabling KONG_STREAM_LISTEN in Kong 2.x.
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/tools/utils.lua:625: error loading module 'kong.plugins.kong-plugin-template-transformer.handler':
/usr/local/share/lua/5.1/resty/template.lua:147: attempt to index field 'location' (a nil value)
stack traceback:
/usr/local/share/lua/5.1/resty/template.lua:147: in main chunk
[C]: in function 'require'
...ong-plugin-template-transformer/template_transformer.lua:1: in main chunk
[C]: in function 'require'
...ong/plugins/kong-plugin-template-transformer/handler.lua:20: in main chunk
[C]: at 0x7f6e985f40f0
[C]: in function 'xpcall'
/usr/local/share/lua/5.1/kong/tools/utils.lua:616: in function 'load_module_if_exists'
/usr/local/share/lua/5.1/kong/db/dao/plugins.lua:149: in function 'load_plugin_handler'
/usr/local/share/lua/5.1/kong/db/dao/plugins.lua:227: in function 'load_plugin'
/usr/local/share/lua/5.1/kong/db/dao/plugins.lua:275: in function 'load_plugin_schemas'
/usr/local/share/lua/5.1/kong/init.lua:483: in function 'init'
init_by_lua:11: in main chunk
stack traceback:
[C]: in function 'error'
/usr/local/share/lua/5.1/kong/tools/utils.lua:625: in function 'load_module_if_exists'
/usr/local/share/lua/5.1/kong/db/dao/plugins.lua:149: in function 'load_plugin_handler'
/usr/local/share/lua/5.1/kong/db/dao/plugins.lua:227: in function 'load_plugin'
/usr/local/share/lua/5.1/kong/db/dao/plugins.lua:275: in function 'load_plugin_schemas'
/usr/local/share/lua/5.1/kong/init.lua:483: in function 'init'
init_by_lua:11: in main chunk

Expected Behavior

Checklist

Malformed JSON using template transformer.

Issue Description

Malformed JSON using template transformer.

If the response body from the internal service is like

body: "onclick=\"require(&quot;IntlUtils&quot;).setCookieLocale(&quot;ar_AR&quot;, &quot;en_US&quot;, &quot;https:\\/\\/ar-ar.facebook.com\\/&quot;, &quot;www_list_selector&quot;, 3);"

The transformation plugin is encoding the htmlentities resulting in non scaped quotes

onclick=\"require("IntlUtils").setCookieLocale("es_LA", "en_US", "https:\\/\\/es-la.facebook.com\\/", "www_list_selector", 0); return false;

So is changing the "&quot;" into quotes , and then the returning json response is invalid

Expected Behavior

The expected behavior is that the htmlentites should be returned without any transformation

"onclick=\"require(&quot;IntlUtils&quot;).setCookieLocale(&quot;ar_AR&quot;, &quot;en_US&quot;, &quot;https:\\/\\/ar-ar.facebook.com\\/&quot;, &quot;www_list_selector&quot;, 3);"

Checklist

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.