Setup for Multi-Tenant Micro-services about tyk HOT 13 CLOSED

API consumer makes a request to https://{client}.mydomain.com/{service}/{path} where {client} is a client name and {service} is the particular micro-service that they are attempting to access and {path} is everything after the name of the micro-service. Once this request is processed by Tyk it is sent to https://{service}.myinternaldomain.com/{path} but I need to provide the {client} value to my target somehow.

This is possible, but might involve some duplication, Tyk has two variables it can play with when generating domain configurations (requires NginX) the org_slug - an "organisation" in Tyk parlance, this can be an "owner" of a series of API Definitions, and an api_slug which in this case could be the {service} description.

Tyk makes these variables available to it's host-manager - a process that can manage nginx configurations, these are templated, so you could potentially generate a configuration tempalte that manages this, but you would need to do this for every "client" (this is where the duplication is), so:

https://{client}.mydomain.com/{service}/{path}

Is actually:

https://{org-slug}.mydomain.com/{api-slug}/{path}

Which is rewritten to:

https://tyk-service/{api-id}/{path}

(this is the bit that Tyk processes), in Tyk, your API definition would need to proxy to:

https://{service}.myinternaldomain.com/{path}

The only issue is that you would need to duplicate each service (API Defintion) for each organisation (client), which could be quite serious.

But since you are separating out by Organisation and have seperate API definitions for them, you could then use the header injection middleware to hard-code the 'client' (org) value as a header to the upstream service.

Now this would be quite painful to do manually, for each client. But you could probably script the Organisation and API definition creation/deletion using the REST API (this is how our uickstart script does it with the docker images)

Alternatively, if you don't mind a bit of latency, you could use the scripting option to create some custom Javascript middleware that would "move" headers from inbound request to an outbound one, you could use this to create a catch-all nginx configuration using a wildcard and then leave the rest to the middleware script.

The next would be to be able to dynamically rewrite the URL. In this case I would have Tyk rewrite https://{client}.mydomain.com/{service}/{path} (where {client} and {path} are both dynamic values) into something along the lines of https://{service}.myinternaldomain.com/clients/{client}/{path} but I'm not seeing a clear path as to how I would accomplish this with Tyk right now either.

For this, I'm afraid Tyk can't help, this seems like full-blown URL rewriting, which is a feature set we'd rather leave to other web servers... we might look at doing something like this at some point, but we'd need some help :-)

`

from tyk.

Thanks for the response!

That makes sense with what you are saying with organizations the duplication does suck, but could definitely be automated easily. I'm going to be using organizations anyways to make use of the quotas at the organization level. The duplication is mitigated some for me since we might have services A, B, C, D but a particular organization might only have access to A and C so that would have to be handled somewhere and the gateway makes sense for that.

As for the URL rewriting, yup it's definitely a full-blown implementation there and I understand the desire to avoid it. It would definitely be pretty cool to be able to do easily via Tyk but the cost in complexity might not make it worth it.

Thanks again, going to close this issue.

from tyk.

Just a side note - in the latest build of the portal / dashboard, we've
changed organisation level quotas:

• you can still set the overall request quota for a whole organisation
• you can no longer set a "key quota" - this was almost impossible to
  enforce in the architecture, especially because of key hashing and keys
  that potentially expire (I.e an org can only issue x number of keys)
• you can set a developer quota though (I.e. Only x number of registered
  developers allowed per portal), this is the change that is only available
  in v0.9.4.4 - the change went out yesterday

Just a heads up - the docs need to catch up :-)

from tyk.

Thanks for the heads up! Luckily I'm only looking for setting an overall request quota.

from tyk.

@lonelycode I'm running into some issues trying to implement this, I think I'm missing one of the middle pieces which is causing my configuration to not work as expected.

Here's my setup so far.

Users:

1. Super User account created without any organization ties.

Organizations:

1. Organization for test is created.
2. Organization for www is created.

API Definitions:

1. Definition for HTTPBin - target URL: http://httpbin.org API slug: httpbin Auth: keyless created in both organizations. (everything else is default)
2. Definition for Google - target URL: https://www.google.com API slug: google Auth: keyless created in both organizations. (everything else is default)

What's happening is that when I have more than one API for a particular organization the requests always are sent to the first API defined irregardless of my host headers or path.

Note: I setup my hosts file to point www.localhost.com and test.localhost.com to the Tyk server.

Based on my understanding of the Host Manager I should be able to visit http://www.localhost.com:8888/httpbin/get to proxy to http://httpbin.org/get. This doesn't work as it proxies the request to http://httpbin.org/httpbin/get. Similarly I tried navigating to http://httpbin.www.localhost.com:8888/get.

But the problem is that no matter which API I try to communicate with my request always goes to HttpBin (only when it is the first API on the Organization). So when I try to go to http://test.localhost.com:8888/google it instead proxies to http://httpbin.org.

Hoping that you can point me in the right direction, as I've spent a good bit of time trying various thing but nothing seems to be resolving the issue.

from tyk.

@DavidTPate

It's a hard one to diagnose, some questions may clarify th answer a bit:

1. Are you using the docker containers, the quickstart script or are you running each process raw?
2. Are you using the dashbaord or file-based definitions?

Here's an attempt at an anwer, or at least places to look:

I am assuming you are using the host manager (one of the docker containers from the quickstart). If you are, then your url structure will look like this:

{api-slug}.{org-slug}.localhost.com:8080/{resource}

Assumption: You have set an org-slug (can do this on create or using the admin API) and an api-slug (can be set with dashboard under "Advanced" or just add an "api_slug": "{apiname}" to the api defintion object at the root)

If you haven't, then your URL's might look like this (ick):

{api-id}.{org-id}.localhost.com:8080/{resource}

Obviously if your API ID isn't a UUID then it will be whatever string you put in that field - this is only possible with file-based definitions.

I have a feeling that the proxy is acting up because nginx is just reverting to the first domain it finds matching localhost.com.

If you are using file based definitions, have you switched strip_listen_path to false?

If youa re not using file-based definitions, have you tried accessing the API proxy directly:

http://192.168.1.BLAH:8080/{api-id}/{resource}

All the host maanger does is create nginx configurations that translate the structure at the top of this answer into the one above.

Keep in mind, you can compeltely change how these redirects work, if you want to have a listen structure like:

{org-slug}.localhost.com:8080/{api-slug}/{resource}

Then all you need to do is modify (or override) the nginx template the host manager uses here, you can see there's a URL rewrite under the hood:

location / {
        rewrite /(.*) /{{.ApiId}}/$1 break;

        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_pass http://tyk;
    }

Unfortunately, the host manager is not clever enough yet to generate multiple API rewrites under a single org subdomain, it's an option we've thought of adding for those who want to have this kind of structure, however, you can always do it yourself if you have a fixed set of API names.

This is a long and rambling answer, so may not be very helpful, if you answer the top two questions I think I'll be able to offer more insight.

:-S

from tyk.

It's a hard one to diagnose, some questions may clarify th answer a bit:

1. Are you using the docker containers, the quickstart script or are you running each process raw?
2. Are you using the dashbaord or file-based definitions?

Yes, I'm using the docker containers and the quickstart script. I'm also using the dashboard to create the API definitions manually.

I am assuming you are using the host manager (one of the docker containers from the quickstart). If you are, then your url structure will look like this:
{api-slug}.{org-slug}.localhost.com:8080/{resource}

Yup, I've been trying with that URL structure as well with the same results.

Assumption: You have set an org-slug (can do this on create or using the admin API) and an api-slug (can be set with dashboard under "Advanced" or just add an "api_slug": "{apiname}" to the api defintion object at the root)

Correct, I have the slugs set on the APIs. And just to make sure no one else sees this and gets the wrong idea, it is "slug": "{apiname}" instead of the "api_slug" parameter.

Obviously if your API ID isn't a UUID then it will be whatever string you put in that field - this is only possible with file-based definitions.

I'm not using the file-based definitions, so my API_IDs are the UUIDs (confirmed by looking at the result of /api/apis as well).

I have a feeling that the proxy is acting up because nginx is just reverting to the first domain it finds matching localhost.com.

That's my feeling as well. Accessing the APIs by their API_ID directly works fine.

If youa re not using file-based definitions, have you tried accessing the API proxy directly:
http://192.168.1.BLAH:8080/{api-id}/{resource}

Yup, they work just fine accessing them in this way.

Keep in mind, you can compeltely change how these redirects work, if you want to have a listen structure like:

I'm definitely planning to change these redirects once I get it working the default way, thanks for pointing out how I will be able to do it.

So that's where I'm at right now, it seems like that issue is with the Host Manager doing the mappings based on everything else appearing to work as expected.

from tyk.

@DavidTPate

Ok, you can actually directly diagnose this on the docker container by looking at what it's generating, first off, connecto to the running tyk_nginx container and start a bas shell:

sudo docker exec -i -t tyk_nginx bash

The nginx configurations that are generated can be seen in the default nginx folder, just do:

ls -al /etc/nginx/nginx/conf.d/

There should be a few *.tconf files with long file names, these are the ID's of your API's, to see what was generated.

Take a look and see how many there are, there should be 6 files, two configurations for the portals, and four API configurations.

Would be interesting to see what you find... :-)

from tyk.

Looks about like what you are expecting:

A few things to note. The directory for the nginx configurations is /etc/nginx/conf.d/ and it is just symlinked to the config files at /opt/tyk-dashboard/host-manager/nginx_confs.

5fb3afce708c443e6046804aad2bb564.tconf
6582f6f6419243786f5871bb83184acb.tconf
bbd1f881d3d14ada6c02fd56d2d39c4b.tconf
e415d7cd4546498873370868295a70b0.tconf
_upstream.conf
test.admin.portal.tconf
www.admin.portal.tconf

from tyk.

@DavidTPate

Yes that looks right, Tyk will write to the local folder and NginX picks it up via the symlink. That's the way it's meant to be set up :-)

What do the files look like? Can you post them (test and www for only one organisation, should give us an idea of what's going on)

I setup my hosts file to point www.localhost.com and test.localhost.com to the Tyk server.

Have you set up your hosts file to take into account the additional subdomains? As in:

127.0.0.1 httpbin.test.localhost.com
127.0.0.1 google.test.localhost.com
127.0.0.1 httpbin.www.localhost.com
127.0.0.1 google.www.localhost.com
127.0.0.1 test.localhost.com
127.0.0.1 test.localhost.com

Realise it's obvious, just wanted to make sure ;-)

from tyk.

Yeah, my hosts file is setup correctly. No worries, sometimes it's the easiest stuff to miss.

192.168.1.xxx     www.localhost.com
192.168.1.xxx     test.localhost.com
192.168.1.xxx     httpbin.www.localhost.com
192.168.1.xxx     google.www.localhost.com
192.168.1.xxx     httpbin.test.localhost.com
192.168.1.xxx     google.test.localhost.com

Here's what the files look like:

5fb3afce708c443e6046804aad2bb564.tconf

server {

    server_name httpbin.www.tyk.docker;

    gzip              on;
    gzip_buffers      16 8k;
    gzip_comp_level   4;
    gzip_http_version 1.1;
    gzip_min_length   1280;
    gzip_types        text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
    gzip_vary         on;

    # The Go application server
    location / {
        rewrite /(.*) /5fb3afce708c443e6046804aad2bb564/$1 break;

        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_pass http://tyk;
    }
}

6582f6f6419243786f5871bb83184acb.tconf

server {

    server_name google.www.tyk.docker;

    gzip              on;
    gzip_buffers      16 8k;
    gzip_comp_level   4;
    gzip_http_version 1.1;
    gzip_min_length   1280;
    gzip_types        text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
    gzip_vary         on;

    # The Go application server
    location / {
        rewrite /(.*) /6582f6f6419243786f5871bb83184acb/$1 break;

        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_pass http://tyk;
    }
}

bbd1f881d3d14ada6c02fd56d2d39c4b.tconf

server {

    server_name httpbin.test.tyk.docker;

    gzip              on;
    gzip_buffers      16 8k;
    gzip_comp_level   4;
    gzip_http_version 1.1;
    gzip_min_length   1280;
    gzip_types        text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
    gzip_vary         on;

    # The Go application server
    location / {
        rewrite /(.*) /bbd1f881d3d14ada6c02fd56d2d39c4b/$1 break;

        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_pass http://tyk;
    }
}

e415d7cd4546498873370868295a70b0.tconf

server {

    server_name google.test.tyk.docker;

    gzip              on;
    gzip_buffers      16 8k;
    gzip_comp_level   4;
    gzip_http_version 1.1;
    gzip_min_length   1280;
    gzip_types        text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
    gzip_vary         on;

    # The Go application server
    location / {
        rewrite /(.*) /e415d7cd4546498873370868295a70b0/$1 break;

        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_pass http://tyk;
    }
}

test.admin.portal.tconf

server {

        server_name test.tyk.docker;

        gzip              on;
        gzip_buffers      16 8k;
        gzip_comp_level   4;
        gzip_http_version 1.1;
        gzip_min_length   1280;
        gzip_types        text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
        gzip_vary         on;

        # The Go application server
        location /portal/ {
            rewrite /portal/(.*) /5554fa4019406a0001000003/portal/$1 break;

            proxy_pass_header Server;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Scheme $scheme;
            proxy_pass http://dashboard;
        }
        location /portal-assets/ {
            proxy_pass_header Server;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Scheme $scheme;
            proxy_pass http://dashboard;
        }
        location /api/public {
            proxy_pass_header Server;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Scheme $scheme;
            proxy_pass http://dashboard;
        }
}

www.admin.portal.tconf

server {

        server_name www.tyk.docker;

        gzip              on;
        gzip_buffers      16 8k;
        gzip_comp_level   4;
        gzip_http_version 1.1;
        gzip_min_length   1280;
        gzip_types        text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
        gzip_vary         on;

        # The Go application server
        location /portal/ {
            rewrite /portal/(.*) /5554fa4019406a0001000002/portal/$1 break;

            proxy_pass_header Server;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Scheme $scheme;
            proxy_pass http://dashboard;
        }
        location /portal-assets/ {
            proxy_pass_header Server;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Scheme $scheme;
            proxy_pass http://dashboard;
        }
        location /api/public {
            proxy_pass_header Server;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Scheme $scheme;
            proxy_pass http://dashboard;
        }
}

I went through and pulled up the APIs to make sure that the organizations and the API IDs matched. Everything looked fine:

Organizations

{
  "organisations": [
    {
      "id": "5554fa4019406a0001000002",
      "owner_name": "WWW",
      "owner_slug": "www",
      "apis": [
        {
          "api_human_name": "HttpBin",
          "api_id": "5fb3afce708c443e6046804aad2bb564"
        },
        {
          "api_human_name": "Google",
          "api_id": "6582f6f6419243786f5871bb83184acb"
        }
      ],
      "developer_quota": 0,
      "developer_count": 0
    },
    {
      "id": "5554fa4019406a0001000003",
      "owner_name": "Test",
      "owner_slug": "test",
      "apis": [
        {
          "api_human_name": "HttpBin",
          "api_id": "bbd1f881d3d14ada6c02fd56d2d39c4b"
        },
        {
          "api_human_name": "Google",
          "api_id": "e415d7cd4546498873370868295a70b0"
        }
      ],
      "developer_quota": 0,
      "developer_count": 0
    }
  ],
  "pages": 0
}

Here's the results I get when I try to navigate to the different URLs with different ports.

Going to the APIs directly yields the results that I expect.

from tyk.

@DavidTPate

I think I see the problem :-)

Your hosts file says:

192.168.1.xxx www.localhost.com

NginX confs are listening out for:

server_name google.www.tyk.docker;

in your hosts file, change localhost.com to tyk.docker

Or change it the other way in the templates, NginX is trying to service an inbound request and since there's no default.conf it's picking the first one it finds.

(At least I hope this is the problem) :-D

from tyk.

That did it! Definitely makes sense now that you've pointed that out.

With this, I think I can continue on and start getting things setup with the templates. Thanks a bunch for your time, really appreciate it.

from tyk.