apigee / microgateway-core Goto Github PK

Edgemicro will only indicate when a path isn't found. Should also indicate whether or not a host was found too.

1. Change edgemicro to leverage encrypted KVM instead of Vault for new configurations.
2. Provide a script to migrate keys and cert from vault to encrypted KVM.

Currently the config headers section is defined as follows:

headers:
  x-forwarded-for: true
  x-forwarded-host: true
  x-request-id: true
  x-response-time: true
  via: true

It should also include the X-Forwarded-Proto header as well

Hi Apigee team,

This error page rendered will introduce a XSS vulnerability.

Detailed sample:

GET /"><script>alert(document.domain)</script> HTTP/1.1
Host: scdci0000097.cn.bg.corpintra.net:10080
Connection: Keep-Alive

HTTP/1.1 404 Not Found
Date: Fri, 28 Sep 2018 04:35:37 GMT
Connection: keep-alive
Content-Length: 89

{"message":"no match found for /&quot;><script>alert(document.domain)</script>","status":404}GET /"><script>alert(document.domain)</script>.html HTTP/1.1
Host: scdci0000097.cn.bg.corpintra.net:10080
Connection: Keep-Alive

HTTP/1.1 404 Not Found
Date: Fri, 28 Sep 2018 04:35:38 GMT
Connection: keep-alive
Content-Length: 94

{"message":"no match found for /&quot;><script>alert(document.domain)</script>.html","status":404}

Here's what I did:

1. Start apid and MGW
2. Send some API calls that include verifying the API key to show that the MGW works
3. Kill apid
4. Send the same API calls again
5. I get the stack trace below

Error long polling apid. Waiting 10 second then will retry... connect ECONNREFUSED 127.0.0.1:9010
error verify-api-key code=ECONNREFUSED, errno=ECONNREFUSED, syscall=connect, address=127.0.0.1, port=9010
error m=GET, u=/iloveapis, h=localhost:8000, r=::ffff:127.0.0.1:59505, s=200, name=Error, message=connect ECONNREFUSED 127.0.0.1:9010, code=ECONNREFUSED, stack=Error: connect ECONNREFUSED 127.0.0.1:9010
at Object.exports._errnoException (util.js:1029:11)
at exports._exceptionWithHostPort (util.js:1052:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1093:14)
error verify-api-key message=connect ECONNREFUSED 127.0.0.1:9010, code=ECONNREFUSED, errno=ECONNREFUSED, syscall=connect, address=127.0.0.1, port=9010
/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/async/lib/async.js:43
if (fn === null) throw new Error("Callback was already called.");
^

Error: Callback was already called.
at /Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/async/lib/async.js:43:36
at /Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/async/lib/async.js:723:17
at /Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/async/lib/async.js:167:37
at fx (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/microgateway-core/lib/plugins-middleware.js:414:13)
at Request._callback (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/microgateway-plugins/verify-api-key/index.js:44:13)
at self.callback (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/request/request.js:186:22)
at emitOne (events.js:96:13)
at Request.emit (events.js:191:7)
at Request.onRequestError (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/request/request.js:845:8)
at emitOne (events.js:96:13)
Removing the socket file as part of cleanup
/Users/gregbrail/homebrew/lib/node_modules/edgemicro/cli/lib/reload-cluster.js:96
var interval = respawnIntervalManager.getIntervalForNextSpawn(now);
^

TypeError: respawnIntervalManager.getIntervalForNextSpawn is not a function
at replaceWorker (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/cli/lib/reload-cluster.js:96:43)
at EventEmitter.replaceAndTerminateWorker (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/cli/lib/reload-cluster.js:120:5)
at emitOne (events.js:96:13)
at EventEmitter.emit (events.js:191:7)
at emit (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/cli/lib/reload-cluster.js:56:15)
at EventEmitter.emitWorkerDisconnect (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/cli/lib/reload-cluster.js:168:5)
at emitOne (events.js:96:13)
at EventEmitter.emit (events.js:191:7)
at ChildProcess.worker.process.once (internal/cluster/master.js:211:13)
at Object.onceWrapper (events.js:293:19)
$

As per release notes, the configuration for HTTP_PROXY were broken in 2.3.3 version.
This is what we are observing in 2.3.5 as well.
Is this fixed now, or is there another way that this functionality can be achieved.

CC: @srinandan @mdobson

We created a plugin to set targetpath same as request url. This allowed appropriate path to be sent to backend.

Test Execution results
Three proxies were created as follows -

1. edgemicro_abc - /abc
2. edgemicro_abc__ - /abc/*
3. edgemicro_abc_xyz - /abc/*/xyz

Request path: /abc
Proxy: edgemicro_abc
Target: /abc
Test Status: PASS

Request path: /abc?x=y
Proxy: edgemicro_abc
Target: N/A. Proxy returned 403
Test Status: FAIL

Request path: /abc/123
Proxy: edgemicro_abc__
Target: /abc/123
Test Status: PASS

Request path: /abc/123?x=y
Proxy: edgemicro_abc__
Target: /abc/123?x=y
Test Status: PASS

Request path: /abc/123/xyz
Proxy: edgemicro_abc__xyz
Target: /abc/123/xyz
Test Status: PASS

Request path: /abc/123/xyz?x=y
Proxy: edgemicro_abc__xyz
Target: N/A. Proxy returned 403
Test Status: FAIL

Request path: /abc/123/11
Proxy: edgemicro_abc__
Target: /abc/123/11
Test Status: PASS

Request path: /abc/123/11?x=y
Proxy: edgemicro_abc__
Target: /abc/123/11?x=y
Test Status: PASS

Request path: /abc/123/xyz/11
Proxy: edgemicro_abc__xyz
Target: /abc/123/xyz/11
Test Status: PASS

Request path: /abc/123/xyz/11?x=y
Proxy: edgemicro_abc__xyz
Target: /abc/123/xyz/11?x=y
Test Status: PASS

Edge Microgateway doesn't execute the plugins for the request with invalid paths. Few plugins should be executed no matter the type of request. For instance, we have written a custom plugin which processes the X-Forwarded-For header of the request to get the actual client IP but that plugin never executes for the request paths that are not configured in edgemicro_* proxies.

Currently, overriding a header from the target response in the source
response requires doing it in both ondata_response (when receiving the first
chunk of data) and in onend_response (for when the response doesn't contain
any data), after verifying that res.headersSent is false. That's clumsy at
best.

I've proposed a fix in #129 a long time ago, but never got any feedback.

I've noticed that microgateway-core depends on microgateway-config, although it's not used.
During the 3.1.3 update, the version for microgateway-core was changed, but the dependency (npm-shrinkwrap) not updated.
Now, when installing edgemicro, I have two different versions of microgateway-config.

Can the dependency be removed from this repo? (The differing versions lead to a problem elsewhere for me)

EDIT: Got my edgemicro/microgateways all mixed up ;-)

This should be a fairly straightforward change. We'll have to add configuration options for key and cert files per host. Then we can update the HTTP agent to use these based on the hostname for the backend request.

Hello

By default, EMG checks server certificates' trust using SSL to connect to target backend server. This is https default setting.

This setting is not practical when testing against a server which only has self-signed certificates. Unfortunately, there's no possibility to let EMG set rejectUnauthorized https option to allow self-signed certificate.

Could you add an option to EMG to enable using self-signed certificates ?

All the best

NB: The workaround to to add rejectUnauthorized:false to targetRequestOptions in plugins-middleware.js

Provide a mechanism for custom plugins to rewrite/change the target endpoint. We need to discuss how TLS parameters will be passed (if set). Will those still come from the config.yaml file?

Hello All,

I have a plugin that overrides the target URL for one set in the plugin configuration.

On this plugin, i'm changing the req.targetHostname and req.targetPath and, if the target protocol is https, i'm also changing the req.targetSecure to true and the req.targetPort to 443

req.targetHostname = targetPathHostname // new target hostname
req.targetPath = targetPath // new target path

if (targetPathProtocol === 'https') {
  req.targetSecure = true
  req.targetPort = 443
} 

With this, when the request is executed, i'm getting the following error:

curl -i http://localhost:8000/health
HTTP/1.1 500 Internal Server Error
Date: Wed, 24 May 2017 12:57:14 GMT
Connection: keep-alive
Content-Length: 67

{"message":"Protocol \"https:\" not supported. Expected \"http:\""}

Just to be sure, i've configured my edgemicro instance to work with ssl but the error was the same:

curl -i https://localhost:8000/health
HTTP/1.1 500 Internal Server Error
Date: Wed, 24 May 2017 12:57:14 GMT
Connection: keep-alive
Content-Length: 67

{"message":"Protocol \"https:\" not supported. Expected \"http:\""}

Is it possible to do this? Change the target URL and also change from HTTP to HTTPS?

I'm using the following versions:

current nodejs version is v6.10.0
current edgemicro version is 2.4.5-beta

Helo all,

In your documentation Rewriting target URLs in plugins, it mentions:

You can override the default target URL dynamically in a plugin by modifying these variables in your plugin code: req.targetHostname and req.targetPath.

I have a plugin that needs to override the target URL for one set in a custom configuration.

I made some tests using the version:

current nodejs version is v6.10.1
current edgemicro version is 2.3.3

In here I'm passing a fixed new URL, but the final one would be caught from configuration:

var EDGEMICRO_HEALTH = '/health'
if (req.url === EDGEMICRO_HEALTH) {
    debug('%s endpoint was called with mode %s', EDGEMICRO_HEALTH, _healthMode)    
    debug('request ==>', Object.keys(req))
    debug('port ==>', req.targetPort)
    debug('targetHostname before==> %s', req.targetHostname)
    req.targetHostname = 'localhost:3000'
    debug('targetHostname after ==> %s', req.targetHostname)
    next()       
}

I have a local app running in localhost:3000 returning '200 OK' just for testing, if i call "http://localhost:8000/health", I get in the logs:

plugin:healthcheck /health endpoint was called with mode forward +6s
  plugin:healthcheck request ==> [ '_readableState',
  'readable',
  'domain',
  '_events',
  '_eventsCount',
  '_maxListeners',
  'socket',
  'connection',
  'httpVersionMajor',
  'httpVersionMinor',
  'httpVersion',
  'complete',
  'headers',
  'rawHeaders',
  'trailers',
  'rawTrailers',
  'upgrade',
  'url',
  'method',
  'statusCode',
  'statusMessage',
  'client',
  '_consuming',
  '_dumped',
  'reqUrl',
  '_overrideHeaders',
  '_headersToUnset',
  'setOverrideHeader',
  'unsetHeader',
  'targetPath',
  'targetHostname' ] +0ms
  plugin:healthcheck port ==> undefined +3ms
  plugin:healthcheck targetHostname before==> localhost +1ms
  plugin:healthcheck targetHostname after ==> localhost:3000 +0ms

So, as you can see, I'm able to change the targetHostname adding the port, although there is no property in request object regarding to port itself. I tried to force a req.targetPort to see, but its undefined. And printing the properties in request object, there is nothing close to what i need.

The thing is, the answer from EMG after the call is:

 gateway:main targetRequest error +3ms dc6cc1c0-2c6a-11e7-afe7-bde5ed24fc46 Error: getaddrinfo ENOTFOUND localhost:3000 localhost:3000:8080
    at errnoException (dns.js:28:10)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)
  gateway:main plugin analytics does not provide handler function for error_request +2ms
  gateway:main plugin healthcheck-plugin does not provide handler function for error_request +1ms
  gateway:errors Error: getaddrinfo ENOTFOUND localhost:3000 localhost:3000:8080
    at errnoException (dns.js:28:10)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26) +1ms
  gateway:main sourceRequest close +8ms

As it can be seen, EMG keeps the original port (which in this case was 8080 as defined in apigee cloud - target endpoint is http://localhost:8080), so it concatenate the changed target hostname with the original port, so localhost:3000:8080, which obviously does not work.

Also, i believe it works the same for the HTTP scheme..it seems that there is no way to change from HTTP to HTTPS and HTTPS to HTTP as well.

I need to change the full target endpoint. If in apigee cloud the target is:
http://someurl:8080
I need to be able to change for instance to:
https://otherurl:9090

Is that possible somehow? Or even, is there a way for you to provide a change where its possible to override the full target endpoint?

Thanks
Ciro

The stats object in lib/stats.js isn't used for anything. Next release should deprecate it's use, and it should be removed in a subsequent release.

Can you please add property which would allow to increase connection timeout between EdgeMicro and Target Server ? Currently EdgeMicro drops the connection after 60 seconds.

As a user I should be able to disable polling edge for new configuration.

I tried un-staging the bundle that was staged to the gateway while the MGW was running. The gateway crashed.

Upon restart, I get the output below.

I think that if nothing is staged, the gateway should remain up and running, awaiting something to be deployed.

edgemicro start -s ./systemConfig.yaml -a http://localhost:9010
current nodejs version is v7.7.1 current edgemicro version is 3.0.0-early-access
undefined:1

SyntaxError: Unexpected end of JSON input
at JSON.parse ()
at Request._callback (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/apid.js:52:31)
at Request.self.callback (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/request/request.js:186:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:194:7)
at Request. (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/request/request.js:1081:10)
at emitOne (events.js:96:13)
at Request.emit (events.js:191:7)
at IncomingMessage. (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/request/request.js:1001:12)
at Object.onceWrapper (events.js:293:19)

Hello

EMG logging produces the following output:

info installed plugin from cors 

info installed plugin from router-plugin 

info 0bd05a00-221d-11e8-a021-4dd0443402a1 edge micro listening on port 8090 uid=0bd05a00-221d-11e8-a021-4dd0443402a1, port=8090

info sourceRequest m=GET, u=/ping, h=127.0.0.1:8090, r=::ffff:127.0.0.1:33712, i=2d63eec0-221d-11e8-a021-4dd0443402a1

info REQ verb/host/path: GET 127.0.0.1:8090/ping 

info Searching configuration for 127.0.0.1:8090:/ping:GET 

info Found a match for 127.0.0.1:8090:/ping:GET in regexps .*:.*:.* 

info targetRequest m=GET, u=/, h=localhost:3000, i=2d63eec0-221d-11e8-a021-4dd0443402a1

info targetResponse s=404, d=37, i=2d63eec0-221d-11e8-a021-4dd0443402a1

Each EMG log line is followed by a blank line. In our deployment, EMG is used in a micro-services architecture with log aggregation. These blank log result in useless records in our kibana servers.

Please fix your logging system to avoid these blank lines.

All the best

looks like we do not have any faulthandlers for faults/errors raised by plugins

today ,we have the following request handlers - onrequest, ondata_request, onend_request, onclose_request, onerror_request
and the following response handlers - onresponse, ondata_response, onend_response, onclose_response, onerror_response

none of them are invoked when an error is raised by plugins - for eg, invalid_auth by oauth plugins

we need a way to handle these faults - either a new fault handler or a faultSequence to execute during faults

@gbrail @alexkhimich brought this issue to my attention. Can you send me the logs for this?

Edgemicrogateway loads all API proxies in an environment that begin with the name "edgemicro_". We should provide the ability to load some/few proxies. I propose introducing a new configuration in the config.yaml file

`proxies:

• name: edgemicro_httpbin
  revision: '1'
• name: edgemicro_twoway
  revision: '1'
  `

At start up, EM should download the configuration (basePath, target endpoint etc.) only for the proxies listed here.

If the proxies configuration is not present in the config.yaml file, the current behaviour should continue.

If I unstage a bundle -- it pretty much doesn't matter if there are multiple bundles staged or just one -- I get a long list of messages saying that the gateway PUT deployment status, followed by a long list of the following stack trace, followed by a crash:

TypeError: Cannot read property 'logging' of undefined
at Object.init (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/microgateway-core/lib/logging.js:27:34)
at new Gateway (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/microgateway-core/index.js:23:11)
at module.exports (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/microgateway-core/index.js:30:10)
at init (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/lib/gateway.js:10:27)
at Agent.start (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/lib/server.js:40:24)
at startServer (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/lib/agent-config.js:24:9)
at getConfigStart (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/lib/agent-config.js:20:3)
at configureAndStart (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/lib/agent-config.js:15:3)
at Object. (/Users/gregbrail/homebrew/lib/node_modules/edgemicro/cli/lib/start-agent.js:16:1)
at Module._compile (module.js:571:32)
/Users/gregbrail/homebrew/lib/node_modules/edgemicro/node_modules/microgateway-core/lib/logging.js:27
const logConfig = config.system.logging;

The configuration file for EM to support TLS northbound is:

 edgemicro:
     ssl:
         key: <absolute path to the SSL key file>
         cert: <absolute path to the SSL cert file>

EM should support the ability to supply an encrypted key (and therefore allow passing a passphrase).

Please see here: https://community.apigee.com/questions/35752/change-default-config-directory.html

FYI: @mdobson

Hi there,I found an issue when using edgemicro tracing function. I have one project with opentracing call edgemicro and edgemicro forward to my backend server.

the call chain is like this:

projectA with opentracing -> edgemciro -> projectB with opentracing

I use jaeger backend,but in fact, edgemicro not send its trace information to my jaeger backend.
And my edgemicro start command is as below:

EDGEMICRO_OPENTRACE=true EDGEMICRO_TRACE_MODULE=path/to/tracer edgemicro start -o internal -e development -k xxxx -s xxx

and my tracer core code is as below, I use jaeger client in initTracer function:

module.exports = {
	 initTracerMod: function(name) {
	 	const tracer = initTracer(config, options)
		return tracer;
	}
}

I this it's ok, but edgemicro trace function not works well.so I try to find root cause in source code.At last I found an issue in source code at this file https://github.com/apigee/microgateway-core/blob/master/lib/trace-helper.js

at line 32:

 `requestspan = proxyTrace.extract(FORMAT_HTTP_HEADERS, req.headers);`

proxyTrace.extract function need to return an spanContext, not a span.and at line 37-39, use it to set tag

 requestspan.setTag(Tags.HTTP_URL, req.url);
 requestspan.setTag(Tags.HTTP_METHOD, req.method);

here will occur setTag is not a function error, but in source code it catch the err but do nothing, so the error will gone.the trace information will not be sent as well.

That's all about edgemicro tracing issue.Hope it will be fixed as soon.

thanks!

I'm working through the first custom plugin tutorial for edgemicro found here: http://docs.apigee.com/microgateway/latest/develop-plugins. I've run into some peculiar behavior though.

Here is my plugin code:

'use strict';

var debug = require('debug')('plugin:response-override');


module.exports.init = function(config, logger, stats) {


  return {

    ondata_response: function(req, res, data, next) {
      debug('***** plugin ondata_response');
      next(null, null);
    },

    onend_response: function(req, res, data, next) {
      debug('***** plugin onend_response');
      next(null, "Hello, World!\n\n");
    }
  };
}

Instead of replacing the response body with custom text. My plugin simply appends the text to the current response body.

HTTP/1.1 200 OK
content-type: application/json; charset=utf-8
date: Mon, 10 Oct 2016 14:31:11 GMT
etag: W/"1cd-igZDs61JMAiJo6ZDx/UFDQ"
x-powered-by: Apigee
x-response-time: 635
Connection: keep-alive
Transfer-Encoding: chunked

{"headers":{"host":"mocktarget.apigee.net","accept":"*/*","user-agent":"curl/7.43.0","via":"undefined1.1 localhost","x-authorization-claims":"eyJzY29wZXMiOltdfQ==","x-forwarded-host":"undefinedlocalhost:8000","x-request-id":"d68c00d0-8ef5-11e6-8c68-5758f301a7dc.30bc13b0-8ef6-11e6-8c68-5758f301a7dc","x-forwarded-for":"undefined::1, 68.49.164.231","x-forwarded-port":"80","x-forwarded-proto":"http","connection":"keep-alive"},"method":"GET","url":"/","body":""}Hello, World!

The passphrase for TLS (key and pfx files) are stored in the clear in the config.yaml file. We need a more secure way to do this.

1. Get this information from APID. The keys/certs are going to be there anyway.
2. Get it from the environment variable
3. Get it from a secure vault/storage

Hi there, when logging to console the microgateway does not record a time stamp. The relevant piece of code is in lib/logging.js - line 232.
const preamble = logToConsole ? '' : Date.now() + ' '
By removing logToConsole ? '' : I can get it to log the unix timestamp to the console. Can you explain why this logic is in there and if it is possible to remove it or add a config parameter for logging timestamp when using console logging?
Cheers, Swithin.