apocas / docker-modem Goto Github PK

At the moment if Docker connection fails then the modem can hang requests. However when using timeout combine with Attach then the two seem to conflict. Having set my timeout to 1000, it will also still the on-going attach connection even if it has successfully connected.

The node.js documentation says: "Sets the socket to timeout after timeout milliseconds of inactivity on the socket" where as I want the timeout to be on the initial connection.

Proposed Solution:
On Connect / Data / Error we clear the timeout.

Debug log:
modem Sending: { path: '/containers/create?Image=',
method: 'POST',
port: '2376' } +0ms
modem Received: {"Id":"a54e6dc2a17dd84493bf1a9335e499bb80a659be990d268cf2e9185cd99db28f","Warnings":null}
+154ms
modem Sending: { path: '/containers/a54e6dc2a17dd84493bf1a9335e499bb80a659be990d268cf2e9185cd99db28f/attach?stream=true&stdin=true&stdout=true&stderr=true',
method: 'POST',
port: '2376' } +3ms
modem Sending: { path: '/containers/a54e6dc2a17dd84493bf1a9335e499bb80a659be990d268cf2e9185cd99db28f/start',
port: '2376' } +29ms
modem Received: +199ms
modem Sending: { path: '/containers/a54e6dc2a17dd84493bf1a9335e499bb80a659be990d268cf2e9185cd99db28f/resize?w=100&h=25',
method: 'POST',
port: '2376' } +2ms
modem Sending: { path: '/containers/a54e6dc2a17dd84493bf1a9335e499bb80a659be990d268cf2e9185cd99db28f/resize?w=118&h=25',
method: 'POST',
port: '2376' } +5ms
modem Received: +47ms
modem Received: +15ms
modem Timeout of 1000 ms exceeded +971ms

Is there reason we don't just pass an object like?

{
  "DOCKER_HOST": "unix:///var/run/docker.sock",
  "DOCKER_TLS_VERIFY": "0"
}

Originally opened by @sureshsankaran in microsoft/vscode-docker#1169

Test setup:
Docker extension version - Alpha version built from source code as it is on July 27, 2019.
Docker engine on remote machine - 18.03 version

I was trying to connect to remote docker engine by setting config parameters corresponding to
DOCKER_HOST and DOCKER_CERT_PATH. Didn't set any value for "DOCKER_TLS_VERIFY" associated setting. Now the extension throws the error saying invalid server certificate, mismatched common name in the cert. I understand that the IP address used to connect to the docker engine is different from the one present in ssl cert common name. But this is the expected behavior with DOCKER_TLS_VERIFY=1. But why i am getting this error without even setting VERIFY=1.

Expected behavior:
Extension should successfully connect to the remote docker engine without having to verify the server ssl cert.

When loading a saved docker image from a file, the readFile operation is done synchronously, thus blocking Node.js event loop for the duration of the operation. A docker image export file can be quite big, which would result in other tasks to starve, since they would not get any CPU. An asynchonous file read approach should be preferred.

Is there a reason for reading these file synchronously?

The related snippet:

if (options.file) {
    if (typeof options.file === 'string') {
      data = fs.readFileSync(path.resolve(options.file));
    } else {
      data = options.file;
    }
    optionsf.headers['Content-Type'] = 'application/tar';
  } else if (opts && options.method === 'POST') {
    data = JSON.stringify(opts._body || opts);
    optionsf.headers['Content-Type'] = 'application/json';
}

Getting the following error, using latest version of docker-modem.

TypeError: Cannot call method 'request' of undefined
    at Modem.buildRequest (/Users/knownasilya/sandbox/strider-docker-build/node_modules/dockerode/node_modules/docker-modem/lib/modem.js:143:33)
    at Modem.dial (/Users/knownasilya/sandbox/strider-docker-build/node_modules/dockerode/node_modules/docker-modem/lib/modem.js:138:8)
    at Docker.buildImage (/Users/knownasilya/sandbox/strider-docker-build/node_modules/dockerode/lib/docker.js:146:14)

Options passed to dial are:

{
  path: '/build?',
  method: 'POST',
  file: '/var/folders/_l/79gd0f_50pn7f_nnhbymjf5mxh060c/T/archive-docker-build-1419862803621.tar',
  options: { t: 'dockerreg:5000/name-of-proj', q: false },
  isStream: true,
  statusCodes: { '200': true, '500': 'server error' }
}

Hi,
while it doesn't prevent my test suite from running, I keep getting these errors in the console for which docker-modem seems to be the root cause:

  console.error node_modules/jest-jasmine2/build/jasmine/Env.js:289
    Unhandled error

  console.error node_modules/jest-jasmine2/build/jasmine/Env.js:290
    Error: (HTTP code 500) server error - a disk usage operation is already running 
        at /home/jean/dev/startups/yupwego/src/yupwego/node_modules/docker-modem/lib/modem.js:257:17
        at getCause (/home/jean/dev/startups/yupwego/src/yupwego/node_modules/docker-modem/lib/modem.js:287:7)
        at Modem.Object.<anonymous>.Modem.buildPayload (/home/jean/dev/startups/yupwego/src/yupwego/node_modules/docker-modem/lib/modem.js:256:5)
        at IncomingMessage.<anonymous> (/home/jean/dev/startups/yupwego/src/yupwego/node_modules/docker-modem/lib/modem.js:232:14)
        at IncomingMessage.emit (events.js:187:15)
        at endReadableNT (_stream_readable.js:1094:12)
        at process._tickCallback (internal/process/next_tick.js:63:19)

docker version reports

 docker version                                                                                                                                                                          ±[●●][master]
Client:
 Version:           19.03.3-ce
 API version:       1.40
 Go version:        go1.13.1
 Git commit:        a872fc2f86
 Built:             Wed Oct  9 02:28:51 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.3-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.1
  Git commit:       a872fc2f86
  Built:            Wed Oct  9 02:28:23 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.0.m
  GitCommit:        d50db0a42053864a270f648048f9a8b4f24eced3.m
 runc:
  Version:          1.0.0-rc9
  GitCommit:        d736ef14f0288d6993a1845745d6756cfc9ddd5a
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

uname -srv reports :

Linux 5.3.6-1-MANJARO #1 SMP PREEMPT Sat Oct 12 09:30:05 UTC 2019

using docker-modem "version": "1.0.9"
which is pulled by testcontainers: "version": "2.1.0"
in my project.

Hi Pedro,

I'm using your docker-modem to try to stream over thousands of containers.
But it seems that over http your method Attach completly freeze all my http request.

What i'm doing:

var Docker  = require('dockerode');
var docker  = new Docker({ socketPath: false, host: host, port: port});
var container = docker.getContainer(container.id);
container.attach({stream: true, stdout: true, stderr: true}, function(err, stream) {
  stream.on('data', function (data) {
   console.log(data.toString());
  });
});

After executing this exactly 5 times with different container.id all my http request to my single docker server are hanging.
IE:

setInterval(function() {
  request.get("http://********:4243/", function (err, data, body) {
    console.log("blih");
  });
}, 2000);

Nevers output "blih" after 5 times calling .attach()

I'll try to debug today but I wanted to know if someone else had the same problem.

regards,

Found an interesting situation that i was able to track down a bit. I am unsure if this is a bug with docker-modem or potentially how Docker for Windows sets up the named pipe to begin with.

Here is a file to reproduce (uses latest node-docker-api but upgraded to docker-modem 1.0.0), the same thing happens if you use analogous code for dockerode

'use strict'
const Docker = require('node-docker-api').Docker

const promisifyStream = (stream) => new Promise((resolve, reject) => {
  stream.on('data', (d) => console.log(d.toString()))
  stream.on('end', () => {
      if (process.stdin.setRawMode) {
        process.stdin.setRawMode(false);
      }
      process.stdin.pause();
      resolve()
  })
  stream.on('error', reject)
})

var thing
const docker = new Docker({ socketPath: '//./pipe/docker_engine' })
//const docker = new Docker({host: '127.0.0.1', port: 2375})
let _container

  docker.container.create({
    Image: 'debian:jessie',
    Cmd: [ '/bin/bash', '-c', 'tail -f /var/log/dmesg' ]
  })
  .then((container) => container.start())
  .then((container) => {
    _container = container
    return container.exec.create({
	    AttachStdin: true,
	    AttachStdout: true, 
	    AttachStderr: true,
	    Cmd: ['echo', 'thing'],
	    Tty: true
    })
  })
  .then((exec) => {
    return exec.start({ 
      hijack: false,
      Detach: false,
      Tty: true,
      stdin: true,
    })
  })
  .then((stream) => {
  	thing = stream
  })
  .then(() => {
  	thing.pipe(process.stdout);    

  	// Restart stdin with correct encoding
  	process.stdin.resume();
  	process.stdin.setEncoding('utf8');

  	// Make sure rawMode matches up
  	if (process.stdin.setRawMode) {
  	  process.stdin.setRawMode(true);
  	}        

  	// Send our processes stdin into the container
  	process.stdin.pipe(thing);

  })
  .then(() => promisifyStream(thing))
  .then(() => _container.delete({force: true}))
  .catch((error) => console.log(error))

If you connect over tcp (need to allow insecure connections in the docker for windows app first), it works as expected but if you switch over to the named pipe it will hang after "thing" is written to stdout. What is extremely interesting is after this happens you can get the stream to close and the process to resume by pressing any key three times.

Another interesting data point here is it looks like docker-compose itself avoids using the API for this on Windows, deciding to just delegate directly to a shell call to docker exec.
see: https://github.com/docker/compose/blob/master/compose/cli/main.py#L411

My best guesses at this point are:

1. there is is some additional buffer with named pipes that is not being correctly flushed and its causing things to hang up
2. writes are not happening or getting lost duringish reads

Any thoughts about why this could be happening?

Currently the docker api returns a multiplexed buffer for log files when follow=false. As it is not a stream the current function of demuxStream doesn't function with it. This is an idea I had in mind for lib/modem.js along side demuxStream

Modem.prototype.demuxBuffer = function(buffer) {
  return new Promise((res, rej) => {
    var nextDataType = null;
    var nextDataLength = null;
    var output = {
      stdout: [],
      stderr: []
    }
    function processData() {
      if (!nextDataType) {
        if (buffer.length >= 8) {
          var header = bufferSlice(8);
          nextDataType = header.readUInt8(0);
          nextDataLength = header.readUInt32BE(4);
          // It's possible we got a "data" that contains multiple messages
          // Process the next one
          processData();
        }
      } else {
        if (buffer.length >= nextDataLength) {
          var content = bufferSlice(nextDataLength);
          if (nextDataType === 1) {
            output.stdout.push(content);
          } else {
            output.stderr.push(content);
          }
          nextDataType = null;
          // It's possible we got a "data" that contains multiple messages
          // Process the next one
          processData();

          // Done processing, send data
          if (buffer.length == 0) {
            res(output);
          }
        }
      }
    }

    function bufferSlice(end) {
      var out = buffer.slice(0, end);
      buffer = Buffer.from(buffer.slice(end, buffer.length));
      return out;
    }

    processData();
  });
}

I'm trying to use a pull (docker pull, to pull from a registry, in my case it's red:5000/image-name).
I noticed that in function: Modem.prototype.dial, it is escaping the path when adding the opts to the address (address += querystring.stringify(opts);)

When it comes to the Modem.prototype.buildRequest, and creates the http req (var req = http[self.protocol].request(options, function() {});), this function also escapes the path.
so now the path will look like /images/create?fromImage=red%253A5000%252Fimage-name

So it seems like it does a double escape (which I assume the docker server doesn't like, since I get a response back :
Invalid repository name (red%3A5000%2Fimage-name), only [a-z0-9-_.] are allowed

So with docker 1.7.0 and forward, the build endpoint (and possibly others) wants the X-Registry-Config header which is the base64 encoded contents of the .docker/config.json or the .dockercfg file (the first being the new formatted config file, the second being the legacy location.

A quick fix for this would be at Line 119 of modem.js simply putting optionsf.headers['X-Registry-Config'] = optionsf.headers['X-Registry-Auth'] but I am unsure if this is the right move. I could see putting in a new option registryconf or something in the options to populate this file and then just doing = options.registryconfig || optionsf.headers['X-Registry-Auth']

Either way this is a change that needs to be made soon to keep up with the docker api versions, without it, the build process will not work with version 1.7.x and greater.

Docker api docs say that X-Registry-Auth should be base64urlencoded (here), but what I see from the code in the version 3.2.1 the value passed to this header is simply base64 encoded. (here)

This lead our company (codefresh.io) to errors when using credentials from Azure Container Registry for one of the customers.

Please check ASAP

The regex parsing the DOCKER_HOST URI does not support IPv6 IP addresses:

E.g., 'tcp://[::1]:12345' would split at the second colon instead of the last. You could use Node.js' 'url' module instead of a regex.

https://github.com/apocas/docker-modem/blob/master/lib/modem.js#L112

I don't think it needs to be super structured but likely should be an actual error object.

Hello,

i can see that version 2.1.4 is latest.
improvements were done, in particular the ability to connect with ssh and private key but they are not on available on npm.

can you push a new version to npm with current source code from github please ?

It looks like the recent release which updated JSONStream from 0.10.0 to 1.3.2 has broken followProgress().
With version 1.0.5 onFinished, onProgress callbacks are no longer being called.

Problem

This is the same as the issue here, on a pull the docker daemon responds to with an error: context cancelled. The issues seems to be a couple of things:

• the Content-Length is calculated using a data payload that is never sent in the request. Perhaps the query string should be used https://github.com/apocas/docker-modem/blob/master/lib/modem.js#L167 ?
• undefined opts are passed in as 'undefined' strings, eg: tag, leading to empty query params
• the Content-Type is set as application/json though no json payload is sent, maybe application/x-www-form-urlencoded or text/plain?

Here is an example failing request options, changing the type to text/plain or removing the Content-length fixes the issue.

 { path: '/images/create?fromImage=coinstac%2Fcoinstac-base&tag=',
   method: 'POST',
   headers: { 'Content-Type': 'application/json', 'Content-Length': 38 },
   key: undefined,
   cert: undefined,
   ca: undefined,
   socketPath: '//./pipe/docker_engine' }

Would it be possible for docker-modem to also support named pipes?
On windows it's the de-facto:
https://github.com/docker/engine-api/blob/master/client/client_windows.go

With Windows 2016 container in the works it'd be nice to have this available right off the bat.

Hello and thanks for terrific package :)

I followed the examples in the readme and did:

const options = {
    protocol:'http',
    host: '127.0.0.1',
    port: 2375
}

But it would still connect to the default unix socket. Finally I figured out that explicitly setting socketPath to null I could get it to respect the options. I.e. this works:

const options = {
    protocol:'http',
    host: '127.0.0.1',
    port: 2375,
    socketPath: null
}

the solution lies under here, it should first check for http then go for socket option;
in lib/modem.js

if (this.socketPath) {
    optionsf.socketPath = this.socketPath;
  } else {
    var urlp = url.parse(address);
    optionsf.hostname = urlp.hostname;
    optionsf.port = urlp.port;
    optionsf.path = urlp.path;
  }

Instead of passing the ca.pem/cert.pem/key.pem, is it possible to pass a pfx similar to the request library?

This seems like it would be really easy to do - I can take an attempt at it locally as well.

Thanks for this library, it rocks.

We are using docker engine swarm mode and one scenario is:
On one swarm worker, our service, which is running inside a docker container, will make some remote api calls to swarm manager leader. Now we want to enable the TLS for docker daemon on the manager leaders.
To make those remote api calls, we use dockerode module, and what we want to do is to set the headers(Host: ), then make the remote call by IP, not the common name, so we don't have to set it in /etc/hosts to tell what IP the common name will be resolved to.

e.g.

https://github.com/apocas/dockerode

var docker = new Docker({
  protocol: 'https',
  host: '192.168.1.10',
  headers:{Host: 'my-common-name'}, // we need it!!!
  port: process.env.DOCKER_PORT || 2375,
  ca: fs.readFileSync('ca.pem'),
  cert: fs.readFileSync('cert.pem'),
  key: fs.readFileSync('key.pem')
});

And in docker-modem:

var optionsf = {
    path: address,
    method: options.method,
    // headers when making this dial, if not, then the headers when new the modem object, and 
    // then no headers
    headers: options.headers || self.headers || {},
    key: self.key,
    cert: self.cert,
    ca: self.ca
  };

So playing with attach streams... specifically for docker exec, I've found that you can't end the stdin stream... The HttpDuplex.end() implementation breaks the TCP stream - not just closing this side.

So if you do the equivalent of echo "test" | docker exec CONTAINER wc -c you don't get the expected output.

Looking at the docker implementation, it seems that we should:

1. Set Connection: upgrade and Upgrade: tcp headers, see docker cli code.
2. And then just hijack the raw TCP socket and forget all about HTTP, or ensure that HttpDuplex only closes this side of the request (should be the same right?).

Either way, this seems to fix my issue:

HttpDuplex.prototype.end = function(chunk, encoding, cb) {
  var req = this.req;
  return req.end(chunk, encoding, function() {
    req.socket.end();
    if (cb) {
      cb();
    }
  });
};

Not sure why I'm having to close the TCP socket. It ought to just end on it's own.
Current code does this._output.socket.destroy(); which is probably why I miss any output arriving after stdin is closed.

After attaching and writing data to stdin I need to close it somehow. I have some code that is being executed any expects the stdin in the container to be closed at a certain poinrt. Is there a way for this to be accomplished?

With the support for ssh:// for the Docker host in, it would be great if .ssh/config would be supported like the Docker CLI does too. There is the ssh2-config NPM package helping with that.

This came up from microsoft/vscode-docker#1292 originally. It seems like Node.js does DNS lookups every single HTTP request: nodejs/node#5893

Supposedly, this can be resolved by configuring a system-wide DNS cache, or using packages like dnscache.

Can docker-modem make use of that package, or make it an option, in order to reduce DNS traffic in things like the vscode-docker extension?

Hello,

Looking at:

As the header X-Registry-Config is bound to contain the user docker configuration which holds all her credentials to the different repositories she is actually logged in, these information are also sensible and should not appear in the query string.
So I think opts.registryConfig should also be removed from the options in order to not show up in the query string.

@apocas I can provide a PR if necessary but you might want to look into it first.

Vulnerability is in ms package used by debug. Please refer to https://snyk.io/vuln/npm:ms:20151024.

debug package v2.6.7 has addressed this issue.

DOCKER_HOST has the following form:

$IP:$PORT
(for example)
127.0.0.1:4243

Ideally I would like to pass DOCKER_HOST directly or some other compliant thing so I can configure the docker CLI and everything else will "just" work with those details.

I'm invoking the top method of dockerode like so

docker.getContainer(containerId).top({ps_args: 'aux'}, fn);

and docker-modem is building a URL like

.../top?ps_args=aux?ps_args=aux

This is causing a parsing error on the server.

Some of the logic in modem.js seems a bit confused.

On line 38 there's a check for a ? character. If it passes and opts is set the query string is added to address—which becomes optionsf.path.

if(options.path.indexOf('?') !== -1) {
  if (opts && Object.keys(opts).length > 0) {
    address += querystring.stringify(opts);
  } else {
    address = address.substring(0, address.length - 1)
  }
}

Then on line 80, address is parsed and optionsf.path is reset to optionsf.path + "?" + querystring.stringify(opts);. It seems like you may want to use urlp.pathname + "?" + querystring.stringify(opts); here instead. Otherwise the querystring is being doubly appended.

I also noticed that some of the API calls in dockerode use a trailing ? in the endpoint path. Is there a specific reason for this?

Once in a while, I have this exception caused by the following code:

TypeError: Cannot read property 'pipe' of null
    at Modem.followProgress (/booya/dunya/node_modules/dockerode/node_modules/docker-modem/lib/modem.js:289:9)
    at /booya/dunya/lib/client.js:127:38
    at /booya/dunya/node_modules/dockerode/lib/docker.js:69:5
    at /booya/dunya/node_modules/dockerode/node_modules/docker-modem/lib/modem.js:235:7
    at IncomingMessage.<anonymous> (/booya/dunya/node_modules/dockerode/node_modules/dock

const debug = require('debug')('docker')
const dockerode = require('dockerode')({
    host: 'http://localhost',
    port: 2375
})

function pullImage(imageName) {
    return new Promise(function(resolve, reject) {
        dockerode.pull(imageName, function(err, stream) {
            if (err) {
                reject(err)
            }
            dockerode.modem.followProgress(stream, onFinished, onProgress)
            function onFinished(err, output) {
                if (err) {
                    reject(new Error(err))
                }
                resolve(output)
            }
            function onProgress(event) {
                debug(event)
            }
        })
    })
}

$ docker -v
Docker version 1.10.3, build 20f81dd

Could not really track down the issue. Any ideas?

Hi There,

Listing in the package dependencies is the npm package [email protected]. That package has a dependency for underscore but no version is specified.

As a result when using the npm shrinkwrap command in my project, I am experiencing this error #npm/npm#11493

Is there any way you can update and release a your dependencies to a more recent version of follow-redirects. They have fixed this issue in later releases by modifying their package.json dependencies and not having any dependencies.

Your package is used by the npm package docker-modem which will use any 0.3.x release. Thus if you update your package version to 0.3.2, this problem would be fixed for myself and any other user who uses npm shrinkwrap.

Let me know if you require any clarification.

For some reason if an options object like this is submitted:

{
  "method":"POST",
  "path":"/v1.37/containers/...blah.../attach",
  "options":{
    "_body":{},
    "_query":{
      "logs":"true",
      "stream":"true",
      "stdin":"false",
      "stdout":"true",
      "stderr":"true"
    }
  },
  "statusCodes":{"101":true,"200":true,"400":true,"404":true,"500":true},
  "isStream":true
}

the parameters in _query are not added to the URL. However, if there is a question mark in the URL (in path), they are added:

{
  "method":"POST",
  "path":"/v1.37/containers/...blah.../attach?",
  "options":{
    "_body":{},
    "_query":{
      "logs":"true",
      "stream":"true",
      "stdin":"false",
      "stdout":"true",
      "stderr":"true"
    }
  },
  "statusCodes":{"101":true,"200":true,"400":true,"404":true,"500":true},
  "isStream":true
}

The line that causes this is modem.js#L115, but I'm wary of touching it since I'm not quite sure what the logic was.

Example of error:

starting container with non-empty request body was deprecated since v1.10 and removed in v1.12

This is caused by modem.dial sending {} and setting Transfer-Encoding: chunked on POST methods when no body is specified.

Here's a quick hack to work around it, illustrating the problem bits:

diff --git a/lib/modem.js b/lib/modem.js
index a096ad9..0b1f2db 100644
--- a/lib/modem.js
+++ b/lib/modem.js
@@ -140,14 +140,18 @@ Modem.prototype.dial = function(options, callback) {
     optionsf.headers['Content-Type'] = 'application/tar';
   } else if (opts && options.method === 'POST') {
     data = JSON.stringify(opts._body || opts);
-    optionsf.headers['Content-Type'] = 'application/json';
+    if (data !== '{}' && data !== '""') {
+      optionsf.headers['Content-Type'] = 'application/json';
+    } else {
+      data = undefined;
+    }
   }

   if (typeof data === "string") {
     optionsf.headers['Content-Length'] = Buffer.byteLength(data);
   } else if (Buffer.isBuffer(data) === true) {
     optionsf.headers['Content-Length'] = data.length;
-  } else if (optionsf.method === 'POST' || optionsf.method === 'PUT') {
+  } else if (optionsf.method === 'PUT' || options.hijack || options.openStdin) {
     // Has body
     optionsf.headers['Transfer-Encoding'] = 'chunked';
   }

When preparing the URL here, the schema is missing, which in my case results in a malformed URL localhost:/containers/json? instead of http://localhost:2375/containers/json? something like.

In the following I get obviously a 'Connection refused' error.

Here's my sample code:

var docker = new Docker({"host": "localhost", "port" : 2365});
docker.listContainers(function (err, containers) {
    if (err) {
        throw err;
    }
    containers.forEach(function (containerInfo) {
        console.log(containerInfo.Id);
    });
});

Using http://localhost as host works, though, but I guess this is not the way it is expected.

Ok, just checked the documentation. At least in the example you use the schema as well, so my fault since it was indeed expected ;-).

However, I think the API can be improved: Either use url as an option or add schema as an additional option. http://localhost is not a host as we know it.

Is there a specific reason behind silently supressing an error and returning a string instead of a parsed object as shown below?

https://github.com/apocas/docker-modem/blob/master/lib/modem.js#L199-L203

Modem.prototype.buildRequest = function(options, context, data, callback) {
    ....
    req.on('response', function(res) {
    ...
        try {
          json = JSON.parse(chunks);
        } catch (e) {
          json = chunks;
        }
    ...
})

I had a bunch of stopped containers and docker.listContainers({all: true}, ... ) broke silently.

If there's no reason, then I would like to submit a patch.

I have this issue, on some user machines this works fine but for some it throws the following errors:

TypeError: Cannot read property 'pipe' of null
    at Modem.followProgress (/data/app/node_modules/docker-modem/lib/modem.js:316:10)

Following is the implementation

docker.pull(repoTag, function(err, stream) {
    docker.modem.followProgress(stream, onFinished, onProgress);
    function onFinished(err, output) {
        ....
    }
    function onProgress(event) {
        ....
    }
}

Docker Version 18.03.1-ce-mac65 (24312)

Request to add support for Docker v18.09 native SSH.
This new connection method between client and engine allows for simple and shared SSH configuration which is far more common and more easily administered that the prior custom CA/certs solution.

Via the docker cli, it is easily done by setting the env var DOCKER_HOST=ssh://hostname or directly on the docker command using the -H parameter as in

docker -H ssh://hostname info

Here is the docker API code docker/cli#1014
and some articles on it
https://raesene.github.io/blog/2018/11/11/Docker-18-09-SSH/
https://medium.com/lucjuggery/docker-tips-access-the-docker-daemon-via-ssh-97cd6b44a53

Hello,

Thanks for creating dockerode & docker-modem!

The following bug is happening in dockerode, but I think it is more docker-modem related.

It happens if there are any "special characters" (like German äöü or any other like àò) in options:

docker.createContainer({
    'Image' : 'ubuntu',
    'Cmd' : ['echo','üüüäö'],
}, function(err, container) {
 console.log(err);
}

Error: HTTP code is 500 which indicates error: server error - unexpected EOF

The docker debug log says:

[debug] api.go:1038 Calling POST /containers/create
2014/01/15 22:54:04 POST /containers/create...
[error] api.go:1064 Error: unexpected EOF
[error] api.go:87 HTTP Error: statusCode=500 unexpected EOF

If I remove the setting of "Content-Length" in docker-modem it works:

https://github.com/josiasmontag/docker-modem/commit/cf798fd34a87cd4df993507d105ec724253b8561

I removed Content-Length in my private repo and now I do not have any problems with these special characters.

I do not know why exactly this is happening - maybe it is a bug in docker and not in dokcer-modem?

The API states that some of the query parameters can be duplicated. for example the 'extrahosts' and 't' parameters of ImageBuild.

I see this library makes use of querystrings, and correctly converts objects to JSON strings. But, it doesn't handle arrays. So, for example, given an object like:

{
   extrahosts: [
     "somehost:162.242.195.82",
     "otherhost:50.31.209.229",
   ]
}

querystrings will covert to 'extrahosts[0]=somehost:162.242.195.82&extrahosts[1]=otherhost:50.31.209.229' (escaped of course).

docker-py uses the requests package which simply repeats the name and is consistent with the API docs: 'extrahosts=somehost:162.242.195.82&extrahosts=otherhost:50.31.209.229' .

While the array notation may work on some systems, it all depends on the binary the daemon uses. It may very well break on some OS's or daemon versions.

Docker uses tcp protocol when working with boot2docker, and https://github.com/apocas/docker-modem/blob/master/lib/modem.js#L102 doesn't support tcp, get undefined.

While it is pretty standard to use TLS when port 2376 it should not be assumed this is a given or at the very least we should be able to override it.

For comparison when using docker from the command line there were no issues when using the same host + port configuration.

When switching docker remote server / vscode extension to use other port it works without issues.

This is the line in question:
if (process.env.DOCKER_TLS_VERIFY === '1' || opts.port === '2376')

Options I see are either:

• Remove check based on port number entirely
• Or make the check more explicit to enable overriding the fallback to https
  if (process.env.DOCKER_TLS_VERIFY === '1' || (process.env.DOCKER_TLS_VERIFY !== '0' && opts.port === '2376'))

From correctness standpoint the check on port should be removed, but I do understand this might break a lot of peoples setup if they have not properly set the verify flag as officially required when wanting to use TLS.

Referencing the related issue:
microsoft/vscode-docker#2164

Due to a combination of defaultOpts (https://github.com/apocas/docker-modem/blob/master/lib/modem.js#L16) and this line, any value passed in as the user in DOCKER_HOST would be ignored. I believe the behavior would be that the current user is used. There's a high chance that won't work, however--if the local username is not the same as the SSH username.

Ideally we should alter defaultOpts to permit things like ssh://user@host

the buildImage uses fs.readFileSync on the source tar.

1. this blocks the event-loop.

worse yet, if the tar file is large, a buffer is allocated the size of the tar file. for a 1GB build context, for example, you exceed the max buffer size node allows.
you'll have to recompile v8 to get this to work.

pull request for working with stream instead opened.

Right now, docker-model hides option passing to HttpDuplex. I'm trying to configure the initial highWaterMark option for streaming, but am unable to pass that to the underlying stream. Is it possible to expose more surface area?

First off, thanks for creating docker-modem and dockerode. They have been very helpful recently.

I have a use-case where I would like to connect to a docker container on a server which is accessible through ssh. I am doing some slightly unholy acts to make this work locally, but I'm pretty sure there's an easy solution that could be implemented by giving docker-modem the ability to tunnel to a docker socket through an ssh connection. I'd be happy to add this functionality if it were something that you would be open to accepting in a pull request. I wanted to reach out first before to make sure their was interest before taking the time to do it.

Let me know what you think.

Would be handy for programs generating large tar output (or receiving inbound archive data from elsewhere) for image builds to avoid buffering large amounts of data.

Commit 8be3b43 seems to have added the console.log to the code: