Is/are the maintainer(s) interested in making this project into a native Debian package? Asking because I use session-manager-plugin on Debian and Ubuntu.

We ran into an issue where running the session manager plugin while using temporary credentials obtained through calling:

aws sso login

We were getting this error, which appeared at first to be an issue with the plugin being unable to reach the AWS KMS endpoint:

Encountered error while initiating handshake. Handshake timed out. Please ensure that you have the latest version of the session manager plugin.

I verified that I was running the latest version of the plugin, and yet still it wasn't working. I turned on logging for the plugin and found this error:

2021-06-23 15:26:29 ERROR [generateEncryptionKey @ encrypter.go.60] Error generating data key from KMS: Error calling KMS GenerateDataKey API: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors,

Based on some searching, it appears to be a result of the AWS Go API not properly supporting the AWS SSO temporary credentials. I verified this by switching over to hardcoded keys copied from the AWS SSO applications page.

I looked in the repo, and it appears that this project is using a very old version of the AWS Go SDK from early last year. Could we get the Go SDK version updated so get some of the fixes that have occurred in the last 1.5 years? Thank you!

I don't know if this is a feature or a bug... but I regularly use the joe text editor and CTRL-D is one of the standard shortcuts which when used in an SSM SSH session it instantly disconnects with an error...

Cannot perform start session: read /dev/stdin: resource temporarily unavailable

On a fresh SSH connection, when I CTRL-D it exits...

Starting session with SessionId: username-0ed94becce0968820
$    <-  HERE I HIT CTRL-D

Exiting session with sessionId: username-0ed94becce0968820.

I assume/hope this is not a feature or some internal SSM hook to disconnect or something, but incase it is, is there a way to rebind it perhaps to another keystroke? I assume I'm not the only person to hit this issue, as more people transition off of using SSH and onto SSM, imho having an fully-compatible ssh-like tunnel is critical. And having special key combinations that are "caught" by the ssm engine feels like a potential incompatibility with software run in the remote console that may use this hook.

Invoking session-manager-plugin without "sessionId" causes a panic.

session-manager-plugin "{\"streamUrl\":\"wss://ssmmessages.us-west-2.amazonaws.com/v1/data-channel/123\",\"tokenValue\":\"abc\"}" "us-east-1" "StartSession"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x151bb35]

goroutine 1 [running]:
github.com/aws/SSMCLI/src/sessionmanagerplugin/session.ValidateInputAndStartSession(0xc00001e080, 0x4, 0x4, 0x1772260, 0xc00000e018)
        /local/p4clients/pkgbuild-_Kjh0/workspace/src/SSMCLI/build/private/src/github.com/aws/SSMCLI/src/sessionmanagerplugin/session/session.go:171 +0x5d5
main.main()
        /local/p4clients/pkgbuild-_Kjh0/workspace/src/SSMCLI/src/sessionmanagerplugin-main/main.go:26 +0x5d

Same behavior with an empty JSON string:

session-manager-plugin "{}" "us-east-1" "StartSession"

I am working on a Raspberry PI 4 and it requires armhf compatible packages. I see the build for arm in the release makefile but I can't find the package itself and it's not referenced anywhere else.

Is there a reason why the package is not distributed and the target for arm is missing in the makefile?

I'd like to avoid start building things on my own if there's a blocker or even better if there's an available .deb for arm to download and install.

I can connect to instance i-123456 in the Console.

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
$ aws --version
aws-cli/2.2.17 Python/3.8.8 Linux/4.4.0-18362-Microsoft exe/x86_64.ubuntu.20 prompt/off
$ session-manager-plugin --version
1.2.205.0
$ aws ssm start-session --target i-123456
An error occurred (TargetNotConnected) when calling the StartSession operation: i-123456 is not connected.
$ curl https://ssm.us-east-1.amazonaws.com/
<UnknownOperationException/>

Any pointers on what might be causing this?

Current situation

Currently, this project uses the first version of AWS SDK for Go.

Proposal

My proposal is to upgrade to the latest SDK for Go made by AWS (v2): https://github.com/aws/aws-sdk-go-v2

When using the AWS session manager on windows I'm noticing a few issues:

• If I open any sort of text editor or just type a command, it is limiting to 80 columns, 24 lines, rather than use the window size (same can be confirmed when echoing $COLUMNS or $LINES (ssh works fine)
• If I open any sort of text editor, the home/end keys no longer work (ssh works fine)

Seems to be happening on both linux & windows

There appears to be a v1.2.458.0 version available, even though there have been no code changes in this repo and the latest GitHub release is Release 1.2.398.0 - 2022-10-13.

AWS Copilot CLI

AWS Copilot CLI Looks like the Session Manager plugin is using version 1.2.398.0.
Would you like to update it to the latest version 1.2.458.0? 

Checking latest hosted version:

$ curl -s https://s3.amazonaws.com/session-manager-downloads/plugin/latest/VERSION
1.2.458.0

Version v1.2.458 seems to be buggy and is likely also the cause of this issue: aws/copilot-cli#4627.

As of some recent version the tunneling experience is really terrible and every query times out with:

SSL SYSCALL error: EOF detected

In another repo for TablePlus someone commented that they found an issue with a Google SDK, so I suspect this is the same underlying Python package in session-manager-plugin that is causing these issues.

This did not happen even a few months ago.

Hi, I would like to use it on Manjaro. It is possible? I didn't find any docs about it.

Describe the issue

If someone has to forward multiple ports they have to use the start-session command multiple times. For example, I want to forward port 8000 and 8080 to my local machine, then I'll do

aws ssm start-session --target <ec2-instance-id> --region <ec2-instance-region> --document-name AWS-StartPortForwardingSession  --parameters '{"portNumber":["8080"], "localPortNumber":["8080"]}

aws ssm start-session --target <ec2-instance-id> --region <ec2-instance-region> --document-name AWS-StartPortForwardingSession  --parameters '{"portNumber":["8000"], "localPortNumber":["8000"]}

Is there (or could there be) any easy way to achieve this multiple port forwarding?

Expected behavior

Ideally, this should be achieved by some command like this:

aws ssm start-session --target <ec2-instance-id> --region <ec2-instance-region> --document-name AWS-StartPortForwardingSession  --parameters '{"portNumber":["8080", "8000"], "localPortNumber":["8080", "8000"]}

Hello,

I am using the PortForwardingSession to access some services in our AWS account. When using it locally everything works fine and I have access via localhost:<port>. However, inside a docker container I'm not able to access the same port. At the same time I am able to access a different port with a local web server without any issues. So the problem only seems to exist in combination of SSM and Docker.
When using nmap from inside the container it also shows the port used for the forwarding session as closed.

docker --version -> 20.10.8
session-manager-plugin --version -> 1.2.245.0

running on Ubuntu 20.04.3 LTS

Do you know any reason why the SSM port behaves differently in comparison to e.g. a webserver.

Many Thanks!

Tried both the Ubuntu server deb package, and also tried using alien to convert the rpm to deb package.

Error returned as follows:

Running checkstyle.sh on mainline produces the following output:

./Tools/src/checkstyle.sh
Run checkstyle script
Run 'gofmt'
Error: Found files not formatted by gofmt
/path/to/source/session-manager-plugin/src/log/log_unix.go
/path/to/source/session-manager-plugin/src/log/log_windows.go
/path/to/source/session-manager-plugin/src/sessionmanagerplugin/session/sessionutil/control_signals_unix.go
/path/to/source/session-manager-plugin/src/sessionmanagerplugin/session/sessionutil/control_signals_windows.go
/path/to/source/session-manager-plugin/src/sessionmanagerplugin/session/sessionutil/sessionutil_unix.go
/path/to/source/session-manager-plugin/src/sessionmanagerplugin/session/sessionutil/sessionutil_windows.go
/path/to/source/session-manager-plugin/src/sessionmanagerplugin/session/shellsession/shellsession_unix.go
/path/to/source/session-manager-plugin/src/sessionmanagerplugin/session/shellsession/shellsession_windows.go
Please run 'gofmt -w' for files listed.

Summary

When building the session-manager-plugin binary from source code using Docker, the resulting binary exits unexpectedly when used with the AWS-StartPortForwardingSessionToRemoteHost document.

Steps to reproduce

1. Clone repo (reproduces on 1.2.398.0 and previous versions)
2. Follow the Working with Docker instructions to build the binary
   1. Run docker build -t session-manager-plugin-image .
   2. Run docker run -it --rm --name session-manager-plugin -v pwd:/session-manager-plugin session-manager-plugin-image make release
3. Copy the resulting binary (bin/darwing_amd64_plugin/session-manager-plugin for MacOS) to the installation folder (/usr/local/bin/ for MacOS)
4. Run AWS CLI against the binary:

   aws ssm start-session \
       --target  <your-target-ec2-instance> \
       --document-name AWS-StartPortForwardingSessionToRemoteHost \
       --parameters '{"host":["<your-target-host-name>"],"portNumber":["<your-target-port>"], "localPortNumber":["<your-local-port>"]}'
   

5. In 2 to 5 minutes, the session will end unexpectedly. (Idle session timeout: 20m, Maximum session duration: 60m are set in the session manager settings in AWS console)

Considerations

The session-manager-plugin binary downloaded from the link provided in the installation manual normally works in the same scenario. Looks like the build process used for the installation packages differs from what's described in the readme. Another piece of evidence for this is that the VERSION file in the repository contains 1.2.0.0 while the downloaded package contains the correct version (1.2.398.0 in my case).

It would be great if properly built binaries were released along with the source code for each release.

Or maybe my way of using the build artifacts is wrong?

Hi,

After a MacOs upgrade Monterey 12.1 Rosetta has been removed from my laptop.

I got:
bad CPU type in executable: /usr/local/bin/session-manager-plugin

I can not run the plugin anymore.

Do you plan to provide this plugin for M1 soon?
Is there a Python alternative to this plugin?
How to build it for M1?

Thanks.

Hi,

Most likely user/installation error but I've been using execute-command using the aws cli for a few months. I've not used it for a month or two and coming back to it I get the error below.

aws ecs execute-command  \
    --region eu-west-1 \
    --cluster ***** \
    --task ****************** \
    --container xray-daemon \
    --command "/bin/bash" \
    --interactive


The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


Starting session with SessionId: ecs-execute-command-0228164ca8ff6e2c1


SessionId: ecs-execute-command-0228164ca8ff6e2c1 : Plugin with name InteractiveCommands not found. Step name: InteractiveCommands

I've upgrade my aws-cli to aws-cli/2.7.22 Python/3.10.6 Darwin/21.3.0 source/arm64 prompt/off and session-manager-plugin to 1.2.339.0 but the problem persists.

Is there anything I need to do to get InteractiveCommands plugin added?

awscli v2.9.0 introduced sso-session section in $HOME/.aws/config. But it seems session-manager-plugin doesn't support it yet, and failed to load the config. In my understanding the root cause is that aws-sdk-go v1 doesn't support it.

To fix the issue it might be better to migrate aws-sdk-go v1 to v2 since it looks like aws-sdk-go-v2 supports it.

related issue: aws/aws-sdk-go#4649

Thanks.

Hi,

Would it be possible to integrate X11 forwarding into session-manager? I've attempted a few things such as port forwarding, but I've had no luck.

Is it possible to forward UDP ports?

Sessions opened with the session manager plugin always report an exit code of zero even when the last executed command within the session returns a non-zero value. For example:

$ aws ssm start-session --target [ec2 instance]
  Starting session with SessionId: [name]-0ac90007f8af508b9
  $ exit 1
  Exiting session with sessionId: [name]-0ac90007f8af508b9.

$ echo $?
  0

Compare this to SSH where the exit code is carried over:

$ ssh [host]
  [host]:~$ exit 1
  logout
  Connection to [host] closed.

$ echo $?
  1

Not reporting the exit code makes scripting extremely difficult since there's no way to tell if a given command was successful or not. Reporting the exit code of the last command run in the session would be more inline with other standard tools.

NOTE: This issue's contents and title have been significantly changed as of Jan 15 because of some folks getting to the underlying issue.

The underlying issue is that when using a BSD-based system (like OS-X) this plugin sends some undesirable control signals. This appears to ONLY affect Macs, not Windows or Linux.

@pedros007 Believes this issue is related to the ControlSignals handled by a ShellSession

Reproduce steps which seems to only happen on a MacOS terminal:

• Start a shell session aws ssm start-session --target i-00112233445566778
• Once connected, press Ctrl-y
• Expected behavior: paste from the kill-ring (which is empty, so nothing should happen)
• Actual behavior: The session terminates with the log line

Cannot perform start session: read /dev/stdin: resource temporarily unavailable

The kludgy workaround currently is

• Run stty dsusp undef on your host.
• Start a shell session aws ssm start-session --target i-00112233445566778
• Press Ctrl-y to paste from the kill ring (which is empty, so nothing happens). This is what every other OS does, and what Linux normally does when you SSH.

For more details about this issue please see some of the comments towards the bottom of the thread/issue.

There is a awscli2 nix package, but it does not support session-manager-plugin installation.

It would be great if we can install session manager plugin using nix package manager.

awscli2: https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/admin/awscli2/default.nix

NixOS/nixpkgs#217443

Original issue: aws/aws-cli#5496 (I tried to transfer this but wasn't able to.)

Hi everybody,
I recently started to use this plugin and it is very powerful and flexible but I noticed a strange behavior:
If I type the "@" (the "at" character) in a remote ssh session started with
aws ssm start-session --target "<instance-id>"
the character is simply ignored.
This makes impossible to type any email address for example.

I also tried the in-browser version and it works with no issues.

On my Windows machine I also tried classical cmd and new Windows terminal but same stuff.

Do you have the same issue?

Recently, I made a simple program to connect the ECS task interactively by using the session-manager-plugin. It work fine most of the cases but sometimes the session initially freeze and cannot exit or close the session properly. The only way to exit the session is that just go to the session manager console, find the wrong session and close it.

The environments are as follows:

• Plugin Version : 1.2.245.0
• OS : Windows 10 Pro, 19043.1165
• Shell : Git Bash
• AWSCLI : aws-cli/2.2.22 Python/3.8.8 Windows/10 exe/AMD64 prompt/off

My code is here:

• https://github.com/daeho-ro/execs/blob/main/internal/ecs.go#L250

The real work example for my task:

session-manager-plugin '{"SessionId":"ecs-execute-command-05907f33b122d680e","StreamUrl":"wss://ssmmessages.ap-northeast-2.amazonaws.com/v1/data-channel/ecs-execute-command-05907f33b122d680e?role=publish_subscribe","TokenValue":"<tokenValue>"}' 'ap-northeast-2' 'StartSession' '' '{"Target":"ecs:ECS-FIN-DEV-OPSNOW_8f01c7eb36314ce6a5bc234a236504fd_8f01c7eb36314ce6a5bc234a236504fd-3107182542","DocumentName":"8f01c7eb36314ce6a5bc234a236504fd-1630142996","Parameters":null}' 'https://ssm.ap-northeast-2.amazonaws.com'

The debug logs for the session-manager plugin are here with 3 cases:

well connected

2021-08-28 18:14:06 DEBUG [Start @ config_watcher.go.48] Start File Watcher On: C:\Program Files\Amazon\SessionManagerPlugin\seelog.xml
2021-08-28 18:14:06 DEBUG [Start @ config_watcher.go.52] Start Watcher on directory: C:\Program Files\Amazon\SessionManagerPlugin
2021-08-28 18:14:06 DEBUG [Initialize @ streaming.go.161] Calling Initialize Datachannel for role: publish_subscribe
2021-08-28 18:14:06 INFO [OpenConnection @ websocketutil.go.59] Opening websocket connection to: %!(EXTRA string=wss://ssmmessages.ap-northeast-2.amazonaws.com/v1/data-channel/ecs-execute-command-0c259474041b9f66a?role=publish_subscribe)
2021-08-28 18:14:06 INFO [OpenConnection @ websocketutil.go.67] Successfully opened websocket connection to: %!(EXTRA string=wss://ssmmessages.ap-northeast-2.amazonaws.com/v1/data-channel/ecs-execute-command-0c259474041b9f66a?role=publish_subscribe)
2021-08-28 18:14:06 INFO [FinalizeDataChannelHandshake @ streaming.go.199] Sending token through data channel wss://ssmmessages.ap-northeast-2.amazonaws.com/v1/data-channel/ecs-execute-command-0c259474041b9f66a?role=publish_subscribe to acknowledge connection
2021-08-28 18:14:06 DEBUG [func1 @ websocketchannel.go.94] WebsocketChannel: Send ping. Message.
2021-08-28 18:14:07 WARN [ProcessFirstMessage @ sessionhandler.go.81] Setting session type to shell based on PayloadType!
2021-08-28 18:14:07 DEBUG [func1 @ shellsession.go.125] Sending input size data: {"cols":226,"rows":34}
2021-08-28 18:15:29 INFO [HandleChannelClosedMessage @ streaming.go.764] Exiting session with sessionId: ecs-execute-command-0c259474041b9f66a with output:

connected but found error

2021-08-28 18:15:59 DEBUG [Start @ config_watcher.go.48] Start File Watcher On: C:\Program Files\Amazon\SessionManagerPlugin\seelog.xml
2021-08-28 18:15:59 DEBUG [Start @ config_watcher.go.52] Start Watcher on directory: C:\Program Files\Amazon\SessionManagerPlugin
2021-08-28 18:15:59 DEBUG [Initialize @ streaming.go.161] Calling Initialize Datachannel for role: publish_subscribe
2021-08-28 18:15:59 INFO [OpenConnection @ websocketutil.go.59] Opening websocket connection to: %!(EXTRA string=wss://ssmmessages.ap-northeast-2.amazonaws.com/v1/data-channel/ecs-execute-command-03b97060017ed267c?role=publish_subscribe)
2021-08-28 18:15:59 INFO [OpenConnection @ websocketutil.go.67] Successfully opened websocket connection to: %!(EXTRA string=wss://ssmmessages.ap-northeast-2.amazonaws.com/v1/data-channel/ecs-execute-command-03b97060017ed267c?role=publish_subscribe)
2021-08-28 18:15:59 INFO [FinalizeDataChannelHandshake @ streaming.go.199] Sending token through data channel wss://ssmmessages.ap-northeast-2.amazonaws.com/v1/data-channel/ecs-execute-command-03b97060017ed267c?role=publish_subscribe to acknowledge connection
2021-08-28 18:15:59 DEBUG [func1 @ websocketchannel.go.94] WebsocketChannel: Send ping. Message.
2021-08-28 18:15:59 ERROR [OutputMessageHandler @ streaming.go.398] Invalid outputMessage: {112 start_publication 1 1630142161946 1 3 bb07bbdb-b21b-42d5-abd0-ad4568138236 [227 176 196 66 152 252 28 20 154 251 244 200 153 111 185 36 39 174 65 228 100 155 147 76 164 149 153 27 120 82 184 85] 17 1937006962 [115 116 97 114 116 95 112 117 98 108 105 99 97 116 105 111 110]}, err: payload Hash is not valid.
2021-08-28 18:16:00 WARN [ProcessFirstMessage @ sessionhandler.go.81] Setting session type to shell based on PayloadType!
2021-08-28 18:16:00 DEBUG [func1 @ shellsession.go.125] Sending input size data: {"cols":226,"rows":34}
2021-08-28 18:16:06 INFO [HandleChannelClosedMessage @ streaming.go.764] Exiting session with sessionId: ecs-execute-command-03b97060017ed267c with output:

session freeze initially

2021-08-28 18:17:45 DEBUG [Start @ config_watcher.go.48] Start File Watcher On: C:\Program Files\Amazon\SessionManagerPlugin\seelog.xml
2021-08-28 18:17:45 DEBUG [Start @ config_watcher.go.52] Start Watcher on directory: C:\Program Files\Amazon\SessionManagerPlugin
2021-08-28 18:17:45 DEBUG [Initialize @ streaming.go.161] Calling Initialize Datachannel for role: publish_subscribe
2021-08-28 18:17:45 INFO [OpenConnection @ websocketutil.go.59] Opening websocket connection to: %!(EXTRA string=wss://ssmmessages.ap-northeast-2.amazonaws.com/v1/data-channel/ecs-execute-command-01e59c207cd39799b?role=publish_subscribe)
2021-08-28 18:17:45 INFO [OpenConnection @ websocketutil.go.67] Successfully opened websocket connection to: %!(EXTRA string=wss://ssmmessages.ap-northeast-2.amazonaws.com/v1/data-channel/ecs-execute-command-01e59c207cd39799b?role=publish_subscribe)
2021-08-28 18:17:45 DEBUG [func1 @ websocketchannel.go.94] WebsocketChannel: Send ping. Message.
2021-08-28 18:17:45 INFO [FinalizeDataChannelHandshake @ streaming.go.199] Sending token through data channel wss://ssmmessages.ap-northeast-2.amazonaws.com/v1/data-channel/ecs-execute-command-01e59c207cd39799b?role=publish_subscribe to acknowledge connection
2021-08-28 18:17:45 ERROR [OutputMessageHandler @ streaming.go.398] Invalid outputMessage: {112 start_publication 1 1630142267767 1 3 dd5c0814-e81b-44d7-86ed-c2a1eaecba75 [227 176 196 66 152 252 28 20 154 251 244 200 153 111 185 36 39 174 65 228 100 155 147 76 164 149 153 27 120 82 184 85] 17 1937006962 [115 116 97 114 116 95 112 117 98 108 105 99 97 116 105 111 110]}, err: payload Hash is not valid.
2021-08-28 18:17:46 WARN [ProcessFirstMessage @ sessionhandler.go.81] Setting session type to shell based on PayloadType!
2021-08-28 18:17:46 DEBUG [func1 @ shellsession.go.125] Sending input size data: {"cols":226,"rows":34}
2021-08-28 18:17:46 DEBUG [func1 @ streaming.go.339] Resend stream data message: 1
2021-08-28 18:17:46 DEBUG [func1 @ streaming.go.339] Resend stream data message: 1
<repeated the resend stream data forever here>

I cannot figure out what is wrong here and need help.
Thanks.

I originally opened this issue on packer-plugin-amazon, but I've realised this might be a session issue.

Overview of the Issue

I'm attempting to create an AMI in Amazon EBS that involves a reboot of the mac2.metal instance. I'm attempting to use pause_before after rebooting in shell in the step above. I'm also setting expect_disconnect to true

This unfortunately fails with Connection to destination port failed, check SSM Agent logs. when the instance reboots and the SSM agent is no longer connected.

Using this config works on our existing implementation using the vsphere-clone builder.

Is there a way to retry until the host is back up and for packer not to fail deployment ?

The websocket keepalives are set fairly high at 5 minutes.

I am a large customer whose developers operate behind a busy corporate proxy which aggressively closes idle connections.

Would you be able to make this valuable configurable? Even if it was via environment variable that is fine. I need to lower this to around 15-30 seconds probably.

2021-10-13 00:48:01 DEBUG Closing websocket connection to:%!(EXTRA string=1.2.3.4:1234)
2021-10-13 00:48:01 ERROR Failed to close websocket: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp 4.3.2.1:50638->1.2.3.4:1234: write: broken pipe
2021-10-13 00:48:01 DEBUG Closing datachannel failed with error: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp 4.3.2.1:50638->1.2.3.4:1234: write: broken pipe
2021-10-13 00:48:01 INFO Opening websocket connection to: %!(EXTRA string=wss://ssmmessages.us-east-1.amazonaws.com/v1/data-channel/[email protected]?role=publish_subscribe)
2021-10-13 00:48:01 DEBUG Resend stream data message: 1
2021-10-13 00:48:01 ERROR Unable to send stream data message: Can't send message: Connection is closed.
2021-10-13 00:48:01 DEBUG Resend stream data message: 1
2021-10-13 00:48:01 ERROR Unable to send stream data message: Can't send message: Connection is closed.
......
2021-10-13 00:48:01 DEBUG Resend stream data message: 1
2021-10-13 00:48:01 DEBUG Resend stream data message: 2
2021-10-13 00:48:01 DEBUG Resend stream data message: 3
2021-10-13 00:48:01 DEBUG Resend stream data message: 4
2021-10-13 00:48:01 DEBUG Resend stream data message: 5
2021-10-13 00:48:01 DEBUG Resend stream data message: 6

Hi,

Currently your installer for windows requires Admin permission regardless of whether you change the install path to one that doesn't require Admin rights. I'm guessing this is because the installer attempts to add install directory to system PATH.
Could you provide an option to not add to the PATH so that users with no Admin rights can use the installer?

Alternatively you could just include the session-manager-plugin.exe in the releases on github?

This might be a niche of devs wit no admin rights but I would appreciate if you could consider one of the two options.

Thanks

Hi there,

Thanks for publishing the source code. I'd like to integrate the components of session-manager-plugin into my own tooling, but I'm not able to import them, because the paths in the source code refer to github.com/aws/SSMCLI instead of github.com/aws/session-manager-plugin.

go get github.com/aws/session-manager-plugin/src/sessionmanagerplugin/session

go: downloading github.com/aws/session-manager-plugin v0.0.0-20210609234953-e4ad4017df96
go: downloading github.com/aws/aws-sdk-go v1.38.60
github.com/aws/session-manager-plugin/src/sessionmanagerplugin/session imports
	github.com/aws/SSMCLI/src/config: cannot find module providing package github.com/aws/SSMCLI/src/config
github.com/aws/session-manager-plugin/src/sessionmanagerplugin/session imports
	github.com/aws/SSMCLI/src/datachannel: cannot find module providing package github.com/aws/SSMCLI/src/datachannel
github.com/aws/session-manager-plugin/src/sessionmanagerplugin/session imports
	github.com/aws/SSMCLI/src/log: cannot find module providing package github.com/aws/SSMCLI/src/log
github.com/aws/session-manager-plugin/src/sessionmanagerplugin/session imports
	github.com/aws/SSMCLI/src/message: cannot find module providing package github.com/aws/SSMCLI/src/message
github.com/aws/session-manager-plugin/src/sessionmanagerplugin/session imports
	github.com/aws/SSMCLI/src/retry: cannot find module providing package github.com/aws/SSMCLI/src/retry
github.com/aws/session-manager-plugin/src/sessionmanagerplugin/session imports
	github.com/aws/SSMCLI/src/sdkutil: cannot find module providing package github.com/aws/SSMCLI/src/sdkutil
github.com/aws/session-manager-plugin/src/sessionmanagerplugin/session imports
	github.com/aws/SSMCLI/src/sessionmanagerplugin/session/sessionutil: cannot find module providing package github.com/aws/SSMCLI/src/sessionmanagerplugin/session/sessionutil
github.com/aws/session-manager-plugin/src/sessionmanagerplugin/session imports
	github.com/aws/SSMCLI/src/version: cannot find module providing package github.com/aws/SSMCLI/src/version
``

On installing the plugin I see this - note the error on the last line:

doylands@HGRSDELTCG30106:~$ sudo dpkg -i session-manager-plugin.deb
Selecting previously unselected package session-manager-plugin.
(Reading database ... 84984 files and directories currently installed.)
Preparing to unpack session-manager-plugin.deb ...
Preparing for install
/var/lib/dpkg/tmp.ci/preinst: 4: stop: not found
Unpacking session-manager-plugin (1.2.398.0-1) ...
Setting up session-manager-plugin (1.2.398.0-1) ...
Starting session-manager-plugin
/var/lib/dpkg/info/session-manager-plugin.postinst: 4: start: not found

Looking at /var/lib/dpkg/info/session-manager-plugin.postinst it shows this:

echo "Starting session-manager-plugin"
if [ $(cat /proc/1/comm) = init ]
then
    start session-manager-plugin || true
elif [ $(cat /proc/1/comm) = systemd ]
then
    systemctl enable session-manager-plugin
    systemctl start session-manager-plugin
    systemctl daemon-reload
fi
if [ -f /usr/local/bin/session-manager-plugin ]
then
    rm /usr/local/bin/session-manager-plugin
fi
ln -s /usr/local/sessionmanagerplugin/bin/session-manager-plugin /usr/local/bin/session-manager-plugin

It is trying to run a start command as my system is using init, but this does not exist.

Same sort of issue happens on uninstall due to this file containing a stop command:

/var/lib/dpkg/info/session-manager-plugin.prerm

When installing on macOS via brew:

brew install session-manager-plugin

On run the plugin errors with:

"session-manager-plugin" cannot be opened because the developer cannot be verified.

The workaround is to use --no-quarantine, eg:

brew reinstall session-manager-plugin --no-quarantine

However it would great if the binary were code signed and notarized so the above doesn't appear and to provide confidence to end-users.

The plugin does not properly handle the pause_publication message.

If connected to an ECS task that is killed while still being connected, the logs show:

2021-11-05 03:57:33 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:33 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:33 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:33 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:33 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
2021-11-05 03:57:34 WARN [OutputMessageHandler @ streaming.go.411] Invalid message type received: %spause_publication
2021-11-05 03:57:34 DEBUG [func1 @ streaming.go.339] Resend stream data message: 241
...

(This is an infinite loop and will just fill up the log file).

Ideally, the server should have closed the connection (and sent the channel_closed message).

Since that doesn't appear to be happening, it would be nice if the client could at least handle the pause_publication message.

In my use case, it would be best if this would exit the plugin.

Steps to recreate:

• Enable debug logging (not sure if this is required, but obviously I have it enabled)
• Start an ECS task where the command is sleep 86400 (you can probably set something lower)
• run an ExecuteCommand against that task with /bin/bash in interactive mode
• Wait for the sleep timer to expire
• plugin will remain running, but does not respond to input.
• log files will accumulate until you forcibly terminate the plugin.

The issue

If I have an open session-manager-plugin session and the endpoint running amazon-ssm-agent stops responding, my session-manager-plugin shell will become unresponsive for an indeterminate amount of time.

Would it make sense and be feasible for the session-manager-plugin to detect an unresponsive endpoint and end the session-manager-plugin process itself, returning control to the parent process?

I recognize that there are many factors to consider; for example, how long to wait before considering the endpoint unresponsive, or whether to allow overriding that time to wait.

If it is decided that this behavior is not worth supporting, could documentation be added to state that session-manager-plugin will never terminate itself when the endpoint becomes unresponsive? At least that would provide some official documentation that I can point to in my projects.

I am unsure whether this issue should go here or in the Systems Manager forums, since it depends on what end you guys feel the issue should be handled.

This issue is independent of #6, but a solution to that issue would minimize the need for a solution to this issue.

My use case

I am connecting to ECS Fargate containers via session-manager-plugin. I have been using session-manager-plugin directly, but the issue is also reproducible with aws ecs execute-command.
If I have an open session with one of my containers and then the container dies, my shell session will freeze. I can go to the AWS Systems Manager Session Manager dashboard and manually terminate the session, which will return control of my shell to me.
It would be much more convenient in my eyes for the session-manager-plugin itself to detect an unresponsive endpoint and automatically end itself.

This behavior can be reproduced by starting up an ECS Fargate task with the execute agent enabled and with an entrypoint of sleep 180, then connecting with session-manager-plugin and waiting 3 minutes. At that point, the session-manager-plugin shell will become unresponsive.

My session-manager-plugin version is 1.2.339.0.

My containers are running on Fargate platform version 1.4.0.

Or otherwise please indicate if PRs are welcome as this is not clear from the other issues.

Hi, I hope I've found the right place for this bug.

After running aws ecs execute-command --interactive, I loose handling of special characters for erase, werase, kill, eof, etc by the terminal. Running stty icanon is sufficient to get things working again.

I am using version 1.2.295.0 on linux.

I found it necessary to run a shell to reproduce this. Executing /bin/true did not reproduce the issue.

:; aws ecs execute-command --interactive ... --command /bin/bash

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


Starting session with SessionId: ecs-execute-command-0ffda0a0ffda0a0ff
root@ip-1-2-3-4# exit


Exiting session with sessionId: ecs-execute-command-0ffda0a0ffda0a0ff.

:; cat # observe erase and eof not being processed, interrupt still works
^?^?^?^D^C
:; stty icanon
:; cat # observe handlng of special characters is restored
:;

The issue in this case was the icanon flag not being reset to it's previous state, but it's a good idea to capture the existing termios settings and restore them before exiting.

It isn't possible to build the binary with go 1.16+. Support for 1.15 was dropped back in August. Right now it's not possible to build the package with a supported version of go. The go vet step in checkstyle.sh fails because the project doesn't use modules (see #11 for a fix for that). Then doing the build it fails with multiple go.mod file not found in current directory or any parent directory; see 'go help modules' errors.

Please apply #11 so this project can be built with modern go.

Hi,

It would be great if there was a markdown file that had different tools that used session-manager-plugin. There is some older tools I only found out about recently that really helped with work I was doing. Would be great to highlight these non official tools as such but also let people find them.

Cheers,
Andrew

It appears that the change to remove the idle timeout in smux does not respect the regional SSM session timeout if you're using an older version of the agent with some types of SSM documents. Specifically, if you start a StartPortForwardingSession and leave it idle it will close after ~30s, which is far shorter than the default SSM session timeout.

I'm currently running 3.1.1374 of the ssm-agent since that is the latest version that is available for Debian in the public S3 bucket.

Problem

When a session freezes there is no way to exit the session.

How SSH addresses the issue

SSH has support for escape characters, one of which closes the session from the client side which is implemented by typing <Enter>~.. This is different from a EOF on stdin in that it doesn't require the remote to be responsive to exit the session.

Info on this can be found by searching for "Escape Characters" at https://linux.die.net/man/1/ssh.

I am using SSM plugin as proxy command for SSH to execute commands on EC2 instances. This is mostly used in jenkins pipelines.

Latest version of SSM plugin (session-manager-plugin-1.2.398.0-1.x86_64) is running on Centos7, installed with official rpm.

Occasionally (pretty rarely), I get this in output:
Command '['session-manager-plugin', '{"SessionId": "xxxxx", "TokenValue": "xxxxxxxxx", "StreamUrl": "wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/xxxx?role=publish_subscribe&cell-number=xxxxx", "ResponseMetadata": {"RequestId": "xxxx", "HTTPStatusCode": 200, "HTTPHeaders": {"server": "Server", "date": "Tue, 10 Jan 2023 12:10:51 GMT", "content-type": "application/x-amz-json-1.1", "content-length": "1059", "connection": "keep-alive", "x-amzn-requestid": "xxxx"}, "RetryAttempts": 0}}', 'eu-west-1', 'StartSession', 'xxxx', '{"Target": "i-xxxxxx", "DocumentName": "AWS-StartSSHSession", "Parameters": {"portNumber": ["22"]}}', 'https://ssm.eu-west-1.amazonaws.com']' died with <Signals.SIGPIPE: 13>.

Command doesn't fail, just getting this message where http status code is 200.

Any idea what's causing this? Or any way to hide the message (since it reveals some sensitive data) if it's not causing any issue?

This request is for AWS to hire someone to fix the packaging for its various customer-exposed tools (like this project). Specifically it's intended for the upper-management and executive team that should be overseeing customer experience.

The Linux packages are pretty bad. No dependency on AWS CLI, obscure package name, non-standard file paths, faulty install scripts, unnecessary install-time dependencies, and no official support for any particular Linux distro other than Ubuntu Server (presumably because that's what the developers use). No CI (seriously? it's built right into GitHub!), no matrix of target platforms, no packages submitted to distributions, no generic installer script.

Not only that, but the downloads and source code don't have any cryptographic signatures. If the packages were distributed by a package manager, you'd at least get signatures for free. But since there's not even a checksum listed on the AWS docs, the user has no way of knowing if their package or download has been tampered with. You're asking your customers to expose their internal networks to administrative users that have downloaded tools whose security is literally impossible to verify.

These are not senior-level tasks requiring rare talent. A core AWS team dedicated to open source could liaise with project teams like this one to standardize things like CI, linting, packaging, installing. Hiring for this shouldn't be challenging, even with the current job market - people all over the world do this for fun in their spare time.

Considering that AWS makes $59 Billion dollars a year, and has something like 100,000 engineers, there's no justification for the state of things. If this is how AWS manages their public tools, I can't imagine how shoddy the internal ones are. And judging on the number of bugs in the new Web Console, there's a much larger organizational issue at play (which doesn't bode well for AWS' stock price). All Azure needs to do to start eating your lunch is make better packages, and all the Linux nerds (like myself) will start opting for Azure first. The irony!

Describe the bug

Note: this issue was originally opened here: aws/aws-cli#7574, but I was advised to re-open it here.

On WSL the aws ssm start-session command corrupts my terminal after it terminates due to inactivity. Regardless of what key I press, no characters appear in the terminal any more. I have tried Ctrl + C too, but it doesn't change the unresponsiveness of my terminal either.

Expected Behavior

I should be able to use the terminal for other commands.

Current Behavior

Regardless of what key I press, nothing appears in the terminal. The consequence of this is that I have to close the terminal and open a new one after more-or-less every connection, which is uncomfortable, considering that my history is gone, I have to go back to the same directory and have to initialize my AWS context again.

Reproduction Steps

1. Install Ubuntu 2020 using WSL 2 on a Windows Enterprise machine.
2. Install AWS CLI and the SSM plugin
3. Launch a session with aws ssm start-session --target i-XXXXXXXXXXX
4. Wait 5 minutes
5. Now the terminal becomes unresponsive

Possible Solution

No response

Additional Information/Context

I start the ssm session like this:

aws ssm start-session --target i-0ac4445bb00XXXXXX
Starting session with SessionId: [email protected]
sh-4.2$ id
uid=1001(ssm-user) gid=1007(ssm-user) groups=1007(ssm-user) context=system_u:system_r:unconfined_service_t:s0

Then I wait a couple of minutes and press a random key:

sh-4.2$ Cannot perform start session: write tcp 172.22.193.94:55404->193.106.225.65:8080: write: broken pipe

The session is terminated due to some timeout. The problem occurs after this, as my terminal becomes unusable for further commands, nothing appears on my screen any more as I type.

CLI version used

aws-cli/2.9.12 Python/3.9.11 Linux/5.10.16.3-microsoft-standard-WSL2 exe/x86_64.ubuntu.22 prompt/off

Environment details (OS name and version, etc.)

Windows 10 Enterprise, WSL 2, Ubuntu 2020

Description

When trying to forward a port to a remote host using the new feature, I'm able to open the tunnel and even connect to the remote host through it, but the port is only opened, locally, on an IPv6 interface.

While it's not a problem when running a client directly on my host, it is a problem when trying to run the client inside docker containers, because the docker engine only binds to IPv4 interfaces, so the local port is not available.

Expected

The port is available on both IPv6 and IPv4, and it's possible to connect to the port from a docker container running on the local host.

Actual

The port is only available on IPv6 and not accessible from a docker container.

Versions

I'm on MacOS 11.6.7 (Big sur), with Docker desktop 4.8.2 (I don't think that matters), aws cli 2.7.10 and session-manager-plugin 1.2.339.0.

Steps to reproduce

Here's what I do:

aws --profile experiment ssm start-session \
     --target i-XXXXXXX \
     --document-name AWS-StartPortForwardingSessionToRemoteHost \
     --parameters '{"host":["YYYYYYYYY"],"portNumber":["3306"], "localPortNumber":["3316"]}'

When the tunnel is opened, I run lsof -i -P | grep -i "listen" and get only the following (relevant) line:

session-m 52168 xxxxx   22u  IPv6 0x64bf25a801ab42ab      0t0  TCP localhost:3316 (LISTEN)

No IPv4 equivalent.

Using the mysql client from my host on port 3316 connects without issues, but doing the same thing from a docker container (using -h host.docker.internal) does not.

$> mysql -h ::0 -P 3316 -u admin -p 
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
[...]

$> mysql -h 127.0.0.1 -P 3316 -u admin -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:3316' (61)

$> docker run -ti --rm mysql:8.0 mysql -h host.docker.internal -P 3316 -u admin -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on 'host.docker.internal:3316' (111)

Connecting with a instance with the AWS SSM plugin on Windows 10 with the Spanish (from Spain) Keyboard layout using CMD and PowerShell (v 5.1 and 7.4) some characters are not recorded correctly, characters like ? @ }

Hi team,

I've built the docker image with Session Manager Plugin based on alpine without installing aws cli in the container.
Is it possible to use the containerized plugin along with aws cli installed on my local machine?

For example to execute this command:
aws ssm send-command --document-name "AWS-RunShellScript" --instance-ids ${instance_ID}

In session-manager-plugin, it is in the state of IsAwsCliUpgradeNeeded with 4 arguments as LegacyArgumentLength.

At this time, ProcessKMSEncryptionHandshakeAction will display an error, but please consider changing the error wording and state name at this time.

This is because session-manager-plugin is used in addition to awscli such as copilot-cli and ecspresso, and the error wording "Please upgrade to latest version of AWS CLI" can be misleading.
https://github.com/aws/copilot-cli/blob/ada23de30b1dab0b0d0659a66d45bdfc6646954e/internal/pkg/exec/ssm_plugin.go#L56
https://github.com/kayac/ecspresso/blob/v1/exec.go#L93

Thank you for your consideration.