Weird input to next command line about duf HOT 17 CLOSED

I've opened muesli/termenv#27 which implements the second approach and solves this issue. 🙂

from duf.

Just a heads up: this may have just been fixed upstream and the fix should land in the next duf release:

muesli/termenv#26

from duf.

@reinob It'd be helpful if you could run duf with strace and log into a file, so we can see what's going on:

$ strace -o /tmp/log -e read,write -s 1000 duf

Can you paste the contens of the log file? If it's too large just paste it on https://gist.github.com

from duf.

As far as I understand the issue, termenv issues the request and waits for up to 100ms for the response. Terminals which don't support the request will silently ignore it, and not write anything, so the timeout is needed.

The problem now is that if the latency between the program (running on the server) and the terminal (running on the user's workstation) is higher than 100ms, the timeout happens before the response is received.

I have two ideas to resolve this within termenv:

• Issue a request that all terminals understand (I think there's some kind of identification, \e[c?) and measure the latency, use double of that for the timeout
• Issue the request for reading the background color and immediately also issue a request all terminals will respond to, then read the next response without a timeout. If it is the background color response, then read the second response and return the color. If the first response is the generic response, we have a terminal which does not support the background request.

Thoughts?

(yes, I tend to get distracted by such things and dig in deep...)

from duf.

This is indeed duf trying to detect your terminal's preferred background color. I could imagine this being a problem with really slow connections. Is this reproducible on particular servers only? Is it always reproducible or intermittently happening?

from duf.

I've only tested it using lxterminal on my local computer, where it works fine, and from there ssh'ing to three servers. Two of them (one in my home network, another is a VPS) do not display this issue.

The third one is a google f1-micro (free-tier), which is kinda slow but not THAT slow. Here I consistently get the "11;rgb.." being output after running.

from duf.

I hope there's a way to disable this background color check, as I fear that due to whatever temporary things that may be happening (making a fast computer go slow, whatever) the output garbage will not just be output on the screen but taken as stdin for the next command (!!)
(I've tested this making a script which calls duf and then "cat - > output", and the garbage output is actually read by cat and stored)
So this can result in highly undesirable results, and can even be security relevant (I'm not running duf as root until this is 100% resolved)

from duf.

Thank you @muesli, I will try to test the new fix as soon as I can!

Thank you too @fd0. Here's the output of strace, it's only 9 lines (but should, in as far as strace can catch that, display the problem, as it did happen).

Anyway, here's the output:
read(3, "2097152\n", 20) = 8 write(1, "\33]11;?\7", 7) = 7 read(3, "18 23 0:18 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw\n19 23 0:4 / /proc rw,nosuid,nodev,noexec,relatime shared:15 - proc proc rw\n20 23 0:6 / /dev rw,nosuid,relatime shared:2 - devtmpfs udev rw,size=289728k,nr_inodes=72432,mode=755\n21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000\n22 23 0:20 / /run rw,nosuid,noexec,relatime shared:5 - tmpfs tmpfs rw,size=59728k,mode=755\n23 0 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw,discard,errors=remount-ro\n24 18 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw\n25 20 0:21 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw\n26 22 0:22 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k\n27 18 0:23 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=755\n28 27 0:24 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:10 - cgroup2 cgroup2 rw,nsdelegate\n29 27 0:25 / /sys/f"..., 4096) = 3150 read(3, "", 4096) = 0 write(1, "\342\225\255\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\225\256\n\342\224\202 2 local devices \342\224\202\n\342\224\234\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\244\n\342\224\202 MOUNTED ON \342\224\202 SIZE \342\224\202 USED \342\224\202 AVAIL \342\224\202 USE% \342\224\202 TYPE \342\224\202 FILESYSTEM \342\224\202\n\342\224\234\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\274\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\274\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\274\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\274\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\274\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\274\342\224\200\342\224\200\342\224\200\342\224\200\342\224"..., 1663) = 1663 write(1, "\n", 1) = 1 write(1, "\342\225\255\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\225\256\n\342\224\202 5 special devices \342\224\202\n\342\224\234\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\254\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\244\n\342\224\202 MOUNTED ON \342\224\202 SIZE \342\224\202 USED \342\224\202 AVAIL \342\224\202 USE% \342\224\202 TYPE \342\224\202 FILESYSTEM \342\224\202\n\342\224\234\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\274\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\274\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\274\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\274\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342"..., 2361) = 2361 write(1, "\n", 1) = 1 +++ exited with 0 +++

I can't understand why github offers the option to quote "code" (using backticks) to then make a mess of the line endings.
I've pasted the trace into https://gist.github.com/reinob/44aa118fe37151de540b3c2bcfba5f32 for convenience.

from duf.

Thanks! You can delimit code blocks with three backticks on a separate line, like this:

    ```
    code here
    ```

from duf.

Ah, bummer, the string length is not enough, can you retry with -s 99999?

from duf.

This one -> https://gist.github.com/reinob/731308454423c4a54715bf497b2c650e should be OK, the longest line 6511 characters long :)

from duf.

Ok, hm. When you ran this with strace, were the characters printed as you reported in this issue?

from duf.

Yes, of course. Otherwise I wouldn't have posted it :)

The string "rgb:" is not output, but each (non-printable?) character is shown in octal \xxx, which makes it awkward ot read. I'll see if I can output raw ASCII or hexadecimal.

Even with hexadecimal it doesn't improve, as only non-printable (> 127) characters are encoded. But at least the string "rgb:" should not need that, but it's not there. I could be that this is the output, generated by the terminal itself (not by duf), in response to something received from duf.

from duf.

Ok, thanks for the confirmation!

What happens here is that duf (via the termenv library) queries the terminal to find out what the background color is. This is the query:

write(1, "\33]11;?\7", 7)               = 7

Terminals which support this query will usually respond very fast. Terminals which do not support this will not respond at all, so there's a timeout for waiting for a response.

In your case, the terminal supports the query, but is too slow to respond, so duf continues and prints its output, and only after duf is finished the terminal responds and prints the background color response to the shell.

Are you using ssh to log into a remote server with high latency maybe?

from duf.

Thanks for the explanation!

It's a google cloud VM with a ping time, from home, of ~ 114ms.
The issue does not happen on a VPS (ping ~20ms) nor on local (ping ~1ms) server.

But I'm not so sure if the latency is the only issue here. I tried with my VPS, but over TOR (which does make the interactive response slower than that of the google VM) and the issue did NOT happen.

I think google may be doing something weird, but I cannot know on which level. My own terminal (lxterminal), the connection, and the ssh protocol itself are not affected by this. All servers run Debian 10.

If it's only the google VM then I don't really care much, but I fear that non-interactive use of duf (like in a daily cronjob sending server statistics) could randomly fail due to stdin being unexpectedly feed with "random" -- from the point of view of the next command -- garbage.

Add.: just to make sure I also tested with xterm (instead of lxterminal). Same result..

from duf.

Ok, thanks!

I fear that non-interactive use of duf (like in a daily cronjob sending server statistics) could randomly fail due to stdin being unexpectedly feed with "random" -- from the point of view of the next command -- garbage.

You don't need to worry about that. For non-interactive invocations (e.g. cron, systemd-timers), duf won't even print the query because it detects early in the termenv code that the process is not attached to any kind of terminal. :)

from duf.

I'm able to easily reproduce the issue by adding 200ms of artificial latency on my laptop:

$ sudo tc qdisc add dev wlan0 root netem delay 200ms 10ms

Then run the hello-world example on a server:

$ ssh srv
fd0@srv $ cd tmp/termenv/examples/hello-world
fd0@srv $ go run .

	bold faint italic underline crossout
	red green yellow blue magenta cyan gray
	red green yellow blue magenta cyan gray

	Has dark background? true
fd0@srv $ 11;rgb:fafa/fafa/fafa

Delay can be removed like this:

$ sudo tc qdisc del root dev wlan0

from duf.